search

NIEHS / beethoven

BEETHOVEN is: Building an Extensible, rEproducible, Test-driven, Harmonized, Open-source, Versioned, ENsemble model for air quality
https://niehs.github.io/beethoven/
Other
2 stars 0 forks source link

AQS daily data update #296

Open sigmafelix opened 4 months ago

sigmafelix commented 4 months ago

I found that the AQS data was updated on 10-26-2023, and now the full year data of 2022 is available. We retrieved pre-generated AQS data last August-September. At that time, the last date of update was 11-14-2022 and the 2022 data contained up to October 2022. Another good news is the update included data in 2023 (the latest date is 09-30-2023), which serves as our test data for performance evaluation. I will download the data and work on updating covariates accordingly.

image image
kyle-messier commented 4 months ago

@sigmafelix I think it would be helpful to NOT include all of the currently available data, then do a full integration test of our pipeline at the end. That is, once we develop the models, how long does it take us to add new data and go through the pipeline.

sigmafelix commented 4 months ago

@Spatiotemporal-Exposures-and-Toxicology I will keep the current AQS data then. New data were put into 20240227 under ./input/aqs.

sigmafelix commented 1 month ago

@kyle-messier @MAKassien @mitchellmanware With the minimum POC value at each site, 80 hoursdays of data have duplicate site ID-time-location-sample duration combination with considerably different measurements. Do we take mean or the first record coming up in the dataset?

To note, the data frame below only includes the mainland states. The number of duplicated hoursdays increase to 979 if AK and HI are both considered.

Rows 151-160 are from a site in Oregon. I remember that wildfires peaked during 2020-08-22 - 2020-09-06, thus the higher measurements would make more sense.

           site_id Date Local POC Latitude  Longitude Observation Count Sample Duration Arithmetic Mean
1   06053000288101 2018-11-10   3 36.48187 -121.73333                 1   24-HR BLK AVG            50.7
2   06053000288101 2018-11-10   3 36.48187 -121.73333                 1   24-HR BLK AVG             7.2
3   06053000288101 2018-11-20   3 36.48187 -121.73333                 1   24-HR BLK AVG            13.7
4   06053000288101 2018-11-20   3 36.48187 -121.73333                 1   24-HR BLK AVG             9.0
5   06053000888101 2018-11-09   3 36.20929 -121.12637                 1   24-HR BLK AVG             5.3
6   06053000888101 2018-11-09   3 36.20929 -121.12637                 1   24-HR BLK AVG             6.8
7   06053000888101 2018-11-20   3 36.20929 -121.12637                 1   24-HR BLK AVG             7.7
8   06053000888101 2018-11-20   3 36.20929 -121.12637                 1   24-HR BLK AVG            16.5
9   06061000388101 2018-11-09   1 38.93568 -121.09959                 1   24-HR BLK AVG             3.0
10  06061000388101 2018-11-09   1 38.93568 -121.09959                 1   24-HR BLK AVG            10.0
11  06061000388101 2018-11-10   1 38.93568 -121.09959                 1   24-HR BLK AVG             6.3
12  06061000388101 2018-11-10   1 38.93568 -121.09959                 1   24-HR BLK AVG            28.8
13  06061000388101 2018-11-11   1 38.93568 -121.09959                 1   24-HR BLK AVG             3.2
14  06061000388101 2018-11-11   1 38.93568 -121.09959                 1   24-HR BLK AVG            52.5
15  06061000388101 2018-11-12   1 38.93568 -121.09959                 1   24-HR BLK AVG             5.0
16  06061000388101 2018-11-12   1 38.93568 -121.09959                 1   24-HR BLK AVG            20.4
17  06061000388101 2018-11-13   1 38.93568 -121.09959                 1   24-HR BLK AVG             7.0
18  06061000388101 2018-11-13   1 38.93568 -121.09959                 1   24-HR BLK AVG            16.5
19  06061000388101 2018-11-18   1 38.93568 -121.09959                 1   24-HR BLK AVG             7.0
20  06061000388101 2018-11-18   1 38.93568 -121.09959                 1   24-HR BLK AVG            32.4
21  06061000388101 2018-11-19   1 38.93568 -121.09959                 1   24-HR BLK AVG             8.3
22  06061000388101 2018-11-19   1 38.93568 -121.09959                 1   24-HR BLK AVG            14.8
23  06061000388101 2018-11-20   1 38.93568 -121.09959                 1   24-HR BLK AVG             5.8
24  06061000388101 2018-11-20   1 38.93568 -121.09959                 1   24-HR BLK AVG            12.0
25  06061000388101 2018-11-21   1 38.93568 -121.09959                 1   24-HR BLK AVG             6.6
26  06061000388101 2018-11-21   1 38.93568 -121.09959                 1   24-HR BLK AVG            10.2
27  06069000288101 2018-11-09   3 36.84343 -121.36210                 1   24-HR BLK AVG            21.3
28  06069000288101 2018-11-09   3 36.84343 -121.36210                 1   24-HR BLK AVG             5.8
29  06087000788101 2018-11-21   3 36.98332 -121.98822                 1   24-HR BLK AVG             8.0
30  06087000788101 2018-11-21   3 36.98332 -121.98822                 1   24-HR BLK AVG             5.6
31  06087100588101 2018-11-04   3 37.06315 -122.08309                 1   24-HR BLK AVG            68.3
32  06087100588101 2018-11-04   3 37.06315 -122.08309                 1   24-HR BLK AVG             5.0
33  06087100588101 2018-11-05   3 37.06315 -122.08309                 1   24-HR BLK AVG             3.0
34  06087100588101 2018-11-05   3 37.06315 -122.08309                 1   24-HR BLK AVG            18.8
35  06087100588101 2018-11-08   3 37.06315 -122.08309                 1   24-HR BLK AVG            11.2
36  06087100588101 2018-11-08   3 37.06315 -122.08309                 1   24-HR BLK AVG             6.1
37  06087100588101 2018-11-21   3 37.06315 -122.08309                 1   24-HR BLK AVG             0.5
38  06087100588101 2018-11-21   3 37.06315 -122.08309                 1   24-HR BLK AVG             6.8
39  11001004188101 2018-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            16.1
40  11001004188101 2018-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            27.5
41  11001004188101 2019-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            14.8
42  11001004188101 2019-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            67.5
43  11001004188101 2019-07-05   1 38.89557  -76.95807                 1   24-HR BLK AVG            15.8
44  11001004188101 2019-07-05   1 38.89557  -76.95807                 1   24-HR BLK AVG            17.2
45  11001004188101 2021-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            39.5
46  11001004188101 2021-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG             8.7
47  11001004188101 2021-07-05   1 38.89557  -76.95807                 1   24-HR BLK AVG            37.0
48  11001004188101 2021-07-05   1 38.89557  -76.95807                 1   24-HR BLK AVG            19.1
49  11001004188101 2022-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            13.8
50  11001004188101 2022-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            46.3
51  11001004388101 2018-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG            16.0
52  11001004388101 2018-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG            27.9
53  11001004388101 2019-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG            37.7
54  11001004388101 2019-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG            10.9
55  11001004388101 2019-07-05   1 38.92185  -77.01318                 1   24-HR BLK AVG            14.0
56  11001004388101 2019-07-05   1 38.92185  -77.01318                 1   24-HR BLK AVG            14.8
57  11001004388101 2021-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG            36.0
58  11001004388101 2021-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG             6.2
59  11001004388101 2021-07-05   1 38.92185  -77.01318                 1   24-HR BLK AVG            40.2
60  11001004388101 2021-07-05   1 38.92185  -77.01318                 1   24-HR BLK AVG            19.4
61  11001004388101 2022-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG             8.3
62  11001004388101 2022-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG            22.3
63  11001005188101 2018-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            14.3
64  11001005188101 2018-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            25.5
65  11001005188101 2019-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            56.3
66  11001005188101 2019-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            12.9
67  11001005188101 2019-07-05   1 38.89477  -76.95343                 1   24-HR BLK AVG            16.3
68  11001005188101 2019-07-05   1 38.89477  -76.95343                 1   24-HR BLK AVG            17.4
69  11001005188101 2021-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            11.3
70  11001005188101 2021-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            58.9
71  11001005188101 2021-07-05   1 38.89477  -76.95343                 1   24-HR BLK AVG            42.7
72  11001005188101 2021-07-05   1 38.89477  -76.95343                 1   24-HR BLK AVG            21.8
73  11001005188101 2022-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            13.8
74  11001005188101 2022-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            43.6
75  11001005388101 2019-07-04   1 38.87516  -77.01282                 1   24-HR BLK AVG            43.1
76  11001005388101 2019-07-04   1 38.87516  -77.01282                 1   24-HR BLK AVG            12.0
77  11001005388101 2019-07-05   1 38.87516  -77.01282                 1   24-HR BLK AVG            16.4
78  11001005388101 2019-07-05   1 38.87516  -77.01282                 1   24-HR BLK AVG            15.2
79  11001005388101 2021-07-04   1 38.87516  -77.01282                 1   24-HR BLK AVG            31.9
80  11001005388101 2021-07-04   1 38.87516  -77.01282                 1   24-HR BLK AVG             9.1
81  11001005388101 2021-07-05   1 38.87516  -77.01282                 1   24-HR BLK AVG            20.8
82  11001005388101 2021-07-05   1 38.87516  -77.01282                 1   24-HR BLK AVG            32.9
83  11001005388101 2022-07-04   1 38.87516  -77.01282                 1   24-HR BLK AVG            13.9
84  11001005388101 2022-07-04   1 38.87516  -77.01282                 1   24-HR BLK AVG            36.1
85  24019000488101 2022-02-21   3 38.58752  -76.14101                 1   24-HR BLK AVG             6.5
86  24019000488101 2022-02-21   3 38.58752  -76.14101                 1   24-HR BLK AVG             8.6
87  24029000288101 2018-02-01   3 39.30502  -75.79732                 1   24-HR BLK AVG            18.2
88  24029000288101 2018-02-01   3 39.30502  -75.79732                 1   24-HR BLK AVG             9.0
89  24029000288101 2018-03-19   3 39.30502  -75.79732                 1   24-HR BLK AVG            10.2
90  24029000288101 2018-03-19   3 39.30502  -75.79732                 1   24-HR BLK AVG            10.8
91  30029004988101 2020-07-04   3 48.36369 -114.18927                 1   24-HR BLK AVG             7.9
92  30029004988101 2020-07-04   3 48.36369 -114.18927                 1   24-HR BLK AVG            12.2
93  30029004988101 2020-07-05   3 48.36369 -114.18927                 1   24-HR BLK AVG             5.9
94  30029004988101 2020-07-05   3 48.36369 -114.18927                 1   24-HR BLK AVG             9.6
95  30029004988101 2022-07-04   3 48.36369 -114.18927                 1   24-HR BLK AVG             9.0
96  30029004988101 2022-07-04   3 48.36369 -114.18927                 1   24-HR BLK AVG             4.2
97  30029004988101 2022-07-05   3 48.36369 -114.18927                 1   24-HR BLK AVG             5.9
98  30029004988101 2022-07-05   3 48.36369 -114.18927                 1   24-HR BLK AVG             3.5
99  30053001888101 2018-07-04   3 48.39155 -115.55331                 1   24-HR BLK AVG             3.8
100 30053001888101 2018-07-04   3 48.39155 -115.55331                 1   24-HR BLK AVG             7.2
101 30053001888101 2018-07-05   3 48.39155 -115.55331                 1   24-HR BLK AVG             7.6
102 30053001888101 2018-07-05   3 48.39155 -115.55331                 1   24-HR BLK AVG             8.8
103 30053001888101 2020-07-04   3 48.39155 -115.55331                 1   24-HR BLK AVG            10.1
104 30053001888101 2020-07-04   3 48.39155 -115.55331                 1   24-HR BLK AVG            13.3
105 30053001888101 2020-07-05   3 48.39155 -115.55331                 1   24-HR BLK AVG             6.9
106 30053001888101 2020-07-05   3 48.39155 -115.55331                 1   24-HR BLK AVG            13.0
107 30063002488101 2018-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG             9.3
108 30063002488101 2018-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG             4.8
109 30063002488101 2019-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG            13.9
110 30063002488101 2019-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG             6.0
111 30063002488101 2020-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG            10.8
112 30063002488101 2020-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG            19.0
113 30063002488101 2020-07-05   3 46.84218 -114.02150                 1   24-HR BLK AVG             8.7
114 30063002488101 2020-07-05   3 46.84218 -114.02150                 1   24-HR BLK AVG             7.8
115 30063002488101 2021-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG            13.3
116 30063002488101 2021-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG             5.8
117 30063002488101 2021-07-05   3 46.84218 -114.02150                 1   24-HR BLK AVG             7.8
118 30063002488101 2021-07-05   3 46.84218 -114.02150                 1   24-HR BLK AVG             7.1
119 30063002488101 2022-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG             2.7
120 30063002488101 2022-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG             5.8
121 30081000788101 2018-07-04   3 46.24362 -114.15889                 1   24-HR BLK AVG             1.2
122 30081000788101 2018-07-04   3 46.24362 -114.15889                 1   24-HR BLK AVG             5.0
123 30081000788101 2020-07-04   3 46.24362 -114.15889                 1   24-HR BLK AVG             4.5
124 30081000788101 2020-07-04   3 46.24362 -114.15889                 1   24-HR BLK AVG             9.7
125 30081000788101 2022-07-04   3 46.24362 -114.15889                 1   24-HR BLK AVG             4.3
126 30081000788101 2022-07-04   3 46.24362 -114.15889                 1   24-HR BLK AVG            11.6
127 30111008788101 2018-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG             5.5
128 30111008788101 2018-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG            38.9
129 30111008788101 2018-09-05   3 45.80631 -108.42598                 1   24-HR BLK AVG             9.8
130 30111008788101 2018-09-05   3 45.80631 -108.42598                 1   24-HR BLK AVG             9.9
131 30111008788101 2019-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG             5.3
132 30111008788101 2019-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG            66.7
133 30111008788101 2019-07-05   3 45.80631 -108.42598                 1   24-HR BLK AVG             7.8
134 30111008788101 2019-07-05   3 45.80631 -108.42598                 1   24-HR BLK AVG            12.0
135 30111008788101 2020-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG             5.8
136 30111008788101 2020-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG            72.9
137 30111008788101 2020-07-05   3 45.80631 -108.42598                 1   24-HR BLK AVG             8.2
138 30111008788101 2020-07-05   3 45.80631 -108.42598                 1   24-HR BLK AVG            10.8
139 30111008788101 2022-07-03   3 45.80631 -108.42598                 1   24-HR BLK AVG             8.2
140 30111008788101 2022-07-03   3 45.80631 -108.42598                 1   24-HR BLK AVG            15.6
141 30111008788101 2022-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG             4.7
142 30111008788101 2022-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG            61.8
143 32003150188101 2018-03-05   3 36.13971 -115.17565                 1   24-HR BLK AVG             3.9
144 32003150188101 2018-03-05   3 36.13971 -115.17565                 1   24-HR BLK AVG             4.1
145 32003150188101 2018-06-12   3 36.13971 -115.17565                 1   24-HR BLK AVG            10.1
146 32003150188101 2018-06-12   3 36.13971 -115.17565                 1   24-HR BLK AVG             7.6
147 37173000288101 2022-04-04   3 35.43477  -83.44213                 1   24-HR BLK AVG             5.4
148 37173000288101 2022-04-04   3 35.43477  -83.44213                 1   24-HR BLK AVG            19.5
149 40031065188101 2022-03-02   3 34.63298  -98.42879                 1   24-HR BLK AVG            10.3
150 40031065188101 2022-03-02   3 34.63298  -98.42879                 1   24-HR BLK AVG            35.0
151 41035000488101 2020-07-27   1 42.19030 -121.73137                 1   24-HR BLK AVG            66.3
152 41035000488101 2020-07-27   1 42.19030 -121.73137                 1   24-HR BLK AVG            13.0
153 41035000488101 2020-08-22   1 42.19030 -121.73137                 1   24-HR BLK AVG             4.7
154 41035000488101 2020-08-22   1 42.19030 -121.73137                 1   24-HR BLK AVG            34.0
155 41035000488101 2020-08-27   1 42.19030 -121.73137                 1   24-HR BLK AVG            12.6
156 41035000488101 2020-08-27   1 42.19030 -121.73137                 1   24-HR BLK AVG            27.2
157 41035000488101 2020-09-04   1 42.19030 -121.73137                 1   24-HR BLK AVG            20.4
158 41035000488101 2020-09-04   1 42.19030 -121.73137                 1   24-HR BLK AVG            50.6
159 41035000488101 2020-09-06   1 42.19030 -121.73137                 1   24-HR BLK AVG            21.5
160 41035000488101 2020-09-06   1 42.19030 -121.73137                 1   24-HR BLK AVG            69.9
MAKassien commented 1 month ago

Thanks for catching this issue @sigmafelix that requires some attention to detail! 80 duplicate points in the whole record aren't too many, luckily. Are they duplicate hours or days? If hours, I would say average should be okay. For days, we could do average too or we may consider doing the highest value to catch things like the fires that Insang mentioned.

sigmafelix commented 1 month ago

@MAKassien Oh, day is the right word. I was obsessed into 24-hour averages too much and ended up being confused with the temporal resolution 😅

sigmafelix commented 1 month ago

Duplicates are coming from whether or not accounting for events, which may affect air quality significantly.

           site_id Date Local POC Latitude  Longitude Observation Count Sample Duration Arithmetic Mean Event Type
1   06053000288101 2018-11-10   3 36.48187 -121.73333                 1   24-HR BLK AVG            50.7   Included
2   06053000288101 2018-11-10   3 36.48187 -121.73333                 1   24-HR BLK AVG             7.2   Excluded
3   06053000288101 2018-11-20   3 36.48187 -121.73333                 1   24-HR BLK AVG            13.7   Included
4   06053000288101 2018-11-20   3 36.48187 -121.73333                 1   24-HR BLK AVG             9.0   Excluded
5   06053000888101 2018-11-09   3 36.20929 -121.12637                 1   24-HR BLK AVG             5.3   Excluded
6   06053000888101 2018-11-09   3 36.20929 -121.12637                 1   24-HR BLK AVG             6.8   Included
7   06053000888101 2018-11-20   3 36.20929 -121.12637                 1   24-HR BLK AVG             7.7   Excluded
8   06053000888101 2018-11-20   3 36.20929 -121.12637                 1   24-HR BLK AVG            16.5   Included
9   06061000388101 2018-11-09   1 38.93568 -121.09959                 1   24-HR BLK AVG             3.0   Excluded
10  06061000388101 2018-11-09   1 38.93568 -121.09959                 1   24-HR BLK AVG            10.0   Included
11  06061000388101 2018-11-10   1 38.93568 -121.09959                 1   24-HR BLK AVG             6.3   Excluded
12  06061000388101 2018-11-10   1 38.93568 -121.09959                 1   24-HR BLK AVG            28.8   Included
13  06061000388101 2018-11-11   1 38.93568 -121.09959                 1   24-HR BLK AVG             3.2   Excluded
14  06061000388101 2018-11-11   1 38.93568 -121.09959                 1   24-HR BLK AVG            52.5   Included
15  06061000388101 2018-11-12   1 38.93568 -121.09959                 1   24-HR BLK AVG             5.0   Excluded
16  06061000388101 2018-11-12   1 38.93568 -121.09959                 1   24-HR BLK AVG            20.4   Included
17  06061000388101 2018-11-13   1 38.93568 -121.09959                 1   24-HR BLK AVG             7.0   Excluded
18  06061000388101 2018-11-13   1 38.93568 -121.09959                 1   24-HR BLK AVG            16.5   Included
19  06061000388101 2018-11-18   1 38.93568 -121.09959                 1   24-HR BLK AVG             7.0   Excluded
20  06061000388101 2018-11-18   1 38.93568 -121.09959                 1   24-HR BLK AVG            32.4   Included
21  06061000388101 2018-11-19   1 38.93568 -121.09959                 1   24-HR BLK AVG             8.3   Excluded
22  06061000388101 2018-11-19   1 38.93568 -121.09959                 1   24-HR BLK AVG            14.8   Included
23  06061000388101 2018-11-20   1 38.93568 -121.09959                 1   24-HR BLK AVG             5.8   Excluded
24  06061000388101 2018-11-20   1 38.93568 -121.09959                 1   24-HR BLK AVG            12.0   Included
25  06061000388101 2018-11-21   1 38.93568 -121.09959                 1   24-HR BLK AVG             6.6   Excluded
26  06061000388101 2018-11-21   1 38.93568 -121.09959                 1   24-HR BLK AVG            10.2   Included
27  06069000288101 2018-11-09   3 36.84343 -121.36210                 1   24-HR BLK AVG            21.3   Included
28  06069000288101 2018-11-09   3 36.84343 -121.36210                 1   24-HR BLK AVG             5.8   Excluded
29  06087000788101 2018-11-21   3 36.98332 -121.98822                 1   24-HR BLK AVG             8.0   Included
30  06087000788101 2018-11-21   3 36.98332 -121.98822                 1   24-HR BLK AVG             5.6   Excluded
31  06087100588101 2018-11-04   3 37.06315 -122.08309                 1   24-HR BLK AVG            68.3   Included
32  06087100588101 2018-11-04   3 37.06315 -122.08309                 1   24-HR BLK AVG             5.0   Excluded
33  06087100588101 2018-11-05   3 37.06315 -122.08309                 1   24-HR BLK AVG             3.0   Excluded
34  06087100588101 2018-11-05   3 37.06315 -122.08309                 1   24-HR BLK AVG            18.8   Included
35  06087100588101 2018-11-08   3 37.06315 -122.08309                 1   24-HR BLK AVG            11.2   Included
36  06087100588101 2018-11-08   3 37.06315 -122.08309                 1   24-HR BLK AVG             6.1   Excluded
37  06087100588101 2018-11-21   3 37.06315 -122.08309                 1   24-HR BLK AVG             0.5   Excluded
38  06087100588101 2018-11-21   3 37.06315 -122.08309                 1   24-HR BLK AVG             6.8   Included
39  11001004188101 2018-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            16.1   Excluded
40  11001004188101 2018-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            27.5   Included
41  11001004188101 2019-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            14.8   Excluded
42  11001004188101 2019-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            67.5   Included
43  11001004188101 2019-07-05   1 38.89557  -76.95807                 1   24-HR BLK AVG            15.8   Excluded
44  11001004188101 2019-07-05   1 38.89557  -76.95807                 1   24-HR BLK AVG            17.2   Included
45  11001004188101 2021-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            39.5   Included
46  11001004188101 2021-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG             8.7   Excluded
47  11001004188101 2021-07-05   1 38.89557  -76.95807                 1   24-HR BLK AVG            37.0   Included
48  11001004188101 2021-07-05   1 38.89557  -76.95807                 1   24-HR BLK AVG            19.1   Excluded
49  11001004188101 2022-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            13.8   Excluded
50  11001004188101 2022-07-04   1 38.89557  -76.95807                 1   24-HR BLK AVG            46.3   Included
51  11001004388101 2018-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG            16.0   Excluded
52  11001004388101 2018-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG            27.9   Included
53  11001004388101 2019-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG            37.7   Included
54  11001004388101 2019-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG            10.9   Excluded
55  11001004388101 2019-07-05   1 38.92185  -77.01318                 1   24-HR BLK AVG            14.0   Excluded
56  11001004388101 2019-07-05   1 38.92185  -77.01318                 1   24-HR BLK AVG            14.8   Included
57  11001004388101 2021-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG            36.0   Included
58  11001004388101 2021-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG             6.2   Excluded
59  11001004388101 2021-07-05   1 38.92185  -77.01318                 1   24-HR BLK AVG            40.2   Included
60  11001004388101 2021-07-05   1 38.92185  -77.01318                 1   24-HR BLK AVG            19.4   Excluded
61  11001004388101 2022-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG             8.3   Excluded
62  11001004388101 2022-07-04   1 38.92185  -77.01318                 1   24-HR BLK AVG            22.3   Included
63  11001005188101 2018-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            14.3   Excluded
64  11001005188101 2018-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            25.5   Included
65  11001005188101 2019-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            56.3   Included
66  11001005188101 2019-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            12.9   Excluded
67  11001005188101 2019-07-05   1 38.89477  -76.95343                 1   24-HR BLK AVG            16.3   Excluded
68  11001005188101 2019-07-05   1 38.89477  -76.95343                 1   24-HR BLK AVG            17.4   Included
69  11001005188101 2021-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            11.3   Excluded
70  11001005188101 2021-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            58.9   Included
71  11001005188101 2021-07-05   1 38.89477  -76.95343                 1   24-HR BLK AVG            42.7   Included
72  11001005188101 2021-07-05   1 38.89477  -76.95343                 1   24-HR BLK AVG            21.8   Excluded
73  11001005188101 2022-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            13.8   Excluded
74  11001005188101 2022-07-04   1 38.89477  -76.95343                 1   24-HR BLK AVG            43.6   Included
75  11001005388101 2019-07-04   1 38.87516  -77.01282                 1   24-HR BLK AVG            43.1   Included
76  11001005388101 2019-07-04   1 38.87516  -77.01282                 1   24-HR BLK AVG            12.0   Excluded
77  11001005388101 2019-07-05   1 38.87516  -77.01282                 1   24-HR BLK AVG            16.4   Included
78  11001005388101 2019-07-05   1 38.87516  -77.01282                 1   24-HR BLK AVG            15.2   Excluded
79  11001005388101 2021-07-04   1 38.87516  -77.01282                 1   24-HR BLK AVG            31.9   Included
80  11001005388101 2021-07-04   1 38.87516  -77.01282                 1   24-HR BLK AVG             9.1   Excluded
81  11001005388101 2021-07-05   1 38.87516  -77.01282                 1   24-HR BLK AVG            20.8   Excluded
82  11001005388101 2021-07-05   1 38.87516  -77.01282                 1   24-HR BLK AVG            32.9   Included
83  11001005388101 2022-07-04   1 38.87516  -77.01282                 1   24-HR BLK AVG            13.9   Excluded
84  11001005388101 2022-07-04   1 38.87516  -77.01282                 1   24-HR BLK AVG            36.1   Included
85  24019000488101 2022-02-21   3 38.58752  -76.14101                 1   24-HR BLK AVG             6.5   Excluded
86  24019000488101 2022-02-21   3 38.58752  -76.14101                 1   24-HR BLK AVG             8.6   Included
87  24029000288101 2018-02-01   3 39.30502  -75.79732                 1   24-HR BLK AVG            18.2   Included
88  24029000288101 2018-02-01   3 39.30502  -75.79732                 1   24-HR BLK AVG             9.0   Excluded
89  24029000288101 2018-03-19   3 39.30502  -75.79732                 1   24-HR BLK AVG            10.2   Excluded
90  24029000288101 2018-03-19   3 39.30502  -75.79732                 1   24-HR BLK AVG            10.8   Included
91  30029004988101 2020-07-04   3 48.36369 -114.18927                 1   24-HR BLK AVG             7.9   Excluded
92  30029004988101 2020-07-04   3 48.36369 -114.18927                 1   24-HR BLK AVG            12.2   Included
93  30029004988101 2020-07-05   3 48.36369 -114.18927                 1   24-HR BLK AVG             5.9   Excluded
94  30029004988101 2020-07-05   3 48.36369 -114.18927                 1   24-HR BLK AVG             9.6   Included
95  30029004988101 2022-07-04   3 48.36369 -114.18927                 1   24-HR BLK AVG             9.0   Included
96  30029004988101 2022-07-04   3 48.36369 -114.18927                 1   24-HR BLK AVG             4.2   Excluded
97  30029004988101 2022-07-05   3 48.36369 -114.18927                 1   24-HR BLK AVG             5.9   Included
98  30029004988101 2022-07-05   3 48.36369 -114.18927                 1   24-HR BLK AVG             3.5   Excluded
99  30053001888101 2018-07-04   3 48.39155 -115.55331                 1   24-HR BLK AVG             3.8   Excluded
100 30053001888101 2018-07-04   3 48.39155 -115.55331                 1   24-HR BLK AVG             7.2   Included
101 30053001888101 2018-07-05   3 48.39155 -115.55331                 1   24-HR BLK AVG             7.6   Excluded
102 30053001888101 2018-07-05   3 48.39155 -115.55331                 1   24-HR BLK AVG             8.8   Included
103 30053001888101 2020-07-04   3 48.39155 -115.55331                 1   24-HR BLK AVG            10.1   Excluded
104 30053001888101 2020-07-04   3 48.39155 -115.55331                 1   24-HR BLK AVG            13.3   Included
105 30053001888101 2020-07-05   3 48.39155 -115.55331                 1   24-HR BLK AVG             6.9   Excluded
106 30053001888101 2020-07-05   3 48.39155 -115.55331                 1   24-HR BLK AVG            13.0   Included
107 30063002488101 2018-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG             9.3   Included
108 30063002488101 2018-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG             4.8   Excluded
109 30063002488101 2019-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG            13.9   Included
110 30063002488101 2019-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG             6.0   Excluded
111 30063002488101 2020-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG            10.8   Excluded
112 30063002488101 2020-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG            19.0   Included
113 30063002488101 2020-07-05   3 46.84218 -114.02150                 1   24-HR BLK AVG             8.7   Included
114 30063002488101 2020-07-05   3 46.84218 -114.02150                 1   24-HR BLK AVG             7.8   Excluded
115 30063002488101 2021-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG            13.3   Included
116 30063002488101 2021-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG             5.8   Excluded
117 30063002488101 2021-07-05   3 46.84218 -114.02150                 1   24-HR BLK AVG             7.8   Included
118 30063002488101 2021-07-05   3 46.84218 -114.02150                 1   24-HR BLK AVG             7.1   Excluded
119 30063002488101 2022-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG             2.7   Excluded
120 30063002488101 2022-07-04   3 46.84218 -114.02150                 1   24-HR BLK AVG             5.8   Included
121 30081000788101 2018-07-04   3 46.24362 -114.15889                 1   24-HR BLK AVG             1.2   Excluded
122 30081000788101 2018-07-04   3 46.24362 -114.15889                 1   24-HR BLK AVG             5.0   Included
123 30081000788101 2020-07-04   3 46.24362 -114.15889                 1   24-HR BLK AVG             4.5   Excluded
124 30081000788101 2020-07-04   3 46.24362 -114.15889                 1   24-HR BLK AVG             9.7   Included
125 30081000788101 2022-07-04   3 46.24362 -114.15889                 1   24-HR BLK AVG             4.3   Excluded
126 30081000788101 2022-07-04   3 46.24362 -114.15889                 1   24-HR BLK AVG            11.6   Included
127 30111008788101 2018-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG             5.5   Excluded
128 30111008788101 2018-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG            38.9   Included
129 30111008788101 2018-09-05   3 45.80631 -108.42598                 1   24-HR BLK AVG             9.8   Excluded
130 30111008788101 2018-09-05   3 45.80631 -108.42598                 1   24-HR BLK AVG             9.9   Included
131 30111008788101 2019-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG             5.3   Excluded
132 30111008788101 2019-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG            66.7   Included
133 30111008788101 2019-07-05   3 45.80631 -108.42598                 1   24-HR BLK AVG             7.8   Excluded
134 30111008788101 2019-07-05   3 45.80631 -108.42598                 1   24-HR BLK AVG            12.0   Included
135 30111008788101 2020-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG             5.8   Excluded
136 30111008788101 2020-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG            72.9   Included
137 30111008788101 2020-07-05   3 45.80631 -108.42598                 1   24-HR BLK AVG             8.2   Excluded
138 30111008788101 2020-07-05   3 45.80631 -108.42598                 1   24-HR BLK AVG            10.8   Included
139 30111008788101 2022-07-03   3 45.80631 -108.42598                 1   24-HR BLK AVG             8.2   Excluded
140 30111008788101 2022-07-03   3 45.80631 -108.42598                 1   24-HR BLK AVG            15.6   Included
141 30111008788101 2022-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG             4.7   Excluded
142 30111008788101 2022-07-04   3 45.80631 -108.42598                 1   24-HR BLK AVG            61.8   Included
143 32003150188101 2018-03-05   3 36.13971 -115.17565                 1   24-HR BLK AVG             3.9   Excluded
144 32003150188101 2018-03-05   3 36.13971 -115.17565                 1   24-HR BLK AVG             4.1   Included
145 32003150188101 2018-06-12   3 36.13971 -115.17565                 1   24-HR BLK AVG            10.1   Excluded
146 32003150188101 2018-06-12   3 36.13971 -115.17565                 1   24-HR BLK AVG             7.6   Included
147 37173000288101 2022-04-04   3 35.43477  -83.44213                 1   24-HR BLK AVG             5.4   Excluded
148 37173000288101 2022-04-04   3 35.43477  -83.44213                 1   24-HR BLK AVG            19.5   Included
149 40031065188101 2022-03-02   3 34.63298  -98.42879                 1   24-HR BLK AVG            10.3   Excluded
150 40031065188101 2022-03-02   3 34.63298  -98.42879                 1   24-HR BLK AVG            35.0   Included
151 41035000488101 2020-07-27   1 42.19030 -121.73137                 1   24-HR BLK AVG            66.3   Included
152 41035000488101 2020-07-27   1 42.19030 -121.73137                 1   24-HR BLK AVG            13.0   Excluded
153 41035000488101 2020-08-22   1 42.19030 -121.73137                 1   24-HR BLK AVG             4.7   Excluded
154 41035000488101 2020-08-22   1 42.19030 -121.73137                 1   24-HR BLK AVG            34.0   Included
155 41035000488101 2020-08-27   1 42.19030 -121.73137                 1   24-HR BLK AVG            12.6   Excluded
156 41035000488101 2020-08-27   1 42.19030 -121.73137                 1   24-HR BLK AVG            27.2   Included
157 41035000488101 2020-09-04   1 42.19030 -121.73137                 1   24-HR BLK AVG            20.4   Excluded
158 41035000488101 2020-09-04   1 42.19030 -121.73137                 1   24-HR BLK AVG            50.6   Included
159 41035000488101 2020-09-06   1 42.19030 -121.73137                 1   24-HR BLK AVG            21.5   Excluded
160 41035000488101 2020-09-06   1 42.19030 -121.73137                 1   24-HR BLK AVG            69.9   Included

At this moment we only need to decide whether we include or exclude the event.

mitchellmanware commented 1 month ago

@sigmafelix Can you explain what you mean by an "event"?

sigmafelix commented 1 month ago

@mitchellmanware

Exceptional events are unusual or naturally occurring events that can affect air quality but are not reasonably controllable using techniques that tribal, state or local air agencies may implement in order to attain and maintain the National Ambient Air Quality Standards (NAAQS). Exceptional events may include wildfires, high wind dust events, prescribed fires, stratospheric ozone intrusions, and volcanic and seismic activities. -- EPA (n.d.) https://www.epa.gov/air-quality-analysis/treatment-air-quality-monitoring-data-influenced-exceptional-events

sigmafelix commented 1 month ago

@kyle-messier For the calculation part, I will keep event flags in the data for the pipeline then filter the rows after we decide.

MAKassien commented 1 month ago

Okay so it's to account for pollution events that were not man-made so they don't take those into account when deciding if an area is complying with the pollution standards, since it wouldn't be fair for the local industries to be asked to cut back emissions if they weren't responsible for the pollution. Since we're interested in capturing the actual concentrations I think we should include the high values.

kyle-messier commented 1 month ago

@sigmafelix I agree with @MAKassien here. Hopefully covariates such as the HMS data will allow our model to adjust for these extreme events. Regardless, we should include all data in the daily averages.