CAMS2_83 PM10 timeseries UK missing stations

andagit commented 1 year ago

Evaluation per country show that UK has at least 5 stations in 2023 measuring PM10, but the Overall evaluation timeseries has only 1 station for a shorter period in 2022, see https://docs.google.com/presentation/d/1-ps2IRJVYOYI4MI3qB1vVIm2oPEQdZB0slQY2DnmRCU/edit#slide=id.g22852a2a6e5_1_2

lewisblake commented 1 year ago

https://cams2-83.aeroval.met.no/overall.php?project=cams2-83&exp_name=forecast-last-seasons&station=GB1034A&par=concso2&tab=timeseries#

charlienegri commented 1 year ago

looking at the raw data in /lustre/storeB/project/fou/kl/CAMS2_83/obs it appears that there's no data for pm10 for GB stations for most of 2021:
grep "^GB.*pm10" 2021*/obsmacc4verif_2021*.csv gives output only for
202106/obsmacc4verif_20210616.csv
202106/obsmacc4verif_20210617.csv
202106/obsmacc4verif_202106178.csv

charlienegri commented 1 year ago

the data is there only from 28th of Feb 2022 on,
the question is why the time series stop at 31st May 2022 then...
the colocated data /lustre/storeB/project/fou/kl/CAMS2_83/evaluation/coldata/cams2-83/forecast-last-season looks ok, e.g. ncks -v concpm10 -d time,17471 -d station_name,700 EMEP/concpm10_concpm10_MOD-EMEP_REF-Obs_20200101_20221231_hourly_ALL-wMOUNTAINS.nc so it must happen in computing the statistics... probably has to do with coverage?

andagit commented 1 year ago

But what about 2023 - under the tag: MAPS there are stations all the way into 2023 too: https://aeroval.met.no/maps.php?project=cams2-83&exp_name=forecast-last-seasons&tab=timeseries&reg=Custom&par=concpm10#

man. 24. apr. 2023 kl. 13:28 skrev Charlie Negri @.***>:

the data is there only from 28th of Feb 2022 on

— Reply to this email directly, view it on GitHub https://github.com/metno/pyaerocom/issues/850#issuecomment-1519960778, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFJJKV3VR7E2JPMYYILLZUDXCZPXFANCNFSM6AAAAAAXD5GEII . You are receiving this because you authored the thread.Message ID: @.***>

--

Anna Carlin Benedictow @. @.> Climate modelling and Air pollution Department http://www.emep.int Norwegian Meteorological Institute (MET.NO) http://www.met.no P.O.Box 43 Blindern, 0313 Oslo, Norway Tel. # 47-947 94 853 <callto:47-22-963%20365>

charlienegri commented 1 year ago

I do not know why they get filtered out in the statistics calculation

charlienegri commented 1 year ago

in the config for these runs we have

obs_remove_outliers=False,
model_remove_outliers=False,

but also

min_num_obs=dict(,
       daily=dict(hourly=18),
    ),

but from the raw data I do not see how this could be the issue

charlienegri commented 1 year ago

if we select for example the station GB0728A,
and we look at the raw data for 2022-05-30:

GB0728A;51.544;  0.678;0037;pm10;2022;05;30;00;1; 5.50000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;01;1; 4.70000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;02;1; 5.10000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;03;1; 6.00000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;04;1; 6.50000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;05;1; 8.60000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;06;1; 9.70000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;07;1; 8.60000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;08;1; 7.40000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;09;1; 6.00000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;10;1; 7.00000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;11;1; 7.40000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;12;1; 6.50000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;13;1; 6.80000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;14;1; 7.20000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;15;1; 5.50000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;16;1; 4.30000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;17;1; 6.60000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;18;1; 6.00000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;19;1; 6.20000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;20;1; 5.50000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;21;1; 6.40000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;22;1; 6.30000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;30;23;1; 6.40000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;05;31;00;1; 5.60000e-09

we have one measurement every hour, as it should be, everything ok
..and the same holds for all the later dates that do not show up in the time series.. e.g. 2022-08-02:

GB0728A;51.544;  0.678;0037;pm10;2022;08;02;00;1; 1.01000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;01;1; 9.40000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;02;1; 8.80000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;03;1; 8.30000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;04;1; 7.90000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;05;1; 7.30000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;06;1; 6.40000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;07;1; 6.90000e-09
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;08;1; 1.23000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;09;1; 1.41000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;10;1; 1.49000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;11;1; 1.30000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;12;1; 1.47000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;13;1; 1.28000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;14;1; 1.18000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;15;1; 1.22000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;16;1; 1.06000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;17;1; 1.26000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;18;1; 1.36000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;19;1; 1.59000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;20;1; 1.78000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;21;1; 2.19000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;22;1; 2.39000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;02;23;1; 2.45000e-08
GB0728A;51.544;  0.678;0037;pm10;2022;08;03;00;1; 2.24000e-08

all the data is there, there should be no reason why it gets filtered... I am tempted to re-run the long season evaluation to see if this is reproduced

charlienegri commented 1 year ago

a re-run reproduces the results,
the colocated data for those stations has the initial big gap where the data is _, and some other smaller gaps, but it's filled in for the rest of the period and especially the last part of the time series,
e.g. for GB1055R and pm10: ncks -v concpm10 -d station_name,700 /lustre/storeB/project/fou/kl/CAMS2_83/evaluation/coldata/cams2-83/forecast-last-seasons-map/EMEP/concpm10_concpm10_MOD-EMEP_REF-Obs_20200101_20231231_hourly_ALL-wMOUNTAINS.nc
so something must be happening in the statistics computation

charlienegri commented 1 year ago

the time series data for pm10 and United Kingdom (e.g. /lustre/storeB/project/fou/kl/CAMS2_83/evaluation/data/cams2-83/forecast-last-seasons-map/hm/ts/United_Kingdom-Obs-concpm10-Surface.json) has either time steps like

                            "1645099200000": {
                                "totnum": 35.0,
                                "num_valid": 0.0,
                                "refdata_mean": null,
                                "refdata_std": null,
                                "data_mean": null,
                                "data_std": null,
                                "weighted": 0.0,
                                "rms": null,
                                "nmb": null,
                                "mnmb": null,
                                "fge": null,
                                "R": null,
                                "R_spearman": null,
                                "num_coords_tot": 35.0,
                                "num_coords_with_data": 0.0
                            },

or like

                             "1677585600000": {
                                "totnum": 35.0,
                                "num_valid": 5.0,
                                "refdata_mean": 6.63583,
                                "refdata_std": 1.53003,
                                "data_mean": 6.46769,
                                "data_std": 2.90623,
                                "weighted": 0.0,
                                "rms": 2.45704,
                                "R": 0.53731,
                                "R_spearman": 0.3,
                                "R_kendall": 0.2,
                                "nmb": -0.02534,
                                "mnmb": -0.09934,
                                "fge": 0.30635,
                                "mb": -0.16815,
                                "mab": 1.97071,
                                "num_coords_tot": 35.0,
                                "num_coords_with_data": 5.0
                            }

which should be visualized as non null ... at the end of the period the time steps are all like this latter, so I have no idea why they do not show up in the web page here for eks https://cams2-83.aeroval.met.no/overall.php?project=cams2-83&exp_name=forecast-last-seasons&tab=timeseries&region=United+Kingdom&parameter=concpm10&mean_and_statistic=num_valid#
maybe @AugustinMortier has some ideas

AugustinMortier commented 1 year ago

The UK files are using United_Kingdom instead of United Kingdom.

The default behavior of the web interface is that, if the usual time series file (split by country: hm/ts/United Kingdom-Obs-conco3-Surface.json) is not found, then we are looking for the old format file, which was containing all countries (e.g: hm/ts/ALL-Obs-conco3-Surface.json). This file still exists in the last-seasons experiment, even though it is updated, so it is used.

So two things:

Use the country names as shown in the menus (do not replace blank spaces with underscores)
Clean up directories (hm/ts/ALL-Obs-conco3-Surface.json should not exist)

This issue was probably present for Bosnia and Herzegovina, Czech Republic, HOly See (Vatican City State) and San Marino

charlienegri commented 1 year ago

by "Use the country names as shown in the menus" you mean in pyaerocom? where?
/hm/ts/ALL* files are produced every time I run this, what do you mean they should not exist?

AugustinMortier commented 1 year ago

I actually wrote a comment about it last September: https://github.com/metno/pyaerocom/pull/707#issuecomment-1235585947

My bad for the hm/ts/ALL-Obs-conco3-Surface.json, this one should exist indeed. I was meaning hm/ts/Obs-conco3-Surface.json

charlienegri commented 1 year ago

ok, I do not know what to change to go from United_Kingdom to United Kingdom in the filenames, the files produced at the moment in /hm/ts/ are named

 -Obs-concno2-Surface.json                          Finland-Obs-concso2-Surface.json                         Malta-Obs-concso2-Surface.json
 -Obs-conco3-Surface.json                           France-Obs-concco-Surface.json                           Netherlands-Obs-concco-Surface.json
 -Obs-concpm10-Surface.json                         France-Obs-concno2-Surface.json                          Netherlands-Obs-concno2-Surface.json
 -Obs-concpm25-Surface.json                         France-Obs-conco3-Surface.json                           Netherlands-Obs-conco3-Surface.json
 -Obs-concso2-Surface.json                          France-Obs-concpm10-Surface.json                         Netherlands-Obs-concpm10-Surface.json
 ALL-Obs-concco-Surface.json                        France-Obs-concpm25-Surface.json                         Netherlands-Obs-concpm25-Surface.json
 ALL-Obs-concno2-Surface.json                       France-Obs-concso2-Surface.json                          Netherlands-Obs-concso2-Surface.json
 ALL-Obs-conco3-Surface.json                        Germany-Obs-concco-Surface.json                          Norway-Obs-concno2-Surface.json
 ALL-Obs-concpm10-Surface.json                      Germany-Obs-concno2-Surface.json                         Norway-Obs-conco3-Surface.json
 ALL-Obs-concpm25-Surface.json                      Germany-Obs-conco3-Surface.json                          Norway-Obs-concpm10-Surface.json
 ALL-Obs-concso2-Surface.json                       Germany-Obs-concpm10-Surface.json                        Norway-Obs-concpm25-Surface.json
 Albania-Obs-concco-Surface.json                    Germany-Obs-concpm25-Surface.json                        Poland-Obs-concco-Surface.json
 Albania-Obs-concno2-Surface.json                   Germany-Obs-concso2-Surface.json                         Poland-Obs-concno2-Surface.json
 Albania-Obs-conco3-Surface.json                    Gibraltar-Obs-concno2-Surface.json                       Poland-Obs-conco3-Surface.json
 Albania-Obs-concpm10-Surface.json                  Gibraltar-Obs-conco3-Surface.json                        Poland-Obs-concpm10-Surface.json
 Albania-Obs-concpm25-Surface.json                  Greece-Obs-concno2-Surface.json                          Poland-Obs-concpm25-Surface.json
 Albania-Obs-concso2-Surface.json                   Greece-Obs-conco3-Surface.json                           Poland-Obs-concso2-Surface.json
 Austria-Obs-concco-Surface.json                    Greece-Obs-concpm10-Surface.json                         Portugal-Obs-concco-Surface.json
 Austria-Obs-concno2-Surface.json                   Greece-Obs-concpm25-Surface.json                         Portugal-Obs-concno2-Surface.json
 Austria-Obs-conco3-Surface.json                    Greece-Obs-concso2-Surface.json                          Portugal-Obs-conco3-Surface.json
 Austria-Obs-concpm10-Surface.json                 'Holy_See_(Vatican_City_State)-Obs-conco3-Surface.json'   Portugal-Obs-concpm10-Surface.json
 Austria-Obs-concpm25-Surface.json                  Hungary-Obs-concco-Surface.json                          Portugal-Obs-concpm25-Surface.json
 Austria-Obs-concso2-Surface.json                   Hungary-Obs-concno2-Surface.json                         Portugal-Obs-concso2-Surface.json
 Belgium-Obs-concco-Surface.json                    Hungary-Obs-conco3-Surface.json                          Romania-Obs-concno2-Surface.json
 Belgium-Obs-concno2-Surface.json                   Hungary-Obs-concpm10-Surface.json                        Romania-Obs-conco3-Surface.json
 Belgium-Obs-conco3-Surface.json                    Hungary-Obs-concpm25-Surface.json                        San_Marino-Obs-conco3-Surface.json
 Belgium-Obs-concpm10-Surface.json                  Hungary-Obs-concso2-Surface.json                         Serbia-Obs-concco-Surface.json
 Belgium-Obs-concpm25-Surface.json                  Iceland-Obs-concno2-Surface.json                         Serbia-Obs-concno2-Surface.json
 Belgium-Obs-concso2-Surface.json                   Iceland-Obs-concpm10-Surface.json                        Serbia-Obs-conco3-Surface.json
 Bosnia_and_Herzegovina-Obs-concco-Surface.json     Iceland-Obs-concpm25-Surface.json                        Serbia-Obs-concpm10-Surface.json
 Bosnia_and_Herzegovina-Obs-concno2-Surface.json    Iceland-Obs-concso2-Surface.json                         Serbia-Obs-concpm25-Surface.json
 Bosnia_and_Herzegovina-Obs-conco3-Surface.json     Ireland-Obs-concco-Surface.json                          Serbia-Obs-concso2-Surface.json
 Bosnia_and_Herzegovina-Obs-concpm10-Surface.json   Ireland-Obs-concno2-Surface.json                         Slovakia-Obs-concco-Surface.json
 Bosnia_and_Herzegovina-Obs-concso2-Surface.json    Ireland-Obs-conco3-Surface.json                          Slovakia-Obs-concno2-Surface.json
 Bulgaria-Obs-concco-Surface.json                   Ireland-Obs-concpm10-Surface.json                        Slovakia-Obs-conco3-Surface.json
 Bulgaria-Obs-concno2-Surface.json                  Ireland-Obs-concpm25-Surface.json                        Slovakia-Obs-concpm10-Surface.json
 Bulgaria-Obs-conco3-Surface.json                   Ireland-Obs-concso2-Surface.json                         Slovakia-Obs-concpm25-Surface.json
 Bulgaria-Obs-concpm10-Surface.json                 Italy-Obs-concco-Surface.json                            Slovakia-Obs-concso2-Surface.json
 Bulgaria-Obs-concpm25-Surface.json                 Italy-Obs-concno2-Surface.json                           Slovenia-Obs-concco-Surface.json
 Bulgaria-Obs-concso2-Surface.json                  Italy-Obs-conco3-Surface.json                            Slovenia-Obs-concno2-Surface.json
 Croatia-Obs-concno2-Surface.json                   Italy-Obs-concpm10-Surface.json                          Slovenia-Obs-conco3-Surface.json
 Croatia-Obs-conco3-Surface.json                    Italy-Obs-concpm25-Surface.json                          Slovenia-Obs-concpm10-Surface.json
 Croatia-Obs-concpm10-Surface.json                  Italy-Obs-concso2-Surface.json                           Slovenia-Obs-concso2-Surface.json
 Croatia-Obs-concpm25-Surface.json                  Latvia-Obs-concno2-Surface.json                          Spain-Obs-concco-Surface.json
 Croatia-Obs-concso2-Surface.json                   Latvia-Obs-conco3-Surface.json                           Spain-Obs-concno2-Surface.json
 Cyprus-Obs-concco-Surface.json                     Latvia-Obs-concso2-Surface.json                          Spain-Obs-conco3-Surface.json
 Cyprus-Obs-conco3-Surface.json                     Lithuania-Obs-concco-Surface.json                        Spain-Obs-concpm10-Surface.json
 Cyprus-Obs-concso2-Surface.json                    Lithuania-Obs-concno2-Surface.json                       Spain-Obs-concpm25-Surface.json
 Czech_Republic-Obs-concno2-Surface.json            Lithuania-Obs-conco3-Surface.json                        Spain-Obs-concso2-Surface.json
 Czech_Republic-Obs-conco3-Surface.json             Lithuania-Obs-concpm10-Surface.json                      Sweden-Obs-concno2-Surface.json
 Czech_Republic-Obs-concpm10-Surface.json           Lithuania-Obs-concpm25-Surface.json                      Sweden-Obs-conco3-Surface.json
 Czech_Republic-Obs-concpm25-Surface.json           Lithuania-Obs-concso2-Surface.json                       Sweden-Obs-concpm10-Surface.json
 Czech_Republic-Obs-concso2-Surface.json            Luxembourg-Obs-concco-Surface.json                       Sweden-Obs-concpm25-Surface.json
 Denmark-Obs-concco-Surface.json                    Luxembourg-Obs-concno2-Surface.json                      Switzerland-Obs-concco-Surface.json
 Denmark-Obs-concno2-Surface.json                   Luxembourg-Obs-conco3-Surface.json                       Switzerland-Obs-concno2-Surface.json
 Denmark-Obs-conco3-Surface.json                    Luxembourg-Obs-concpm10-Surface.json                     Switzerland-Obs-conco3-Surface.json
 Denmark-Obs-concpm10-Surface.json                  Luxembourg-Obs-concpm25-Surface.json                     Switzerland-Obs-concpm10-Surface.json
 Estonia-Obs-concco-Surface.json                    Luxembourg-Obs-concso2-Surface.json                      Switzerland-Obs-concpm25-Surface.json
 Estonia-Obs-concno2-Surface.json                   Macedonia-Obs-concco-Surface.json                        Switzerland-Obs-concso2-Surface.json
 Estonia-Obs-conco3-Surface.json                    Macedonia-Obs-concno2-Surface.json                       United_Kingdom-Obs-concno2-Surface.json
 Estonia-Obs-concpm10-Surface.json                  Macedonia-Obs-conco3-Surface.json                        United_Kingdom-Obs-conco3-Surface.json
 Estonia-Obs-concpm25-Surface.json                  Macedonia-Obs-concpm10-Surface.json                      United_Kingdom-Obs-concpm10-Surface.json
 Estonia-Obs-concso2-Surface.json                   Macedonia-Obs-concso2-Surface.json                       United_Kingdom-Obs-concpm25-Surface.json
 Finland-Obs-concno2-Surface.json                   Malta-Obs-concco-Surface.json                            United_Kingdom-Obs-concso2-Surface.json
 Finland-Obs-conco3-Surface.json                    Malta-Obs-concno2-Surface.json
 Finland-Obs-concpm10-Surface.json                  Malta-Obs-conco3-Surface.json

charlienegri commented 1 year ago

update: we found the issue and it's filenames, the production branch will be fixed

charlienegri commented 1 year ago

fixed

metno / pyaerocom

CAMS2_83 PM10 timeseries UK missing stations #850