glamod / glamod-nuim

NUIM code in support of GLAMOD
BSD 3-Clause "New" or "Revised" License
0 stars 1 forks source link

Fix minutes in Observation ID [Subdaily CDM Lite] #8

Closed rjhd2 closed 2 years ago

rjhd2 commented 2 years ago

Input files have off-hour ovservations in the QFF:

FRI0000LFRI|LES AJONCS|2021|04|01|21|30|46.7019|-1.3786|91.1|15.0|223||ICAOLFRI-8|9.0|223||ICAOLFRI-8|||||||||50.0|223||ICAOLFRI-8|3.6|223||ICAOLFRI-8

But output CDM lite files only have on-hour values (no later values than 21:00 for this station on 2021-04-01 in the psv file):

FRI0000LFRI-2-2021-04-01-21:00-106-12|0|2021-04-01 21:00:00+00|1|46.702|-1.379|10|106|320|50|12|0||1|FRI0000LFRI|LES AJONCS|0|1|223

Uncomment in setting of master df

sjnoone commented 2 years ago

Ah good catch ,I just set the minutes to 0 didn’t realise we had off hour obs. Will need to write and if else line of code for minutes .

Get Outlook for iOShttps://aka.ms/o0ukef


From: Robert Dunn @.> Sent: Monday, January 31, 2022 3:38:18 PM To: glamod/glamod-nuim @.> Cc: Simon Noone @.>; Assign @.> Subject: [EXTERNAL] [glamod/glamod-nuim] Fix minutes in Observation ID [Subdaily CDM Lite] (Issue #8)

Warning

This email originated from outside of Maynooth University's Mail System. Do not reply, click links or open attachments unless you recognise the sender and know the content is safe.

Input files have off-hour ovservations in the QFF:

FRI0000LFRI|LES AJONCS|2021|04|01|21|30|46.7019|-1.3786|91.1|15.0|223||ICAOLFRI-8|9.0|223||ICAOLFRI-8|||||||||50.0|223||ICAOLFRI-8|3.6|223||ICAOLFRI-8

But output CDM lite files only have on-hour values (no later values than 21:00 for this station on 2021-04-01 in the psv file):

FRI0000LFRI-2-2021-04-01-21:00-106-12|0|2021-04-01 21:00:00+00|1|46.702|-1.379|10|106|320|50|12|0||1|FRI0000LFRI|LES AJONCS|0|1|223

Uncomment in setting of master df

— Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fglamod%2Fglamod-nuim%2Fissues%2F8&data=04%7C01%7CSimon.Noone%40mu.ie%7C2a97b8d4476b45444df608d9e4cfb52b%7C1454f5ccbb354685bbd98621fd8055c9%7C0%7C0%7C637792403017072157%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2BZXV6ZLblAiMzSxeYgksvX%2BK3Firk6og9voD9%2BVVpk4%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAG3VGPPMOGBWLDJZ7DDDVUDUY2UGVANCNFSM5NGS4H2Q&data=04%7C01%7CSimon.Noone%40mu.ie%7C2a97b8d4476b45444df608d9e4cfb52b%7C1454f5ccbb354685bbd98621fd8055c9%7C0%7C0%7C637792403017072157%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=og%2BieMZeKgPqvdX4eIrSFOXQ9zehaq2R1hi63fmUOtY%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7CSimon.Noone%40mu.ie%7C2a97b8d4476b45444df608d9e4cfb52b%7C1454f5ccbb354685bbd98621fd8055c9%7C0%7C0%7C637792403017072157%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=MV3qWwd5LiAirWteYJHti1ZKNhNeyIsbfU%2BmtGxGg5U%3D&reserved=0 or Androidhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7CSimon.Noone%40mu.ie%7C2a97b8d4476b45444df608d9e4cfb52b%7C1454f5ccbb354685bbd98621fd8055c9%7C0%7C0%7C637792403017072157%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=cyqEAF%2BgjZgsr8EIU07UESsYrnNmiAxvT1eqTAVKJm0%3D&reserved=0. You are receiving this because you were assigned.Message ID: @.***>

sjnoone commented 2 years ago

I think this edit will fix this bug in the cdm_lite and observations code existing df["Minute"]="00" edit df["Minute"]=df["Minute"]

rjhd2 commented 2 years ago

Could we just remove the line df["Minute"]="00"? I'm not sure what your edit achieves?

sjnoone commented 2 years ago

Your right actually Minutes is already in the DF from teh QFf so we can just remove it

rjhd2 commented 2 years ago

Is the report_type also needing to be carried through (the GWS files have it but this code currently overwrites with 0)? If so, I'll amend the issue title and fix both in one go.

sjnoone commented 2 years ago

There is a rule in the recipe tables for the report type, I forgot about coding in for ICAO stations. [column 6]

Int report_type

Enter: 0=sub-daily/hourly data except for ICAO platforms 3=Daily data 2=monthly data 4= ICAO sub-daily data

Dr. Simon Noone


Senior Postdoctoral Researcher

https://orcid.org/0000-0003-1661-1423

https://www.maynoothuniversity.ie/icarus/our-people/simon-noone

The Copernicus Climate Change Service C3S2_311 - Collection and Processing of In Situ Observations. Lot 1 WP2-Lead Access to a comprehensive archive of historical surface observations, with support for data rescue.

@.**@.

Associated expert for the WMO Commission for Climatology Working Group on Climate Data Exchange (WG-CDE)

ICARUS - Irish Climate Research Centre Room 1.8,Laraghbryan House,

North campus, Maynooth University,

Maynooth, Kildare

Ireland

01-7086392 @. @.


From: Robert Dunn @.> Sent: Thursday, February 3, 2022 4:40:09 PM To: glamod/glamod-nuim @.> Cc: Simon Noone @.>; Assign @.> Subject: [EXTERNAL] Re: [glamod/glamod-nuim] Fix minutes in Observation ID [Subdaily CDM Lite] (Issue #8)

Warning

This email originated from outside of Maynooth University's Mail System. Do not reply, click links or open attachments unless you recognise the sender and know the content is safe.

Is the report_type also needing to be carried through (the GWS files have it but this code currently overwrites with 0)? If so, I'll amend the issue title and fix both in one go.

— Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fglamod%2Fglamod-nuim%2Fissues%2F8%23issuecomment-1029178887&data=04%7C01%7CSimon.Noone%40mu.ie%7Cbd724db0b7ae423436ce08d9e733d84c%7C1454f5ccbb354685bbd98621fd8055c9%7C0%7C0%7C637795032124942124%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ADkvxtQHMygsy5WOAIbI3qA2FX%2BgHEufz4GHSYgovWI%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAG3VGPKZK24P6VUW7RHGBI3UZKVWTANCNFSM5NGS4H2Q&data=04%7C01%7CSimon.Noone%40mu.ie%7Cbd724db0b7ae423436ce08d9e733d84c%7C1454f5ccbb354685bbd98621fd8055c9%7C0%7C0%7C637795032124942124%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=o6kk%2BMIwm1EE2MRAtP%2BKEN9qes7QAiPMll4SMP0oaeo%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7CSimon.Noone%40mu.ie%7Cbd724db0b7ae423436ce08d9e733d84c%7C1454f5ccbb354685bbd98621fd8055c9%7C0%7C0%7C637795032124942124%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZTTHSwVEkPKQFJ54UqEasb5WVbH1BVE5etirzbjWmoQ%3D&reserved=0 or Androidhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7CSimon.Noone%40mu.ie%7Cbd724db0b7ae423436ce08d9e733d84c%7C1454f5ccbb354685bbd98621fd8055c9%7C0%7C0%7C637795032124942124%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=brkWG1y7ViTTeuQvNW%2Ftb9jxAYo2Q7HQAE97lYHoIjs%3D&reserved=0. You are receiving this because you were assigned.Message ID: @.***>