bcgov / nr-rfc-climate-obs

Transition of the existiing climate observations data pipeline to enable running off prem
Apache License 2.0
1 stars 0 forks source link

Aquire ECCC Hourly Date #43

Open franTarkenton opened 1 year ago

franTarkenton commented 1 year ago

Create a script that will pull the following information on an hourly basis.

Source of data: https://hpfx.collab.science.gc.ca/20231101/WXO-DD/observations/swob-ml/20231101/

Data Aquisition

Processing

Secondary:

KYSIEMENS commented 1 year ago

Script is mostly complete. Hourly XML files for stations in station list are being downloaded, processed and saved to a dataframe, which is then saved as a parquet file in object store. Daily temperature and precipitation are generated and saved to object store. 'air_temp' variable used instead of 'avg_air_temp_pst1hr' as the latter was missing for many stations.

To do: