Fix issue with the fwx download / reprocess

bcgov / nr-rfc-climate-obs

Transition of the existiing climate observations data pipeline to enable running off prem

Apache License 2.0

1 stars 0 forks source link

Fix issue with the fwx download / reprocess #37

Open franTarkenton opened 1 year ago

franTarkenton commented 1 year ago

Spent a bunch of time getting the original R script to run as a container. Unfortunately this has not resulted in a working script. It is outputting some data, but not all the data.

Need to resolve why the data that is output from the reformat process is does not include all the fields.

franTarkenton commented 1 year ago

Going to pivot and get the data from the wildfire api:

https://bcwsapi.nrs.gov.bc.ca/wfwx-datamart-api/v1/openapi.json

franTarkenton commented 1 year ago

Noticed discrepancies between the data that is comming out of the api and the same data that you can access from the ftp/https site.

For example, this data set: https://www.for.gov.bc.ca/ftp/HPR/external/!publish/BCWS_DATA_MART/2023/2023-09-27.csv

Looking station code 11 and the timestamp 2023092612. from column 12 over has a bunch of data values, but have no idea where these are comming from. They do not align at all with the data that is comming from the api.

franTarkenton commented 1 year ago

Turn out the extra attributes are only populated for the 12th hour timestamps. After a bit more digging around have determined that this data contains the daily summarized information. Can correlate that with the /dailies api call. The following is from the api docs for the dailies end point:

"Resources available in the “Dailies” endpoint detail attributes related to Daily weather reports. Daily weather reports are derived from the noon hourly information, with a sum of precipitation over the last 24 hours. Dailies also include the full indices calculations (when available) and the danger class."

franTarkenton commented 1 year ago

Because the daily information is not used by the models, for now will pad the data file so that it includes blank values for the daily information. If need be we can add an daily api call to populate that data for the 12th hour timestamp records, but am not pursuing that atm.

franTarkenton commented 12 months ago

Downloads are now working. Remaining work:

mypy typing in the script
re-config for ostore integration, modify the main_fwx.py script to use the api for pulling local data, and ensure that pulls data from the ostore if the data exists there, and pushes after processing if it doesn't