search

NASA-IMPACT / veda-data

2 stars 0 forks source link

Fire Weather Index Weather Stations #18

Open wildintellect opened 11 months ago

wildintellect commented 11 months ago

Contact Details

tmccabe@umd.edu

URL/DOI

No response

Data License

No response

Data Location

s3://veda-data-store-staging/EIS/other/station-FWI/20000101.20221021.hrlyInterp/

Size Estimate

No response

Number of Items

No response

Description

Daily and hourly station-based Fire Weather Index calculations over the Western US. The main source of station data is the NOAA NCEI Integrated Surface Database (ISD)

Collection Creation Notebook

No response

Item Creation Notebook

No response

Checklist

Any additional info you think is relevant, possibly including spatial or temporal subset if applicable?

The first of several Vector (Point) datasets in CSV format.

As determined in discussions on https://github.com/MAAP-Project/Community/issues/767

  1. Import into STAC (needs an ongoing pipeline that detects new uploads to the bucket)
  2. Expose s3 prefix as public and/or import to Fire features API https://firenrt.delta-backend.com/ 2a. If using the Features API, please support "format=csv" for the Matlab users of this data. Ideally the original filename can be used as a filter too.

To Do

smohiudd commented 10 months ago

@mccabete as discussed, we're going to ingest the HourlyFWIFromHourlyInterpContinuous data from s3://veda-data-store-staging/EIS/other/station-FWI/20000101.20221021.hrlyInterp/FWI/.

The existing csv files don't have any geospatial data relating to the station ID so I got that data from s3://veda-data-store-staging/EIS/other/station-FWI/20000101.NRT.nStns.1/isd-history.csv. Also I had to concatenate some columns to create a datetime column

The transformed csv files are saved in: s3://veda-data-store-staging/EIS/other/station-FWI/20000101.20221021.hrlyInterp/FWI-transformed/

Please have a look through a sample of those files and let me know if they are ok. Once I get the go ahead, we can start the ingestion process

smohiudd commented 10 months ago

Here is the transformation script for the FWI dataset that will loaded to the features API: https://github.com/NASA-IMPACT/veda-data/blob/main/transformation-scripts/fwi-stations/FWI_Transformation.ipynb

mccabete commented 10 months ago

Hey it looks good @smohiudd. Sorry, I realized belatedly that I did in fact know that the files didn't have lat and lon and that the datetimes are none standard -- and have had to solve those issues in my own work. So sorry about the lack of heads up.

smohiudd commented 10 months ago

@mccabete we have most of the data in the eis_fire_fwi: https://firenrt.delta-backend.com/collections/public.eis_fire_fwi Please try it out and let me know if you have any feedback.

Unfortunately the airflow ECS container timed out at 2 hours. I'll need to increase it to make sure all the data is ingested.

mccabete commented 10 months ago

@smohiudd This is great! I will let you know when I take it for a spin.