search

Earth-Information-System / fireatlas

4 stars 2 forks source link

CONUS NRT V3 Schedule Jobs are Failing #111

Open ranchodeluxe opened 1 month ago

ranchodeluxe commented 1 month ago

Problem

CONUS NRT V3 scheduled jobs are failing.

GH Actions showing which job ids are failing and here's are the docs to walk through to debug

Action Items

mccabete commented 1 month ago

Love the write up!

seems like they are failing because pre-processed timesteps are missing. ie job 412d9f87-0ec3-438a-9315-fe5e45276a33

"FileNotFoundError('maap-ops-workspace/shared/gsfc_landslides/FEDSpreprocessed/NOAA20/20240323_AM.txt')

it looks like we do have a gap in our daily files at doy 83 - skipping from J1_VIIRS_C2_Global_VJ114IMGTDL_NRT_2024079.txt to J1_VIIRS_C2_Global_VJ114IMGTDL_NRT_2024085.txt

mccabete commented 1 month ago

As far as I can tell, the "daily" files are all released through the place we are querying from -- I can only see them going back a certain about of time, to June 3rd. https://nrt4.modaps.eosdis.nasa.gov/api/v2/content/archives/FIRMS/noaa-20-viirs-c2/Global/

Seems like they only distribute the last 60 days worth.

mccabete commented 1 month ago

The firms API seems to only be the last 10 days.

mccabete commented 1 month ago

Looks like it wasn't a data outage: https://ladsweb.modaps.eosdis.nasa.gov/alerts-and-issues/

mccabete commented 1 month ago

For a fix: I don't know. It seems like either we need to structure the code to be ok with "skipping" a few days OR we need to figure out how to derive our own .txt files from the data that we can get more reliably. Or both.

ranchodeluxe commented 1 month ago

For a fix: I don't know. It seems like either we need to structure the code to be ok with "skipping" a few days OR we need to figure out how to derive our own .txt files from the data that we can get more reliably. Or both.

Maybe it's acceptable for now (and 2024) to start our NRT runs on 2024-03-25 instead of 2024-01-01?

Then we'll be forced to deal with this during archival runs but that would be another data source

mccabete commented 1 month ago

Yeah as a temporary fix it's not the worst. We may run into other data holes, but I guess we can cross that bridge if we come to it.

mccabete commented 1 month ago

If this happens later in the season though (may onward) it will probably be chopping up real fires though.

mccabete commented 1 month ago

@ranchodeluxe is there any chance that V3 restarts every time there is a data gap?