search

Earth-Information-System / fireatlas

3 stars 1 forks source link

VIIRS Ingest pipeline not robust against known data errors #53

Open mccabete opened 1 month ago

mccabete commented 1 month ago

Our data pipeline automatically will pull in VIIRS data if it exists. Sometimes the data is available but has known bugs, or was ingested before those bugs were caught. This has looked like anomalies in the data "streaks" (see picture), and possibly like known geolocation issues in the data. (I haven't checked if the data for May 25th- June 5th were taken down or just provided with a warning. )

Screenshot 2023-10-02 at 9 44 08 AM (2)

We don't have a workflow to automatically go fix data gaps, even when we know better data has come back online. This is related to #48.

scoffiel commented 1 week ago

This also relates to issue #62 - it would be nice to replace these cases with the higher science quality archive version as soon as available (a day or 2). Might also be worth checking if filtering out low confidence pixels can alleviate these stripes.

Not super helpful in this case, but here is where general known VIIRS issues get posted https://landweb.modaps.eosdis.nasa.gov/knownissue?sensor=VIIRS

And info on C2 vs C1 updates https://landweb.modaps.eosdis.nasa.gov/data/userguide/VIIRS_Land_C2_Changes_09152022.pdf