Refactor NOAA20 and SUOMI Downloads Should Use FIRMS API with MAP_KEY

[ranchodeluxe]

Problem

We don't want to continue refreshing tokens with the existing ED downloads. FIRMS has an API we can use with non-expiring tokens

Solution (from Otmar's emails)

You can use our API and avoid dealing with the token. You just need to put your email and it will send you the key. https://firms.modaps.eosdis.nasa.gov/api/map_key/

Once you have it, use: https://firms.modaps.eosdis.nasa.gov/api/area/

replace MAP_KEY with the one sent to you and then you can query the WORLD for TODAY as: https://firms.modaps.eosdis.nasa.gov/api/area/csv/MAP_KEY/VIIRS_NOAA20_NRT/world/1 https://firms.modaps.eosdis.nasa.gov/api/area/csv/MAP_KEY/VIIRS_NOAA21_NRT/world/1 Note: no recent data for SNPP right now https://firms.modaps.eosdis.nasa.gov/api/area/csv/MAP_KEY/VIIRS_SNPP_NRT/world/1

If you need the last 2 days: https://firms.modaps.eosdis.nasa.gov/api/area/csv/MAP_KEY/VIIRS_NOAA20_NRT/world/2

By default nasa.gov emails get 100,000 transactions / 10-minute window which is more than enough for most ingest scripts. (most barely use 1000). There is not expiration on our MAP_KEY and the implementation is way simpler.

However, if it is more convenient to use EarthData token, then do contact them. I think that 2-month expiration period is horrible for anyone downloading data regularly, but I guess there was a reason why free public data has to have that level of protection.

Let me know if you need any help or additional instructions.

AC

0. walk through the examples above to get an understanding
1. NRT data has it's own 7-day window to download and uses source "VIIRS_NOAA20_NRT" || "VIIRS_SNPP_NRT": https://firms.modaps.eosdis.nasa.gov/api/area/
2. archival data will the s3-hosted products on NETCDF and LPDAC (VNP14IMG, VJ114IMG) (according to Shane)
3. check for any schema changes or things that might cause problems
4. refactor existing

[ranchodeluxe]

we all agree we can do this after the merge

[mccabete]

@mccabete re-review post Shane debrief

[zebbecker]

Part of this refactor should include adding logic to gracefully handle failures from one satellite

[zebbecker]

If we switch to FIRMS, will we need to go back to LANCE to incorporate Shane's candidate pixel work if we end up deciding that we want candidate pixels as part of our main workflow? Or, does the FIRMS API also include candidate pixels, too? Is that confidence field in the example below? Adding as a reminder to investigate.

FIRMS API example response (from their tutorial here):

[mccabete]

The candidate pixel work will not be on LANCE or LAADS .txt files. Shane is/was the only one extracting them from the swaths to my knowledge. The "Standard Product" (not NRT) has processed candidate info, but I think that info would need to come from the swaths, not the .txt files. See here.

Yes, I believe that to get the swaths, we would need to be working over Earthdata. SNPP and NOAA-20 can be accessed over s3 but as of Shane leaving NOAA-21 involved downloading and reprocessing. Personally, I think of our ingest needs as two tracks: ".txt file based" and "swath-based".

".txt file based"

• What we are doing now. .txt files come from a server. Right now in NRT, but could come faster in "real time" or "ultra-real-time" for different places
• "Science quality" .txt files aren't being generated on a regular cadence.

"swath based"

• Could be used to generate our own "science quality" .txt files when not available.
• Could be used to extract candidate info.