Should we change our VIIRS observation workflow to read directly from cloud-based earthdata?

[mccabete]

I haven't yet investigated this in detail, but I believe that VIIRS observations are available on the cloud through Earthdata. We may be able to read directly from there are some part of the algorithm, instead of automatically downloading the observations into a bucket.

Possible pros?

• Less storage
• don't have to be constantly downloading stuff
• could preferentially look for some data if it exists

Possible cons?

• less portable to non-cloudbased systems.
• I honestly don't know what's available on earthdata
• do we need the data locally anyway for I/O reasons?

[scoffiel]

I've been thinking about this more and would also suggest it as a solution, at least for the archive data - that is, stream in the netcdf swaths directly via EarthData cloud (VNP14IMG and VJ114IMG). They are reasonably small (1-3 MB per file, 241 files per day) and have the columns we need easily accessible.

I think we should (1) check with FIRMS about best pipelines for us to consistently access the archive L2 text files, but also (2) propose just generating those files ourselves. I can easily write that script. There would be much less delay getting the high quality archive data.

[mccabete]

The discussion for how to enable systematic access for a script described above is here: https://github.com/orgs/MAAP-Project/discussions/1031