search

Earth-Information-System / fireatlas

3 stars 1 forks source link

Regularly update monthly data files on s3 #49

Open jsignell opened 1 month ago

jsignell commented 1 month ago

The NRT files are regularly updated as part of the scheduled DPS jobs. It would be great to do the same for the monthly files. According to @eorland this is the current manual process:

The monthly data comes from a sftp site hosted by the university of Maryland. All information for access is available in this guide on page 16: https://lpdaac.usgs.gov/documents/427/VNP14_User_Guide_V1.pdf.

Users can download the VNP14IMGML and VNP14ML data sets by connecting to an ftp server at the University of Maryland using: server: fuoco.geog.umd.edu login: fire password: burnt directory (VNP14IMGML): /VIIRS/VNP14IMGML directory (VNP14ML): /VIIRS/VNP14ML

It would be interesting to try to use the fsspec sftp file system to manage this download process.

mccabete commented 16 hours ago

@scoffiel thought through some of the issues with this. (see here)

Seems like the"ML" files use a .txt files that are generated by a group at UMD on a sporratic basis (and only for SNPP). They are also sporatically generated by FIRMS. They are systematically generated for the NRT files.

@Scoffiel suggested that the .txt would be easy fro us to generate ourselves. This MAAP discussion: https://github.com/orgs/MAAP-Project/discussions/1031

is a request for access that would facilitate generating those files.