[Data]: VJ214IMG from LAADS

[wildintellect]

Dataset Name

VJ214IMG

Dataset Description

The NOAA21 standard product (VJ214IMG) is preliminary/unpublished and only accessible by EarthData accounts with permission from the VIIRS science team.

Right now EIS Fire uses a 60 EDL Token issued to a single approved user.

• This token has to be renewed regularly
• The data is not in CMR so access via EarthAccess is not obvious

Requestor Name/Affiliation

Shane Coffield, EIS Fire GSFC @scoffiel

Platform/Method/Sensor

NOAA21 VIIRS

URL or DOI to Dataset Description

https://ncc.nesdis.noaa.gov/NOAA-21/index.php

URL to Download or Access the Data

https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/4014/VJ214IMG/

Data License

Yes

Intended Science Use Case

EIS Fire existing algorithms.

Format of the Data

No response

Approximate Size of the Data

No response

Date Needed By

No response

Additional Information

The team is looking for a sustainable solution for data access beyond manual token renewal from a single user's EDL account.

Some ideas:

1. DPS has federated tokens, not sure how that works with a scheduled job. In the MAAP ADE when you submit a job your current EDL token is forwarded to the job. Do federated tokens work with this endpoint?
2. DPS is implementing a new Secrets feature to allow caching secrets for use on jobs. This could be in lew of a netrc file.
   • We can make a DPS job that updates tokens via code every 59 days, and store that token in a secret?
3. We can ask LAADS about AWS role based access (not sure this is on S3 at all), or another method.

https://github.com/orgs/MAAP-Project/discussions/1031

cc: @mccabete

[scoffiel]

A bit of code for context. At this stage I'm just doing wget with an EDL token, since the data are only on the DAAC but not EarthData. I'm using Python earthaccess to get an EDL token by pointing to a stored netrc file with my username/password. If we set up an automated job to do this daily, I'd want it to keep working outside my ADE and not fail after 60 days.

auth = earthaccess.login()
session = auth.get_session()
token = session.headers['Authorization']

command = f'wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/4014/VJ214IMG/2024/180/" --header "Authorization: {token}" -P ~/my-public-bucket/viirs'

os.system(command)

Code notebook is at /projects/shared-buckets/coffield/scripts/viirs_L2_extraction.ipynb

[wildintellect]

Additional reference https://github.com/orgs/MAAP-Project/discussions/1031