GEDI Subsetter 403 errors

[wildintellect]

Describe the bug GEDI Subsetter is failing to access L4A data from ORNL with 403 forbidden. Reported by user @pbasak

│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │    item = (                                                              │ │
│ │           │   False,                                                     │ │
│ │           │   HTTPError('403 Client Error: FORBIDDEN for url:            │ │
│ │           https://api.maap-project.org/api/members/self/awsAccess/edcCr… │ │
│ │           )                                                              │ │
│ │    self = <multiprocessing.pool.IMapUnorderedIterator object at          │ │
│ │           0x7fda7b8009d0>                                                │ │
│ │ success = False                                                          │ │
│ │ timeout = None                                                           │ │
│ │   value = HTTPError('403 Client Error: FORBIDDEN for url:                │ │
│ │           https://api.maap-project.org/api/members/self/awsAccess/edcCr… │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
HTTPError: 403 Client Error: FORBIDDEN for url: 
https://api.maap-project.org/api/members/self/awsAccess/edcCredentials/https%253
A%252F%252Fdata.ornldaac.earthdata.nasa.gov%252Fs3credentials
ERROR conda.cli.main_run:execute(47): `conda run /app/gedi-subsetter/src/gedi_subset/subset.py --verbose --aoi input/Caqueta_moist_forests.gpkg --doi L4A --lat lat_lowestmode --lon lon_lowestmode --beams all --columns agbd, agbd_se, sensitivity, geolocation/sensitivity_a2 --query l2_quality_flag == 1 and l4_quality_flag == 1 and sensitivity > 0.95 and `geolocation/sensitivity_a2` > 0.95 --limit 0 --output L4A_Caqueta_moist_forests.gpkg` failed. (See above for error)
+ cp _stderr.txt _alt_traceback.txt

To Reproduce Run a GEDI subsetter job

list_aois= gpkg_files
list_outname = [(‘L4A_’ + os.path.basename(x)) for x in list_aois]
list_jobIDs = [“”]*len(list_aois)
count=-1
for aoi in list_aois:
    count +=1
    output_name = list_outname[count]
    try:
        result = maap.submitJob(
            identifier=“gedi-subset”,
            algo_id=“gedi-subset”,
            version=“0.6.1",
            queue=“maap-dps-worker-32gb”,
            username=“pbasak”,
            aoi=aoi,
            doi=“L4A”,
            lat=“lat_lowestmode”,
            lon=“lon_lowestmode”,
            beams=“all”,
            columns=“agbd, agbd_se, sensitivity, geolocation/sensitivity_a2”,
            query=“l2_quality_flag == 1 and l4_quality_flag == 1 and sensitivity > 0.95 and geolocation/sensitivity_a2 > 0.95",
            limit=0,
            output = output_name,
            temporal = “-”
        )
        job_id = result[“job_id”]
        list_jobIDs[count] = job_id
    except:
        list_jobIDs[count] = ‘None’
print(list_jobIDs)

example aoi https://maap-ops-workspace.s3.amazonaws.com/shared/pbasak/Peru/Ecoregions/Bolivian_Yungas.gpkg

Additional context See also recent issue

• 874

[wildintellect]

I've checked the DOI has not changed, the versions being requested are accurate.

[wildintellect]

@jjfrench found at least one major cause, we're releasing a new version of GEDI Subsetter 0.6.2 that updates the maap-py version. In testing this is working to solve MAAP API timeout errors. We haven't hit the 403 error yet, so we'll need user help to find some jobs that reliably trigger that issue. See more https://github.com/MAAP-Project/gedi-subsetter/pull/55

New version should be registered in the MAAP ADE soon.

[jjfrench]

Running a build here.

[jjfrench]

Interesting - I ran

inputs = dict(
    aoi="https://maap-ops-workspace.s3.amazonaws.com/shared/pbasak/Peru/Ecoregions/Central_Andean_puna.gpkg",
    doi="L4A",
    lat="lat_lowestmode",
    lon="lon_lowestmode",
    beams="all",
    columns="agbd, agbd_se, sensitivity, geolocation/sensitivity_a2",
    query="l2_quality_flag == 1 and l4_quality_flag == 1 and sensitivity > 0.95 and `geolocation/sensitivity_a2` > 0.95",
    limit=0,
    temporal="-",
    output="gedi_subset"
)

Locally in the ADE, worked just fine

Most recent run in the DPS job returns 500 though 🤔 Different error code though, so progress?

[jjfrench]

Just ran it again... no error. Intermittent erroring out, that doesn't sound fun.

[jjfrench]

Closing - it looks as if this was the majority fix. Will open another ticket if we continue to encounter 500's for a different related issue.