NetCDF: Unknown file format: b'hyrax' error message

[btudor01]

Hi- when I try to download the subset data for the GEDI shots in my area of interest, I get the following error message. I'm fairly new to coding, and have not been able to resolve this issue. Any help would be much appreciated.

[rupesh2]

Hi @btudor01 , can you print the hyrax_url after line 14 in the code above? That will help us troubleshoot which particular beam/file has the issue.

hyrax_url = f"{g_name}.dap.nc4?dap4.ce=/{beam}/lon_lowestmode;/{beam}/lat_lowestmode"
print(hyrax_url)
ds = nc.Dataset('hyrax', memory=requests.get(hyrax_url).content)

[btudor01]

Hi, thanks for the quick response. Here is the result of running that code.

[rupesh2]

@btudor01 Can you download the netCDF file by going directly to the following URL? https://opendap.earthdata.nasa.gov/collections/C2237824918-ORNL_CLOUD/granules/GEDI_L4A_AGB_Density_V2_1.GEDI04_A_2019111192847_O02018_04_T04473_02_002_02_V002.h5.dap.nc4?dap4.ce=/BEAM0000/lon_lowestmode;/BEAM0000/lat_lowestmode

[btudor01]

Yes, that works.

[rupesh2]

Hi @btudor01, We noticed that the hyrax server sometimes arbitrarily throws HTTP 500 errors, which might have caused the issue. We have now added max_retries to the GET request. It now attempts to request for a maximum of 3 times if a hyrax request gives a 500 error. We have updated the tutorial to incorporate this change. Please let us know if this resolves the issue you were seeing.

[btudor01]

Hi @rupesh2. It looked as though it was working, and then threw the same OS error once it got to the 7th file.

[rupesh2]

Hi @btudor01 , can you try increasing the max_retries to a higher number, probably 5? retries = Retry(total=5, backoff_factor=0.1, status_forcelist=[ 500, 502, 503, 504 ])

[btudor01]

@rupesh2, I did this, and still hit the 7th file. Sorry, not sure what is going on!

[rupesh2]

@btudor01 can you share which file is the 7th file?

[btudor01]

https://opendap.earthdata.nasa.gov/collections/C2237824918-ORNL_CLOUD/granules/GEDI_L4A_AGB_Density_V2_1.GEDI04_A_2019123032136_O02194_01_T04588_02_002_02_V002.h5.dap.nc4?dap4.ce=/BEAM1011/lon_lowestmode;/BEAM1011/lat_lowestmode

This link takes me to a Hyrax- Bad Request 400 page

[rupesh2]

Hi @btudor01, I can now see that the beam BEAM1011 is missing in the file GEDI04_A_2019123032136_O02194_01_T04588_02_002_02_V002.h5. Few granules do not have all eight beams available. We have now updated the tutorial that should take care of this issue. Thank you so much for your help in debugging this issue.

[btudor01]

Awesome, thanks very much for doing that. These GEDI L4A tutorials have all been super useful for me, so thanks for the great work!