search

ua-snap / data-api

SNAP data API
MIT License
4 stars 0 forks source link

Modify the `get_dim_encodings` function to account for differences in the default DescribeCoverage request + response in Rasdaman. #481

Open charparr opened 2 weeks ago

charparr commented 2 weeks ago

This issue is motivated by a bug encountered while developing new API endpoints. The STR to reproduce this bug are to set your the Rasdaman env var to:

export API_RAS_BASE_URL=https://datacubes.earthmaps.io/rasdaman/

and the observe this error which (a red herring, as the coverages listed as missing do in fact exist):

Warning: Coverage 'crrel_gipl_outputs' is missing from the Rasdaman server https://datacubes.earthmaps.io/rasdaman/ you are using.
Warning: Coverage 'alfresco_relative_flammability_30yr' is missing from the Rasdaman server https://datacubes.earthmaps.io/rasdaman/ you are using.
Warning: Coverage 'alfresco_vegetation_type_percentage' is missing from the Rasdaman server https://datacubes.earthmaps.io/rasdaman/ you are using.
Warning: Coverage 'heating_degree_days_Fdays' is missing from the Rasdaman server https://datacubes.earthmaps.io/rasdaman/ you are using.
Traceback (most recent call last):
  File "/Users/cparr/.local/share/virtualenvs/data-api-_zt2_2sd/bin/flask", line 8, in <module>
    sys.exit(main())
  File "/Users/cparr/.local/share/virtualenvs/data-api-_zt2_2sd/lib/python3.9/site-packages/flask/cli.py", line 1064, in main
    cli.main()
  File "/Users/cparr/.local/share/virtualenvs/data-api-_zt2_2sd/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/cparr/.local/share/virtualenvs/data-api-_zt2_2sd/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/cparr/.local/share/virtualenvs/data-api-_zt2_2sd/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/cparr/.local/share/virtualenvs/data-api-_zt2_2sd/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/cparr/.local/share/virtualenvs/data-api-_zt2_2sd/lib/python3.9/site-packages/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/Users/cparr/.local/share/virtualenvs/data-api-_zt2_2sd/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/cparr/.local/share/virtualenvs/data-api-_zt2_2sd/lib/python3.9/site-packages/flask/cli.py", line 912, in run_command
    raise e from None
  File "/Users/cparr/.local/share/virtualenvs/data-api-_zt2_2sd/lib/python3.9/site-packages/flask/cli.py", line 898, in run_command
    app = info.load_app()
  File "/Users/cparr/.local/share/virtualenvs/data-api-_zt2_2sd/lib/python3.9/site-packages/flask/cli.py", line 309, in load_app
    app = locate_app(import_name, name)
  File "/Users/cparr/.local/share/virtualenvs/data-api-_zt2_2sd/lib/python3.9/site-packages/flask/cli.py", line 219, in locate_app
    __import__(module_name)
  File "/Users/cparr/workspace/repos/data-api/application.py", line 6, in <module>
    from routes import *
  File "/Users/cparr/workspace/repos/data-api/routes/__init__.py", line 29, in <module>
    from .degree_days import *
  File "/Users/cparr/workspace/repos/data-api/routes/degree_days.py", line 28, in <module>
    dd_dim_encodings["scenario"][0] = "modeled_baseline"
TypeError: 'NoneType' object is not subscriptable

This is a red herring error message because those coverages that are getting flagged via the warnings as “missing” are definitely present on Zeus / Datacubes. And the exact same error occurs if there is a nonsense URL like `export API_RAS_BASE_URL=https://datacubes.earthmaps.io/rasdaman/

@Joshdpaul tracked this down to the fetch_data.get_dim_encodings() function, which is returning None after printing the warning and suggests there is an error in scraping the XML metadata in that fetch_data.get_dim_encodings() function.

I think this analysis is correct because there are substantial differences in what might get depending on which Rasdaman you point at and how the DescribeCoverage request is constructed.

Consider this request:

https://datacubes.earthmaps.io/rasdaman/ows?&SERVICE=WCS&VERSION=2.1.0&REQUEST=DescribeCoverage&COVERAGEID=air_freezing_index_Fdays

vs. this one

https://datacubes.earthmaps.io/rasdaman/ows?&SERVICE=WCS&VERSION=2.1.0&REQUEST=DescribeCoverage&COVERAGEID=air_freezing_index_Fdays&outputType=GeneralGridCoverage

note that &outputType=GeneralGridCoverage is only available in WCS 2.1.0 at this is the default DescribeCoverage implementation in Petascope

vs. this one

https://apollo.snap.uaf.edu/rasdaman/ows?&SERVICE=WCS&VERSION=2.0.1&REQUEST=DescribeCoverage&COVERAGEID=air_freezing_index_Fdays

The responses are all slightly different.

Additionally, some coverages return a structured XML like this:

https://datacubes.earthmaps.io/rasdaman/ows?&SERVICE=WCS&VERSION=2.1.0&REQUEST=DescribeCoverage&COVERAGEID=ardac_beaufort_daily_slie&outputType=GeneralGridCoverage

Given this variance, it is maybe not surprising that our get_dim_encodings workhorse function is tripping up. My recommendation is to revist how we construct our DescribeCoverage requests and to perhaps give them their own more atomic functions within the generate_requests and generate_urls modules.