Query IOOS CKAN API for specific free text variable

[MathewBiddle]

• [X] What is language(s) for used in the example?
  • Python
• [X] Is it focused on a particular module/software or an IOOS data service?
  • IOOS Catalog API
• [X] Can you provide a minimum example of the expected code and results in a notebook?
  • Something like this notebook but with free text search capability.

Please provide a detailed description of the suggested example below:

Original request from Hassan: I proposed DO as an ecosystem indicator to be added to the list of indicators identified by the Ecosystem Indicator Working Group (here https://ecowatch.noaa.gov/).

I would like to query DO datasets from our portal and linked to the Ecosystem Indicator portal. I did a simple query for DO and I found 1,116 datasets for "Dissolved Oxygen". Now I would like to look at the coverage of these datasets. Is there a way to visualize these datasets and download all of it in a CSV file or the only way is to go to dataset/dataset.

From @mwengren I think it would be straightforward if all you wanted to do was to see locations of the datasets on a map - all this info is already there in the Catalog database. If you wanted to be more precise than a free text search, you can actually filter by CF standard name, for example 'mass_concentration_of_oxygen_in_sea_water' (assuming that's the correct standard name), 556 results: https://data.ioos.us/dataset?q=&cf_standard_names=mass_concentration_of_oxygen_in_sea_water

And, you can get the same results via the API:

https://data.ioos.us/api/3/action/package_search?fq=cf_standard_names:mass_concentration_of_oxygen_in_sea_water With some Python code, it would just take parsing the API results to obtain bounding boxes, and then plotting on a map.

Retrieving DO data itself is possible, but you would have to go to the source service for each. If you wanted a consistent API, you can also filter in Catalog by service type, for example 'ERDDAP-tabledap' to allow consistent retrieval via erddapy in Python for example. Dataset count now 489: https://data.ioos.us/dataset?q=&cf_standard_names=mass_concentration_of_oxygen_in_sea_water&res_format=ERDDAP-TableDAP

API query: https://data.ioos.us/api/3/action/package_search?fq=cf_standard_names:mass_concentration_of_oxygen_in_sea_water+res_format=ERDDAP-TableDAP

[mwengren]

@benjwadams At one point we were going to work on some IOOS Catalog demonstration notebooks. Do you have any notebooks you started that used ckanapi to query Catalog? Hoping we could use those as a starting point for this.

Basic notebook workflow would be:

1. User selects CF standard name from pre-populated pulldown menu
2. User optionally selects data format type from pre-populated pulldown menu
3. ckanapi queries Catalog API, iterates results, and generates a Pandas (Geopandas?) dataframe with relevant metadata (dataset id, dataset Catalog URL, dataset bounding box, etc)
4. Python plotting library like hvPlot used to plot geographic distribution of results

Optionally, extend by using erddapy to retrieve recent data values (assuming originating from 'ERDDDAP-tabledap' data service/format) to plot in popup window or secondary plot.

[ocefpaf]

Just want to xref. this one: https://github.com/ioos/catalog-ckan/issues/231

[mwengren]

Adding links to a few WIP notebooks that query CKAN API to plot dataset coverage by CF Standard Name, Organization, and Data Format:

https://github.com/mwengren/notebooks-dev/tree/master/ioos_ckan_catalog

These notebooks could either be added to the gallery with some polishing up, or adapted for other CKAN API demos/use cases.

[MathewBiddle]

FWIW here is my notebook using the sensormap erddap to do a similar search: https://github.com/MathewBiddle/sandbox/blob/main/notebooks/DRAFT_sensor_map_query.ipynb

I think the CKAN query is capturing more of what we're after. And it gives a different way to do it.

[MathewBiddle]

There's a couple related notebooks:

• https://ioos.github.io/notebooks_demos/notebooks/2016-12-19-exploring_csw/
• https://ioos.github.io/notebooks_demos/notebooks/2017-12-15-finding_HFRadar_currents/
• http://ioos.github.io/notebooks_demos/notebooks/2016-12-22-boston_light_swim/

[MathewBiddle]

@mwengren, I think you have a solid example to put in the IOOS CodeLab. Mind adding a PR for this in https://github.com/ioos/ioos_code_lab

Then we can close this ticket and comment on the PR.

[MathewBiddle]

moved to code lab. closing.