notebook focusing on models in the Pioneer Array area

[emiliom]

@ocefpaf I'm following up via an issue here (my ohw2018 tutorials fork); hopefully this is easier to track than long email threads.

Have you been able to make progress on a notebook focusing on models in the Pioneer Array area? If not, do you think you'll have time? I'm looking for a realistic answer, not the hopeful and overly optimistic version :wink:

For reference, here's what I wrote a couple of weeks ago about this:

Filipe's original Endurance Array notebook had both obs and model output. I'm trying to keep each notebook to be not so big or so complex (ie, many different topics and capabilities involved). So, I prefer breaking it up, to have one in the Endurance Array doing obs only, and one in the Pioneer Array doing models only. Mixing obs and output is great, of course, but we can leave it for participants to do on their own and point them to other IOOS notebooks.

[emiliom]

Hi @ocefpaf -- just checking in again about prospects for this notebook ...

[ocefpaf]

I'm trying to keep each notebook to be not so big or so complex

I am biased b/c I wrote it but I don't think it was big and complex. Actually it makes sense IMO to show what the catalog returns to you when querying it. How are you filtering out the models BTW?

(ie, many different topics and capabilities involved

Catalog wise it is the same topic and capability, no? Only plotting that may bring extra elements but we don't need to plot, just show the URLs returned. (Although I do like to show at least the model grid.)

Filipe's original Endurance Array notebook had both obs and model output.

Did you run that notebook? Are you getting the models there? My guess is that if you did not maybe we have an issue with the catalog or the data providers.

[ocefpaf]

OK. I went ahead and put this one together, it is basically a copy-n-paste from the missing parts of the original into your new one, but without the station fetching, plotting, etc.

http://nbviewer.jupyter.org/gist/ocefpaf/36ab048b94859e8bf9a070db95fed174

Ping @mwengren and @rsignell-usgs for awareness of the problem we are facing on cells [10]-[11]. We don't have a reliable way to filter using either scheme or geolinks yet!

[emiliom]

Thanks @ocefpaf.

Did you run that notebook? Are you getting the models there? My guess is that if you did not maybe we have an issue with the catalog or the data providers.

Yeah. It was fine. I just want to separate the obs and model results into two notebooks, and extend the exploration of model results, as examples.

What I'd like to do with the model output notebook is show some visualization and maybe a very simple and brief data analysis results from model output identified via CSW search. I see that as nicely complementary to the observations-focused (fixed sites) notebook that I'm building up, https://github.com/emiliom/ohw2018_tutorials/blob/ioosaccess/day2/ioos_data_access/01-iooscatalog_fixedlocationobs.ipynb. I'll see how far I can get with your gist plus stuff from other IOOS gallery notebooks (eg, http://ioos.github.io/notebooks_demos/notebooks/2018-03-30-wave_height_assessment/).

BTW, regarding this:

awareness of the problem we are facing on cells [10]-[11]. We don't have a reliable way to filter using either scheme or geolinks yet!

I'm running out of time (I'll be in a meeting for the second half of next week). So, at this point, I will just need to decide how to best deal with that messiness in a way that's reasonably informative and transparent but doesn't get too far in the weeds. There's a similar issue with the SOS service responses in the observations-focused notebook.

[ocefpaf]

I'm running out of time (I'll be in a meeting for the second half of next week).

I don't mean for us to solve this for OHW18 but we should remove those comments from the final version and keep discussing this internally (with IOOS people) on how to solve it.

[emiliom]

but we should remove those comments from the final version

Absolutely. Still, something about the messiness needs to be mentioned in the final version. Otherwise it's confusing and a bit of black magic. All input on how best to do this is welcome! @mwengren?

and keep discussing this internally (with IOOS people) on how to solve it.

Yup. But that discussion should happen somewhere else, not here :wink:. With time being so tight now, I'd rather keep discussions here focused on finalizing the notebooks, as opposed to side discussions stemming from them.

[mwengren]

Absolutely. Still, something about the messiness needs to be mentioned in the final version. Otherwise it's confusing and a bit of black magic. All input on how best to do this is welcome! @mwengren?

What is the expected result from the 'scheme' and 'geolink' filters in those cells, they should all be 'OPeNDAP:OPeNDAP'? Can you go back to the source metadata records in the Catalog and check to see the content there?

I'm not clear what messiness you're referring to exactly, the issue could either be mislabeled ISO XML that's input into the Catalog or it could that the format fields in CKAN (or pycsw?) aren't being populated properly for all the records. It's a little hard to understand what you're expecting from a quick skim of the notebook, esp without too many comments in there.

[emiliom]

Absolutely. Still, something about the messiness needs to be mentioned in the final version. Otherwise it's confusing and a bit of black magic. All input on how best to do this is welcome! @mwengren?

What is the expected result from the 'scheme' and 'geolink' filters in those cells, they should all be 'OPeNDAP:OPeNDAP'? Can you go back to the source metadata records in the Catalog and check to see the content there?

I'm not clear what messiness you're referring to exactly, the issue could either be mislabeled ISO XML that's input into the Catalog or it could that the format fields in CKAN (or pycsw?) aren't being populated properly for all the records. It's a little hard to understand what you're expecting from a quick skim of the notebook, esp without too many comments in there.

Sorry for the confusion. The issue applies to both the model-focused notebook (@ocefpaf's gist) and the observations-focused notebook, here. I've cleaned up the latter. So, to be more specific here, see cell 27. OGC:SOS has different meanings depending on whether it's from scheme or geolink.sniff_link. In the former it seems to capture only SOS GetCapabilities, while in the latter it captures all SOS service endpoints (GetCapability, GetObservation, DescribeSensor). That distinction in behavior for that same service label would not be intuitive to users (that's what I call a "messiness"). I'll add some text describing that distinction, its origin, and why a specific approach is chosen. If you have any general comments on this situation, they're very welcome.