search

cedardevs / feedback

An issue-only repository for centralizing all external feedback
0 stars 0 forks source link

OneStop Shopping Cart - Data access links, provide Collection and Granule support #2

Open wendy144 opened 5 years ago

wendy144 commented 5 years ago

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like A clear and concise description of what you want to happen.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

wendy144 commented 5 years ago

OneStop Shopping Cart requirements requests: A. Please provide support for Shopping Cart to be filled with Collection Data Access Links as well as Granule Data Access Links. Reasoning - A Collection-level record which does not have Granule-level metadata records associated with it, will have data access links associated with it; and providing both Collection and Granular Data Access Links will best meet needs of end-user.
B. Please provide examples of ISO (Collection-level) and ISO-Lite (Granular-level) metadata records which properly specify Data Access Links. Thanks much, Wendy

forgo commented 5 years ago

@wendy144

We can provide links with a 'download' link function from the collection level to the shopping cart, but there are some concerns we should address.

A. In practice, we find that not all links with a 'download' link function are actually downloadable links, they are actually links to web portals and services. This risk exists for granules, too, but seems to be a higher likelihood with collections.

This is a matter of the metadata using the 'download' function appropriately, but we have no control over that. We can make guesses in our code by further filtering out these links using heuristics like "does not start with 'http'" or it must have a suffix of ".ext", but this is prone to being unnecessarily restrictive and doesn't guarantee some non-downloadable links don't slip through the cracks.

How do we address the issue of non-direct-downloadable links being put in this part of the ISO to begin with?

The risk of just blindly trusting this is that the file we produce from the shopping cart (a file with a list of links) could fail for some of the links which aren't proper direct download links. If that is an acceptable risk for this feature, we can do that; however, it would be best for us as a group to decide how these errors could be prevented in the first place by ensuring the right kind of links are put in the right places in the ISO.

B. As we discussed on Monday, our documentation on where the links are retrieved can be found here:

ISO Indexing Mapping for Links

You can see that the linkFunction == 'download' is determined from here:

Link: /gmi:MI_Metadata/gmd:distributionInfo/gmd:MD_Distribution//gmd:CI_OnlineResource Link 'Function' Subfield: ./gmd:function/gmd:CI_OnLineFunctionCode/@codeListValue[1]

If you go to any collection detail page on data.noaa.gov/onestop, you can go to the "Access" tab and potentially see a "Download Links" section which would essentially include the links we are talking about.

The idea would be the ability to create a button/checkbox here to add one of those entries to the shopping cart.

Examples of where this can go wrong:

It appears it is very easy to run into this situation with most collections, so it would be a pretty big culture shift to get metadata creators to stop putting services and WAFs in 'download' link types.

Our team suggestions:

1) As much as possible, let's start using the 'download' link function in a way where we know it's a direct downloadable piece of content and not a WAF, service, web app, etc.

2) If we know filters we can apply to the information we have on collection that guarantees it is a direct download link, we can apply (within reason) custom filters/rules that can get those links into the shopping cart for those that find that useful.

3) If you are wanting to simply download all of the files from a WAF, anyway, it seems reasonable to use the existing links from the collection detail page on OneStop rather than needing a shopping cart at all.

4) Long term, we would like "service" links to be searchable and it would make sense that a lot of the "Download Links" in the metadata we see actually belongs and should migrate to the "service" section fo the metadata.

5) Also, as we move forward longer-term, we would expect things like WAFs to become obsolete as our tracking of data through Inventory Manager and being able to more directly access/order it becomes possible.

forgo commented 5 years ago

One other thought I had:

I was trying to get to the heart of your feature request, and thinking a lot of it has to do with remembering and storing items you've encountered over various searches, whether they are downloadable or not.

We are kind of rethinking the idea of "Shopping Cart" and would perhaps like to morph that into something more like "Bookmarks" (for lack of a better term currently).

We could apply a similar concept to shopping cart for things that aren't necessarily directly downloadable, but a way for you to navigate through multiple searches and star/flag/bookmark various types of things for your later reference (e.g. - collections, granules, services, etc.)

From there, we could allow you to download things we know for sure are downloadable still, and once we have a persistent user database, you could come back and log in to see the same items.

Just thoughts for the future...

wendy144 commented 5 years ago

Hello Elliot: a couple thoughts from World Data Service for Paleoclimatology (WDS-Paleo) I. Hurestics - Hueristics, like "does not start with 'http'of what is downloadable need to be agreed upon and clearly documented. And ".ext" is too restrive - it is necessary to be able to download a folder of data.

II. A ShoppingCart manifest file is needed for every bundle downloaded (this might be in keeping with your concept of bookmarks) - Reasoning: If a link is NOT "downloadable" it should be listed in a manifest included with the shopping-cart bundle downloaded. A manifest, and check-sums should be included in the shoppingCart bundle to be downloaded. Credit and provenance for all datasets bundled needs to be included in the manifest, along with clear instructions to the end-user to cite appropriately, date downloaded etc. etc. ...