Shopping cart: support for interactive building of collections

[cricketsloan]

From https://app.zenhub.com/workspaces/dcp-5ac7bcf9465cb172b77760d9/issues/humancellatlas/dcp/34:

As a user of the DCP, I will approach it and it's vast amounts of data and, after searching for the data that meets the requirements of my study or analysis, I will want to take those search results and treat them as a unit.

A user can create a persistent list of data, view existing lists, export or download the list. This feature is analogous to a shopping cart capability on an e-commerce site.

Original User Story for #88:

As a user I would like to iteratively add entities to a continually growing list, and then save that list.

This could be done using a Dynamo DB table, and providing some REST api to allow users to do simple CRUD operations on their current collection, and upon clicking save, we would save that collection via the collections API.

┆Issue is synchronized with this Jira Epic ┆containerName: azul ┆Issue Number: AZUL-64

[cricketsloan]

➤ Trevor Heathorn commented:

Will eventually be blocked by "authentication on the web service" which will be required to use the Collections API.

[cricketsloan]

➤ Trevor Heathorn commented:

From CGP Tech Leads Meeting on 7/12: Work on making collections support in Orange for use by download shopping cart, be able to mint into blue box collections, using a collections JSON schema.

[theathorn]

• shopping cart to save search results
• shopping cart per entity type (Files, Projects, ...)
• click button in UI to make a named collection (or use Collection ID)
• share, rename a collection
• once saved, user can't alter content of collection

[theathorn]

See https://docs.google.com/document/d/1dvz7ij5JDAHSFZmpAHsw3_xbDge-EEhuLcvA5SIkhc8/

[theathorn]

Blue Box Collections API Swagger: https://dss.integration.data.humancellatlas.org/ Collections API Spec (not maintained): https://docs.google.com/document/d/10WXG2oh51RgigTBRdLpq8b48APEv8BhPkPTsM1DtpzA/edit#heading=h.6drsf9p5qnbj

[theathorn]

User Story #1 As a user I wish to select files according to a faceted search and save the results in a "shopping cart". I can conduct additional searches and add or delete files to the shopping cart. When I'm ready I can download all the files in the shopping cart. See the GDC Data Portal (https://portal.gdc.cancer.gov/projects) for an example. This can be achieved by creating an (immutable) Collection for the selected files which will return a UUID or url for the Collection. The Collection UUID/url can be passed to the HCA CLI to perform bulk file download.

User Story #2 As a user I can create a Collection (as above) and pass the Collection url to the Matrix Service for further analysis.

User Story #3 As a user I can create a Collection (as above) and pass a reference to the Collection url to a selected "Red Box" (portal analysis application) for further analysis (see https://app.zenhub.com/workspace/o/humancellatlas/data-browser/issues/6).

User Story #4 As a user I wish to manage my Collections by being able to:

• create multiple Collections
• view my list of Collections
• delete a Collection (just the Collection, not the underlying data)
• share a Collection with another user (via a url)

[shiroyuki]

The rough draft is moved to the wiki

[BenWu]

Some additional considerations to the draft:

• Collection names set by the user
• Endpoints to add and remove files in a collection
• A separate CollectionAccessibility object might be awkward to use or not performant if we use a non-relational datastore (probably a non-issue but something to consider when we start implementation)
• Using DSS to store collections (I think this was mentioned at some point)

[shiroyuki]

All X-Access-Key headers are replaced with Authorization for JWT Authorization Bearer Tokens.

[shiroyuki]

@Ben-Wu, you can find the second draft at https://github.com/DataBiosphere/azul/wiki/%5BFor-Discussion%5D-Technical-Considerations-for-Issue-88-(Version-0,-Draft-2).

[shiroyuki]

@hannes-ucsc @Ben-Wu The RFC is now available at https://github.com/DataBiosphere/azul/pull/417.

[shiroyuki]

@theathorn, as @hannes-ucsc and I discuss on the default cart, we wonder whether we should allow only one entity type per cart or multiple types?

[theathorn]

@shiroyuki @Ben-Wu Is it relatively simple to allow multiple entity types per cart? If so, we could allow that in the cart service implementation and then, if we chose to, restrict to a single-entity-per-cart at the UI level (@NoopDog).

[BenWu]

It shouldn't be a problem to allow multiple entity types per cart. It would probably be simpler for us to allow it.

[theathorn]

You could add a "cart type" parameter to the cart creation API that specifies the cart type as "multiple-entity-type" vs. "single-entity-type".

[shiroyuki]

@theathorn Cool. So, the RFC #475 (PR #417) will be updated accordingly.

[cricketsloan]

➤ Brian O'Connor commented:

See the spec below starting at "Saved Searches, Shopping Cart and Handoff" on p.10.

• https://docs.google.com/document/d/1y5bxYR5Y_Xtf4omFFIwzVTBarCn3e87Hqp0kYKo6Nv0

[theathorn]

Test comment added on Zenhub.

[cricketsloan]

➤ Trevor Heathorn commented:

Test comment added on Jira.

[shiroyuki]

@theathorn, Sorry for the late request for verification. I have a few questions/updates regarding to the user stories (within this ticket).

From my understanding, user stories 2 & 3 allow a user to create a collection from a selected shopping cart and pass the reference of the collection to a selected service (the Matrix Service and "Red Box". So, my questions are:

• What is the Matrix Service? Is it deployed in the development environment?
• From the description of "Red Box", is the Matrix Service a "Red Box"?
• Is there anyway to see the API spec of both services?
• From the description, if the user can pass the collection URL to a selected Red Box or the Matrix Service, Azul just needs to provide the full URL to the resource (on DSS), right?

In user story 4, according to the API doc for PUT /collections (what we use to create a collection),

New collections are private to the authenticated user.

That means we can't share a collection with another user (via a URL) directly from DSS with the creator's JWT. Ben and I confirmed this. One way we can do this is to copy the idea of implementing pre-signed URLs from AWS S3. I would need some input from @hannes-ucsc on any approach better than implementing the pre-signed URL generator.

@hannes-ucsc, according to the API doc, we cannot list collections that belong to the requesting user. So, I discussed with Ben on adding collection_id to the Cart table to let Azul track any collections created by the login user.

[theathorn]

To satisfy the SOW-3 M1(c) requirement it is sufficient to be able to create a shopping cart and then optionally be able to turn it into a collection. Collections are intended to be long-lived items and therefore it may be decided to only use them for such items as Data Releases.

For User Story #2 (pass Collection URL to Matrix service) and User Story #3 (pass Collection URL to a Red Box): It hasn't been decided whether or not to use the Collections API for these.

For User Story #4 (manage my Collections): Currently the Collections API does not support shareable collections (humancellatlas/data-store#1728), listing collections (humancellatlas/data-store#1729) or notifying when a collection is made public (humancellatlas/data-store#1729).

[shiroyuki]

@theathorn Just to update on this ticket. According to your comment on December 12, 2018, #629 should satisfy the requirement on the shopping cart part. #627 and #628 will enable data export to DSS as a collection (still non-sharable).

[hannes-ucsc]

Shopping cart has been deprecated.