Collections don't indicate deleted datasets, changing HDAs doesn't propagate up to DCEs

[jennaj]

edit Start with https://github.com/galaxyproject/galaxy/issues/17735#issuecomment-1999297437

Describe the bug The datasets included in a collection appear to be a mix of the original datasets and the datasets copied into the collection depending on the view -- Specifically: which you view nested inside the collection when expanded in the history (original files added to the collection) differs from which metadata is inherited/displayed on the collapsed collection (actual files inside the collection).

In practical use: if any of the hidden datasets are manipulated (active/deleted state changed, datatype changed) what can be viewed in the history panel is a mix of those states! This can impact how users curate data in the hidden panel (it happens..), how tool forms consume the collections, and using some of the state change functions (in particular, when clicking on the "eye" icon for a dataset in an expanded collection via the active tab, the link to "undelete" a datasets flips away too quickly to use (seems to be referencing one of the copies), since it is quickly replaced the file content (seems to be referencing a different copy?).

See screenshots.

Galaxy Version and/or server at which you observed the bug UseGalaxy.org The Galaxy Server is running version 24.0.rc1 , and the web client was built on Thursday Mar 14th 3:00:16 2024 GMT-7 .

Browser and Operating System Operating System: Windows, Linux, macOS Browser: Firefox, Chrome, Chrome-based, Safari

To Reproduce Steps to reproduce the behavior:

0. Load up a few datasets, and put those into a collection
1. Go to the hidden tab, notice that each of the original datasets has been duplicated (expected)
2. Set up some changes in the second set (those that are "in" the collection)
   • Change the datatype to something else (user might do this, I used since batch change on collection is not working and it ended up helping to learn what was going on overall https://github.com/galaxyproject/galaxy/issues/17734)
   • Delete one or more of the datasets (what a user did by accident here https://help.galaxyproject.org/t/dataset-temporarily-displaying-as-deleted-preventing-job-from-running/11923, or maybe there were trying to manage quota directly)
3. Go back to the top level of the history, and view just active datasets, then layer in deleted and hidden
4. Notice that the collection has been updated with the new datatype assigned to the datasets (good!)
5. Click to expand the collection, and notice that the datasets still have the old datatype (seem to be referencing the originally added datasets?, not so good...)
6. Try to view the datasets in the collection using the "eye" icon to figure out the content (users do this, as do I, for first pass troubleshooting, say from a job error reporting "datasets deleted before job could run"), and notice that the dataset that was deleted in Step 2 above quickly flips between a) red bar with an undelete dataset link and b) regular file display of the contents (link is not usable, see the Ghelp ticket above for more about this situation).
7. The collection cannot be used as an input (but can be selected on some tools forms and the run fails, and some bounce as containing deleted content directly on the form -- the latter is better).
8. Big picture: how to recover is not obvious.

Expected behavior Reference/display the datasets actually inside the collection in all collection related views, including display views. As part of this, if a dataset is in a deleted state, leave the "click to undelete" view as a final state and not display the file contents (even if available under the hood). When undeleted, then it can be displayed.

Screenshots These are screenshots showing everything after Step 3 above .. .. .. .. .. .. .. .. .. .. .. ..

Additional context Test history screenshots are from. https://usegalaxy.org/u/jen-galaxyproject/h/test-ux-behavior-when-a-collection-contains-deleted-elements

[mvdbeek]

There are 2 independent issues here:

• [ ] If a history dataset association is manipulated outside of the collection context we don't know that an update has happened to the dataset collection element. This might be a hard bug to fix if we don't want to always load all of the collection elements each time we open a collection.
• [ ] collections with deleted elements have no indicator that an element has been deleted