V2 - Allow dataset to be added to a layout multiple times

[adkinsrs]

This has been a longstanding request and previously users have addressed this by cloning a dataset upload for the sole purpose of changing to a different default display for the cloned dataset.

The idea is to modify the layout_members table and client/server side code to accept multiple datasets in a single layout. This will require, in my opinion, some extensive modifications.

(I also apologize if I use "layout" and "collection" interchangeably)

• We cannot simply put the dataset ID into multiple layout_members, as then we have no idea which displays to show for each member. Therefore we need to change the dataset ID column into a display ID column where we can get the dataset ID from the dataset_display table.
• Currently for each layout member entry we keep tabs on both the single-gene and multi-gene grid-based row and column information. However by switching to a display ID, it guarantees that one set of coordinates will be useless. So from there, we will need to normalize out to just a single standard set of grid attributes (start_row, start_col, grid_width, grid_height). This will also require using the dataset_display table to determine if the layout member is multigene or not. Essentially, the layout will be a collection of single and multigene displays that will ideally fit on their own separate grids when one set is desired.
• Regarding default displays... we currently only handle 1 single-gene and 1 multi-gene default display per dataset. Currently in the dataset explorer, we can ideally add the same dataset multiple times to a layout (just click the "add" button again), but how do we determine what the display should be?
  • Should we just make each display the default display, and let the user change it in the expression viewer? This would certainly be the easiest solution but some users would get confused when the same display shows multiple times on the page.
  • Should we have higher numberings in the dataset_preferences table where a 2 would be 2nd-highest default priority, 3 the 3rd-highest and so on? Where would we change this? The curators would need to be adjusted to handle this, or we build a widget in the layout arranger widget to let the user specify each layout member's display ID (which feels redundant to setting a default display.
• What displays get priority when the layout owner and the dataset owner do not match? Suppose I create a collection "foo", and I curated a display for a dataset "bar" owned by Steve, I would assume that when I view my collection "foo" in the expression viewer my display for dataset "bar" would be the one that renders. But if we are storing display IDs as layout members, and Steve has his own default curation for the dataset he owns, should he see that when he brings up my "foo" collection, or the display I chose for that collection? How does this work if I added 2 unique displays from his dataset into my "foo" collection... would he see those, or his default twice?

These are just some thoughts I had around switching over to a layout (collection) format that allows a dataset to appear multiple times. Overall, while it's a "need-to-have", I think there are some logistical hurdles we need to discuss before diving head-first into this, as simply changing the table has widespread ramifications on default displays, filtering single and multigene displays, and any code that relies on those and layout member information.

[adkinsrs]

@jorvis and I had a chat about this and came to a couple of conclusions

• One thing we can do immediately is change the dataset explorer page to now support adding display IDs to a layout instead of dataset IDs, and reflect this in the expression/projection viewers.
• There are a couple of issues still in contention that we want to pose to the analysts on the gEAR team:
  • Should we allow for the same display to appear twice in a layout?
  • While we felt that default single- and multi-gene datasets curations should still be maintained, we did not come to a conclusion on the role they will serve for the expression/projection viewers. If the dataset appears 2+ times in the collection, does the default apply to all instances (which may violate the above bullet point). I think we agree that if the dataset occurs once, then it is fine to use the user or owner default for that dataset.

[adkinsrs]

TODO:

1. Create a new database table to serve as a temp layout_members using the display ID
2. On the original layout_members table, if the "multigene" columns have not been added, we need to add them. The conversion script I wrote to migrate from layout_members to layout_displays assumes they exist. This way we can also apply a "legacy mode" update to the new layout_displays table if the single- and multigene layout displays have entries that resemble the original layout_members table (only width, no height and starting row and column)
3. Write script to populate new temp layout_members table using the old one. Convert dataset ID into their default single and multigene display ID. If a multigene curation does not exist, just do not include, since single-gene and multi-gene collections can be different sizes.
4. Backup db and wipe old table
5. Modify geardb.py code and CGI code that uses layouts

[adkinsrs]

To ease transition while I rewrite this, I converted the old layout members into a display-based one found in a new table called "layout_displays". Then I plan on adding a LayoutDisplay class in geardb.py and change/link all the calls to the LayoutMember to the new class (and table references everywhere too)