New data libraries - missing features

[bgruening]

Since the old data library is deprecated I ask on our mailing list about features that are missing from the new data library interface.

• [x] Rename and modify folders
• [ ] Move folders and datasets inside a library and to another library (preferably drag/drop)
• [x] When modifying permissions as non-admin user, expose_user_name and expose_user_email config options are not respected
• [x] Edit library dataset information: datatype, genome build, info, message
• [x] Upload folder and datasets to a new history (see #1265)
• [ ] Make searching recursive (see #1045)
• [x] Display the "message" field when browsing (see #1570)
• [x] Select all datasets button when importing (https://github.com/galaxyproject/galaxy/issues/1606)
• [x] Import a data library folder as a collection (see #2108)
• [ ] Handle form definitions attached to library datasets (allow modification?; import?; ignore?; replace?)
• [x] Option to symlink individual files when uploading (see #1529)
• [x] Show the state of the upload job of library datasets
• [x] Ability to order library contents based on columns
• [ ] Cannot set "read" access permission on a folder, must set individually on files within that folder.
• [ ] Cannot share datasets with individual users. I must have a "sharing role" available (I could create these once by sharing a history with a user, but no more?), or by have a normal role available.

• [ ] Cannot share LDAs with multiple roles.

  User has to have all these roles in order to access this dataset. Users without access permission cannot have other permissions on this dataset. If there are no access roles set on the dataset it is considered unrestricted.

  Is absolutely useless to me, I want to share with multiple roles without creating a super-role just for the purpose for sharing with a set of groups.

• [ ] Be an admin. Modify: Role B, Access: Role A, Manage: Role B. Save. All modified roles disappear except for "Access"
• [ ] Ensure that documentation and functionality is consistent and clear. In some places sharing is done via username / email (histories, etc), whilst in other places it is done via roles. These have extremely different behaviours despite similar appearance. Sharing to histories can be done even without expose_username/email. Sharing libraries works via roles and does not allow pasting in an email. We should unify sharing xref #4602

[nsoranzo]

I have added an extra point to the description, i.e. "when modifying permissions as non-admin user, expose_user_name and expose_user_email config options are not respected".

[nsoranzo]

I've also reworded the second point to mention that it would be nice to move folders and datasets from a library to another.

[nsoranzo]

I've found another regression with respect to the deprecated library interface, it's not possible to edit library dataset information: datatype, genome build, info, message. Added to the list.

[nsoranzo]

I've merged #1265 here.

[bwlang]

Since it looks like this is the new spot for data library missing features before the deprecated lib manager can be removed.

• import from user directory on nfs mount with many folders is unusable (never populates list or takes 30+ minutes)
• As Björn mentions, need to be able to move files (preferably drag/drop)
• report sizes of subtrees
• editing names, tags, meta-data of both folders and files

Other (IMHO) important features

• add disk space used to user quotas
• search

[martenson]

@bwlang I am not surprised that import with the file browser is unusable when you need to browse 1000s of files, but importing using the path should still work decently. Are you also having problem using path import?

Thank you for all the suggestions!

[bwlang]

Path import works great... But I don't think it a good option for end users On Jan 13, 2016 4:15 PM, "Martin Cech" notifications@github.com wrote:

@bwlang https://github.com/bwlang I am not surprised that import with the file browser is unusable, but importing using the path should still work decently. Are you also having problem using path import?

Thank you for all the suggestions!

— Reply to this email directly or view it on GitHub https://github.com/galaxyproject/galaxy/issues/1177#issuecomment-171434868 .

[martenson]

@bwlang no it is not (it is meant for admins), but building reliable user-facing web UI for async browsing potentially huge network-connected storage filesystem is probably not a task we want to undercome :/

In my thinking the file library import file browser is targeted at users importing from their own FTP folder after they uploaded files to the instance.

[bwlang]

One layer is deep enough for me... No need to see the whole tree.

Do you think a find -maxdepth 1 would do it?

Brad

On Wednesday, January 13, 2016, Martin Cech notifications@github.com wrote:

@bwlang https://github.com/bwlang no it is not, but building reliable web UI for async browsing potentially huge network-connected storage filesystem is probably not a task we want to undercome :/

— Reply to this email directly or view it on GitHub https://github.com/galaxyproject/galaxy/issues/1177#issuecomment-171494775 .

[gregvonkuster]

Not sure of the voting process, but +1 on adding these important missing features!

[pavanvidem]

I just found this issue here, not moving files around is really a regression.

[bgruening]

:+1: The old data libraries stopped working for us, it's just too large.

[sarah-peter]

I just commented on #1529 and also want to add my +1 here. The options to link individual files and move datasets are really important in my opinion and I have used them a lot in the past.

[gregvonkuster]

+1 from me as well for being able to link individual files.

[sarah-peter]

The old data libraries did also dereference links once when linking files instead of copying. This was very useful for cases like this: https://biostar.usegalaxy.org/p/1297/.

I'm wondering why you decided to completely drop the old interface, when there are still so many features missing from the new one. They existed in parallel for a long time, so why not just switch the default, but keep the old one until all features are implemented?

Are there any plans to implement the missing features or a timeline?

[hexylena]

added another bullet point. Are there any of these we can cross off? Would be nice since the old one is now deprecated.

[bgruening]

I think @mvdbeek fixed one in https://github.com/galaxyproject/galaxy/pull/3559.

[hexylena]

I think that is the other way, collection -> library, not library -> collection as is in the list in the issue.

[chambm]

Right , I found that I could add a collection to a library in 17.05, but it only added them as individual datasets. It'd be far better if the collection was itself presented as a folder and when adding it to a history, it kept its collection status.

[martenson]

I am currently chewing through the list (you can track progress here).

One thing I am pretty sure I would like to completely cut off is libraries using form definitions. If you have a good use case and/or use it heavily, please comment here.

[scholtalbers]

I just spend some time tracking down an issue as to why a certain workflow would not run (the editor complained about some At least 2 datasets are required. Using default: ''. so I looked in the wrong direction..). It turned out that the users trying to use the workflow, me included, were admins and we imported a shared .gtf file from a data library assuming it was publicly available. Only in the handler logs I saw the message galaxy.tools.execute WARNING 2017-09-27 15:29:26,476 There was a failure executing a job for tool [NFS_transfer] - Error executing tool: User does not have permission to use a dataset (1204) provided for input., indicating this was a protected dataset.

In this case, there are a few options to remedy:

1. the admin user should not be able to import and use the dataset in the tool form
2. a clear warning should show up when trying to import
3. allow the admin to use the dataset as tool input ignoring the permission set on the dataset

[scholtalbers]

(Still) missing as admin:

• path of linked datasets ("Disk file:")
• ability to upload a dataset directly i.e. "upload files"