Open mtbc opened 8 years ago
I am not sure about the exact implementation of deep copy, but atm I think we are trying to solve 4 issues by this:
I think we could ask @imunro about the setup in their institute because they were trying to tackle the Public group case seriously to have some concrete user case.
As far as publication goes our current approach is to move data for publication into the public group. Ideally it would be possible to shallow copy into the public group - leaving the existing Project-Dataset structure intact.
@mtbc - I will try and formulate the use-cases I have in a clear format in the next day or two and add these.
This is how I conceptualise Deep vs Shallow copy - (I may well be wrong!) Shallow Copy is what happens with current copy-paste. Note: Deep Copy shown in different group, but could be in same.
Deep Copy Use Cases
Publish 1:
Publish 2:
Analysis:
Notes:
Once we have Deep Copy then the UI terminology will need to make clear the difference between them. May need to call Shallow Copy “alias” or “shortcut” and Deep Copy “copy”. If a copy of Original File + settings.
It is not transparent from the UI what happens when I move a Shallow Copy (linked) to a different group from the original - both “copies” disappear - without any warning. This will differ from moving one of a deep copy-original - where only one will move. Will probably need way to differentiate Deep from Shallow copy - e.g. add - “copy” to name (like OSX Finder).
Questions:
@mtbc - still need to think some more about all this.
From the point of view of what's technically implemented on the server (which may not be how things should be for the end user), deep copy is possible by other users even in read-only groups, so long as they don't expect the resulting image to be in the original owner's dataset, as the copy is owned by the copier.
@pwalczysko - is this what Ian needs?
I think we need to include here what @imunro acutally wants:
Sorry, first commented, then read your above comment @gusferguson
I think basically yes, this would be @imunro 's demand. But for details like Change to one is reflected in other
and whether or not he actually wants all the Annotations with the shallow copy, this would be for him to specify.
I could use some discussion about "shallow copy". The "change to one is reflected in the other" sounds new to me, and also sounds quite expensive.
@joshmoore - I might be causing confusion by using "shallow copy" - I just mean the current "link" copy that exists already.
Shallow Copy is what happens with current copy-paste.
Ah, i.e. an image is linked into two datasets at the same time? Ok. I'll try to adjust my mental map.
From what I can see that looks as if it would satisfy the requests we've had. Basically people want a method of easily making their data public e.g as supporting data when a paper is published. Currently they just move the data in question onto the public group. The shortcoming of this is that if ( as is usual) they are cherrypicking the 'best' examples then moving it disrupts whatever organisation they have in place i.e. if they then want to do further analysis on all the data it's now in 2 different places. = hope that makes sense.
@imunro - thanks for the clarification - I have added that as Publish use case 2.
@joshmoore: Does the above introduce any new dimensions? I am thinking we are facing exactly the issues we feared. :smiley: I wonder how to proceed.
projection will probably fall into the category of deep copy: at least at the graph level (a new set of raw data is created) Currently part of the graph is shared between the original image and the source image preventing move for example.
Absolutely. If the pixels service weren't so annoyingly featureful I'd have fixed this already! /-:
We can probably allow deep-copy of files without any database changes. However, regarding data duplication in the server's binary repository: separately from the pyramid duplication fear above, if we are to avoid data duplication in the managed repository then there is the question of what happens when the original file is deleted.
With the database change to a Boolean "is-a-copy" column that removes that row from the uniqueness constraint,
Without any database changes,
So, UX questions focus on: If the original is deleted, how good or bad is it if the new copy's pixel data (or copied attachments) disappear if the originals are deleted? Does it suffice if at least the copier knows which kind of copy they got?
(A question for @joshmoore might be: do you see a solution to pyramid duplication that also requires database changes? E.g., a pointer from a Pixels
copy back to some other instance's pyramids.)
Another potential factor: how well would this work with an object data store?
So, UX questions focus on: If the original is deleted, how good or bad is it if the new copy's pixel data (or copied attachments) disappear if the originals are deleted? Does it suffice if at least the copier knows which kind of copy they got?
This is very very bad. If I delete the original and the copy is deleted as well, then this is not a deep copy by any stretch of imagination. The user will be completely unable to absorb a concept of any in-betweens regarding deep copy (direct experience of the situation which we have now).
This is very very bad.
Couldn't agree more!
if the user does a deep copy, he/she will not expect to be affected by the deletion to the original. We cannot do that
So, my current guess is:
originalfile.iscopy
with the repo
, path
, name
uniqueness limited to !iscopy
pixels
and thumbnail
add a bigint column binaryfrom
referencing id
non-cyclicallyHowever, I am not sure that these are exactly what we will be glad we did. Chat some more? Go for it? Postpone to >5.3?
(I assume we do still need to permit deletion of the original.)
If we don't want to prevent the deletion of the original, then I'd say the other recent comments above amount to a hard-linked mrepo-internal re-import which we could make available without the DB changes. The primary disadvantages would be:
From my side, I think we're still looking for features (or API breakages) which would require the DB changes, i.e. finding the related original fileset or as Mark asks, prevent an operation like delete on the original. Do we want any relationship between old & new?
NB: The addition of binaryfrom
is new to me, but especially with regards to letting thumbnails prevent deletion of original files I have some concerns.
would be disallowed (i.e. would throw an exception) if multiple mount-points exist under mrepo
is a big problem I'd have thought: after years using the initial volume the admin adds a new and suddenly nobody can deep-copy any more.
binaryfrom
just points back to another (say) thumbnail in the same table; have to mv
its file to a "new" original's ID if the original is deleted; shouldn't prevent deletion of original files.
So, while it requires a run of the import machinery, there's a reimport workaround available, perhaps best by a server-side script, broadly,
Alternatively, the duplicate machinery is already just about there and its genericity will tend to cover edge cases but it needs,
I'd like to think we're relying less on server-built pixel pyramids anyway so can brush that duplication issue under the rug. :smiley:
In chatting with @joshmoore a third option came to my mind, broadly,
This does not require database changes but should be able to at least copy filesets, attachments, thumbnails.
This might be the best in terms of trading off implementation effort with outcome.
In trying to figure where in the managed repository to put the copy of the original files: the DuplicateI
steps are running within an executor.execute
but the code for creating a new receiving space in the managed repository passes Ice.Current
around, fiddles with omero.session.uuid
to sudo, etc. It may be tricky to create the parent directories for the copy of the original files from within an IRequest
step.
I would assume the repository has examples of going from Ice.Current
to a valid executor.execute
context. It may need extracting to a helper though.
And the reverse too? Let's see what we can find. :+1:
(DuplicateI.step
would like to do something like ManagedRepositoryI.internalImport
.)
Perhaps awkwardly, further executor.execute
occur within the import preparation and I don't believe they like to be nested.
It's true they don't like to be nested. Though once one is active other methods shouldn't need to call them. If a separate transaction is necessary, then only submit()
will do.
Aha, that might be exactly the clue I needed to get things working, even if I have to duplicate some execute
-using code at first.
Quite possibly. The RepositoryDao, I think, has two copies of several methods (or did) for just that reason.
Current interesting problem is backing out from failures: how to track what makeCheckedDirs
actually had to create so that we can now delete them.
It could be useful for another devspace to be created for testing deep copy. Bugs might mess up the binary repository or database. Unless it's easy to restore merge-ci's data? cc: @pwalczysko
Unless it's easy to restore merge-ci's data? cc: @pwalczysko
Not very easy. Moderate difficulty - needs manual reimports (although there are scripts, but these need to be run manually).
In considering how to provide access to DuplicateI
from OMERO.web it could be worth reviewing typical duplication workflows, especially given how DuplicateI
supports extra arguments. For example, by default annotations are included when an image is copied. By using typesToIgnore=['IAnnotationLink']
one can wholly omit annotations from a duplicate image or, for instance, by using typesToReference=['Annotation']
one can have the duplicate point to the original's annotations (thus preserving ownership, etc.). Also see https://github.com/ome/omero-cli-duplicate/issues/7.
3 workflows for duplicate:
Puclication workflow
Cooperation workflow:
Organization of own data workflow:
Leaving user workflow
Any combination of the above, and other real-life workflows I might have not thought of.
In short, all three flags highlighted in https://github.com/ome/design/issues/20#issuecomment-688294423 comment are under circumstances interesting and useful in the above four workflows.
@joshmoore @mtbc does that answer the question ?
(A quick extra idea I had during the call: what about duplicate --legacy
which has the previous behavior, i.e. turns off the binary bits?)
what about duplicate --legacy which has the previous behavior, i.e. turns off the binary bits?)
I would be afraid that it is very difficult to explain the behaviour of this option to the users, but maybe someone can come up with a scenario where this flag would be of advantage to a user ?
Seems to me there are basically 2 workflows:
The only issue might be if you do 2, then decide to move it to another group later. By far the biggest driver for the duplicate functionality (as far as I'm aware) is "publish" or other flavours of option 1. So in webclient the "Make a copy" feature could just support the 2 options above?
So in webclient the "Make a copy" feature could just support the 2 options above?
@will-moore If I take your lead on the workflow listing, then i would see a sub-option of your option 2. (say, 2b) with
Option 2b is very interesting for saving space for FileAnnotations I would imagine, but also it could help to keep overview by not proliferating tags etc.
@pwalczysko Your 2b is the same as my 2 (do not duplicate annotations). Is there any reason why you'd want to duplicate annotations if you're NOT moving to a different group? Do we want to support that in webclient? Seems like that would cause some confusion if you duplicate Tags in the same group.
In the 'don't duplicate annotations' scenario, do we mean ALL annotations? E.g. if I duplicate an image with Key-Value pairs, then I edit the KV pairs on one image, would they update on the other image? I probably wouldn't expect that. Same for Comments (although you can't edit in webclient) and Ratings. Files and Tags are OK not to duplicate.
In the 'don't duplicate annotations' scenario, do we mean ALL annotations?
@mtbc might correct me, I think the answer is yes. Not sure if any granularity is possible there @mtbc ?
Is there any reason why you'd want to duplicate annotations if you're NOT moving to a different group?
Yes, to make the annotations yours, as you can duplicate objects of other users in 3 types of groups. The duplicate gives you back a nice, one-owner tree with one-owner annotations and one-owner links.
E.g. if I duplicate an image with Key-Value pairs, then I edit the KV pairs on one image, would they update on the other image?
@will-moore that depends on whether or not you chose option 1 or option 2b. Btw, the option 2 (or 2a ...) was thought by me as "do not duplicate the annotations at all, i.e. have an unannotated duplicate image/Dataset/project as a result.
Same for Comments (although you can't edit in webclient) and Ratings. Files and Tags are OK not to duplicate.
This assumes the granularity which I am not sure is a given, see my first sentence in this comment... cc @mtbc
One more usecase of a granular exclusion of certain annotations duplicates can be seen in https://github.com/ome/omero-blitz/pull/100#issuecomment-689446515 - ROIs might simply take too long to duplicate, and the user might choose not to duplicate tham...
Not sure if any granularity is possible there
By type is available, e.g., duplication can treat tags differently from comments.
OMERO's 5.2 branch already offers the
Duplicate
request to have the server copy model subgraphs; it is tested by https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/tools/OmeroJava/test/integration/DuplicationTest.java. However, copying images is disappointing because the pixel data is missing because it depends on original files which are uniquely named and singly owned. For space reasons we don't want to actually have to copy the files in the binary repository but we probably want the duplicator to own their own duplicate which may be moved to different groups from the original.The initial use case, for which the existing duplicator probably already suffices, is described in http://trac.openmicroscopy.org/ome/ticket/11532 where it would be possible for scripts to duplicate instruments and suchlike instead of sharing them with derived images or, even better, the duplication could be automatically done by need as described by https://trello.com/c/ISnICsrC/16-auto-duplicate-in-graph-operations.
A more general deep copy could be arranged if we could duplicate original files. For instance, we could allow marked "copies" of original files to have the same name but be read-only, deleting the underlying file only when the last of that name is deleted from the database. Or, we could actually use filesystem links to seemingly copy the file, except that the new copy would have to be on the same volume if hardlinking, or would be lost if softlinking when the original is deleted. Then, there is the matter of pyramids: we want to avoid generating duplicate pyramids, but how can the owner of the duplicate have permissions to find the original's pyramids?
This issue exists to collect interesting points regarding: