chgrp scripts

[will-moore]

This adds 2 scripts to use in a workflow for duplicating Figures into a different group.

Workflow

• Make a list of 1 or more Figure ID (File Annotation IDs), e.g. "10,11,12"
• Create an empty Dataset
• Select the Dataset and run script "Figure_Images_To_Dataset.py"
• Enter the Figure IDs (Dataset ID already populated) and run the script
• Dataset should now contain all the Images that are in the Figure(s)
• Now duplicate the Dataset $ omero dupliate Dataset:ID
• Then chgrp the new duplicated Dataset to the target Group (via cli or webclient UI)
• Select the duplicated Dataset in the new Group
• Run the script "Dataset_Images_To_New_Figure.py"
• Enter the Figure IDs (same as before e.g. "10,11,12") (Dataset ID already populated) and run the script
• This will create new Figure(s) in the target Group, with each original Image replaced with the duplicated Images in the target Group
• The Figure File > Open dialog will list all Figures and their groups to check success (might need to refresh)
• If there were duplicate Image names in the Dataset, message says to look in logs which will list those images (these will need to be manually replaced).

cc @pwalczysko

[will-moore]

I'm not sure if we really want these scripts in the OMERO.figure repo or if there's a better place for them. I guess it depends how common the above workflow is needed in the community - It is certainly needed on our nightshade server and currently the steps covered by these scripts are done by users manually, which is laborious and error-prone.

[sbesson]

Are we ready to support these scripts i.e. treat them as API? If the answer is no, then we should definitely look into another place to store them.

The workflow presented here i.e. full duplication of a set if figures + constitutive images into a public group is both advanced but also valuable. Would the content be amenable to be constructed as a new how-to guide? In this case, I could imagine the scripts could live alongside the how-to as supporting material. /cc @pwalczysko @jburel

[will-moore]

Sounds good. Presumably they'd go in omero-guide-figure rather than under data management/duplication workflows.

[pwalczysko]

@will-moore @sbesson Understood the main features of the workflow and the discussion. Will come up with an omero-guide-figure addition PR soon.

[pwalczysko]

@will-moore What can we offer to facilitiate the step 1 better for the end user ?

Make a list of 1 or more Figure ID

Seems trivial if

• you know how to do it, i.e. where to take the Figure IDs from (this needed 2 separate zoom calls to make sure the user understood that Figure IDs have nothing to do with the image IDs contained in those figures and also where to take these from)
• you have only 1 or 2 figures (our first real-life case has 24 figures though to start with) - even if you know that youi can retrieve the ID from the figure url, it is quite painful to get the 24 urls in the first place (by opening all the 24 figures)

[pwalczysko]

I have started working on https://github.com/ome/omero-guide-figure/pull/13 Still very raw. I think we could suggest some hql to retrieve figure ids, such as

omero hql --all --limit 1000 --style plain --ids-only  "select f.id from FileAnnotation f where (f.details.group.name = 'read-only-1' and f.details.owner.id = 454 and f.ns = 'omero.web.figure.json')" | sed -e 's/^.*,//g' | paste -s -d, -

which will retrieve all figure ids in nice format which belong to user-3 in read-only-1 on merge-ci (Edit: command edited to filter out the figures from all the FileAnnotations retrieved)

We need to make sure first the scripts work though, the first one does not do it for me, see https://github.com/ome/omero-figure/pull/443#discussion_r684380911 (Edit: comment deleted, script works, see below)

Edit: Sorry, the reason for the crash of the script was most probably that I was passing just some FileAnnotation ID, which was not OMERO.figure. When a real figure ID is passed, the script works. Deleted the comment.

[pwalczysko]

Some edits in https://github.com/ome/omero-figure/pull/443#issuecomment-894400201 - the first script seems to work when the correct ID is passed.

[pwalczysko]

@will-moore Re: Figure_images_to-Dataset.py script works fine, except two RFEs:

1. Reject wrongly formatted input instead of executing the code and returning nothing.
   • paste in several IDs of Figures, but not spaced with commas, but spaces
   • execute the script, it returns without errors, but also without any action (no images linked to your dataset). The Activities message says "No Images found"

In case the user pastes in IDs in a wrong fomat, check for it and warn and inform the user about the expected format.

2. Deal with repeated execution more gracefully
   • have a Dataset which already contains some images contained in Figures of interest
   • Select that Dataset, and run the script, pasting the IDs which include the IDs of the Figures with images already included in the Dataset
   • The script successfully adds the new images, but reports an error of constraint violation (as it tries to link the images which are already contained in the Dataset)

I would suggest maybe we could check for this and not throw the error, instead not trying to insert the images which are already present ? Otherwise the script is strictly dependent on being run on a fresh Dataset, which might not be what the user wants.

Edit: A case where a user has two Figures which contain non-zero cross section of images between them, the script as it stands would assume that all the images and figures are handled in one single step. What will the Figure_images_to_Dataset.py script do if the images are missing ? (for example because they were already moved into another group as they were contained in another Figure)

[pwalczysko]

The second script Dataset Images to New Figure fails on https://merge-ci.openmicroscopy.org/web/figure/file/69025 with

serverStackTrace = ome.conditions.ApiUsageException: No valid permissions available! DUMMY permissions are not intended for copying. Make sure that you have not passed omero.group=-1 for a save without context

Possibly because that figure has something special about it as being served from IDR ? See the baseurl output below

omero obj get FileAnnotation:69025
Using session for user-3@merge-ci-devspace.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: read-only-1
description={"name": "Images from idr0077", "imageId": 9836841, "baseUrl": "https://idr.openmicroscopy.org/webgateway"}
file=OriginalFile:554061
id=69025
name=
ns=omero.web.figure.json
version=

[will-moore]

@pwalczysko Thanks for that...

As with most scripts, the user will need to follow a guide, since the UI is pretty limited so you need to have an idea what you're doing. Since the Figures that the user wants to work with can't be easily determined via code (e.g. I think that choosing ALL the figures a user owns is a bit of an edge case), I think they will have to manually choose their figures and note the IDs. Doing this is pretty trivial compared to noting IDs for all the images in the figures.

The commits above improves the feedback when you enter invalid Figure IDs (not comma-separated) for both scripts. Screenshot shows what that looks like (bottom row).

However, I found the failure to link Images into the Dataset (either the Image is in the wrong Group, or the Image is already in the Dataset) to be OK already. This fails silently and you just see this in the Info log:

Image 149215 not linked to Dataset. Link exists or permissions failed

In the screenshot, I'd moved the image from one figure into another group and ran the script - the image was ignored (other images got linked OK). Then I ran the script again with same figures. This time, no images got linked as they were already linked. I'm not sure where you're seeing a ConstrainViolation. I can't see where Figure_Images_To_Dataset.py would give that?

Yes, https://merge-ci.openmicroscopy.org/web/figure/file/69025 will certainly fail since those images are on IDR and you can't link them.

[pwalczysko]

I'm not sure where you're seeing a ConstrainViolation. I can't see where Figure_Images_To_Dataset.py would give that?

As discussed, the ConstraintViolation is in the exclamation mark marked field of the Activities and is not governed by the script. Imho it is fine to have the error there, as long as the script does not stop and finishes all the Image linking which it can do.

The other improvements work as expected, the input warning is useful. Also, I was able now to recreate new figures with the whole workflow including duplication.

[will-moore]

@pwalczysko do you want to copy these scripts to https://github.com/ome/omero-guide-figure/pull/13 now that we're happy with them?