Kymograph points string float

[will-moore]

This fixes the Kymograph script when Polylines are created with floating-point coordinates (as in iviewer). Also fixes the joining of sections in the new images for Polyline kymographs. To test, use image http://web-dev-merge.openmicroscopy.org/webclient/?show=image-138052 (user-3).

• There are 2 ROIs drawn on the Image: 1 Line and 1 Polyline that matches the coordinates of the Line.

• Running the Kymograph script on this Image should give 2 very similar Kymographs. This is how I noticed that the Polyline kymograph was previously stitching each section of the Polyline together in the wrong order, fixed by reverse() in aa6e232.

  The image above was created by importing a series of 8-bit pngs with a pattern file since the original image is uint16 and the numpy array from getTile(), when converted to PIL image is basically scrambled.

[sbesson]

A couple of high-level questions:

• I suspect the issue fixed here is also present in the upstream omero.util.roi_handling_utils.points_string_to_xy_list API. What is the plan for this API: should we also fix it, leave it unmodified or deprecate it?
• in the case where we decide to override the upstream API in the scripts repo, should the corresponding upstream unit tests in test_rois.py be migrated here and expanded to cover the new use cases?
• more generally, is this worth investing into integration tests for the kymograph script so that we catch future regression

[pwalczysko]

For the follow-up workflow (Kymograph analysis) I get following output

Basically because I did not specify the timepoints, I end up with time in the units pixels and the speeds pixel per pixel. I understand where this is coming from, but shouldn't the Kymograph Analysis know about the pixels per timepoint and give me the speeds as pixels per frame ?

[will-moore]

This script should now work fine with the original unit16 image, uploaded to outreach at http://outreach.openmicroscopy.org/webclient/?show=image-28632. The script is also updated there and can be tested now.

The fixes above include the "direction" of the kymograph. As discussed with @pwalczysko: E.g. if I draw a line from A to B, the left-edge of the kymograph should correspond to A and the right-edge to B.

[pwalczysko]

Giving to a non-calibrated 5.12x512 image a calibration at the timepoint of the run of the script, namely 0.78 microns per pixel, and running the script, I end up with a Kymograph image which does seem to have a different pixel sizes in x and y directions, see http://outreach.openmicroscopy.org/webclient/?show=image-28639 and the screenshot below.

Is that logical/expected ?

[will-moore]

@pwalczysko Yes, since the pixel size Y is time (seconds?) per pixel

[pwalczysko]

The Kymograph preserves the colors of the original images, but only for plain colors. For LUTs (and also for Greyscale), the Kymograph has the colors of the original image before changes of channel colors were appliead (=green and red)

@will-moore expected ?

[pwalczysko]

Other than that it seems to work nicely. Even the direction of the time from top to bottom is validated (I can see the particles in the image to go to the top right, which is preserved in the Kymographs).

Ready to merge FMPOV

[joshmoore]

See also "A couple of high-level questions" from https://github.com/ome/scripts/pull/142#issuecomment-412792009

[will-moore]

Re: https://github.com/ome/scripts/pull/142#issuecomment-412792009

• Since omero.util.roi_handling_utils.points_string_to_xy_list is also being used by the Plot_Profile.py script, we should fix it and update the unit tests (quite easy).
• Once that fix is released, we can remove the points_string_to_xy_list from Kymograph.py. We only need to add that to the script so we can have a working script for Basel training, without waiting for OMERO release.
• We have integration tests for the Kymograph script at https://github.com/ome/scripts/blob/develop/test/integration/test_analysis_scripts.py#L96 but the test didn't fail because Polygon coordinates are not floats. The test can certainly be updated to test this and to do more checks on the Kymograph image created.

[will-moore]

The other fix here, bringing get_line_data() into the script too, and changing it to work via the renderingEngine instead of getting pixel data is a much bigger change. This works well for Kymograph where we don't care too much about absolute pixel values, but we probably don't want that change elsewhere.

[joshmoore]

@will-moore : what of the changes in https://github.com/ome/scripts/pull/142#issuecomment-413816639 do you see getting into 5.4.8?

[will-moore]

If release is August 30th (https://trello.com/c/Xp8VUnOj/120-omero-548) there should be time enough to do all that, depending on any higher priorities? I'll create a card.

[pwalczysko]

I verified on eel that if you have the original image calibrated and timestamped, the values and speeds correctly propagate into the Kymograph, Kymograph Analysis and the resulting CSV.

On outreach, I have all data imported, but wonder now:

• which folder to upload the script to ? (we had in mind another, new structure of the script folders for outreach, waht was it ?)
• or will that be a released script ?

cc @will-moore @jburel

[jburel]

I think we were just going for workshops instead of the location The script won't be released

[jburel]

@will-moore test failure in Travis

[pwalczysko]

The script won't be released

Yes, so do we put it under workshops or under analysis (instead of the released script) on outreach ? @will-moore is in favor of the latter cc @jburel

[joshmoore]

I'm confused about the above. When there's some more time, a description of the workflow would be appreciated (and it's impact for this PR).

[pwalczysko]

Other than the question https://github.com/ome/scripts/pull/142#issuecomment-415392965, I have tested both the Kymograpph and Kymograph analysis. Both lines and polylines.

Works fine

Ready to merge FMPOV

[pwalczysko]

@joshmoore

a description of the workflow would be appreciated

Workflow might mean:

• workflow of Kymograph script followed by Kymograph Analysis
• or workflow of developing server-side scripts for training purposes and its merging (or not) with the server side scripts to be released as part of OMERO releases

Which one ?

[pwalczysko]

@joshmoore For the whole Kymograph workflow, search for Kymograph in https://docs.google.com/document/d/11vnqqHTNSAi6pAtzfNy8d3-fy1VpFos0vR5kq2Mqmbw/edit#heading=h.f5zjcwbjevdi - the workflow is now ready to consume

[jburel]

Analysis is fine. 5.4.8 release is after the training so we will need to have the version used for Basel under ome/training-scripts as we have done previously and move it to this repo when it is ready for "official" inclusion Still some comments to address

Travis is still failing

[joshmoore]

My apologies, @pwalczysko, I was refering to PR/merging workflows.

[will-moore]

I opened https://github.com/ome/training-scripts/pull/37 to tag these fixes for the Basel workshop.

[joshmoore]

Thanks, @will-moore. Closing this for the moment then so it doesn't appear in the 5.4.8 builds.

[will-moore]

@joshmoore We'd still like to have this in 5.4.8. We're only porting it to training-scripts because we won't have 5.4.8 out before Basel.

[joshmoore]

i.e. the points from https://github.com/ome/scripts/pull/142#issuecomment-413816639 will be addressed this week?

[will-moore]

@joshmoore I already addressed those points (unless I missed something?) - I'll check...

[will-moore]

@joshmoore Currently the Kymograph scripts have their own copies of points_string_to_xy_list() so they can work with polylines containing floats with 5.4.7 server (as required for Basel workshop). Tests in the script repo have been updated to test this. I just opened a PR to port those changes back to the server: https://github.com/openmicroscopy/openmicroscopy/pull/5837 Which actually means that these scripts won't need their own fixes for that, once it is merged (hopefully into 5.4.8). However, the integration tests in the script repo using test-infra currently use OMERO 5.4.7, so we can't remove points_string_to_xy_list() fixes from the scripts in this PR as the tests will fail. Since this PR isn't urgent, I think we can wait for https://github.com/openmicroscopy/openmicroscopy/pull/5837 to get merged into 5.4.8 and then can remove points_string_to_xy_list() from the scripts and re-open this PR after the release.

get_line_data() in the Kymograph script now uses renderingEngine with current settings to get 8-bit data for the line, instead of getTile(), which is fine for Kymograph. I couldn't work out how to fix getTile() -> PIL for >8-bit data, so I don't have a fix get_line_data() in https://github.com/openmicroscopy/openmicroscopy/pull/5837.

[will-moore]

Recent training prep reminded us that this fix never made it into mainline (still need separate script to demo in Training workshops). cc @pwalczysko

[will-moore]

Test_infra testing of this script with OMERO 5.4.8 now passes without using a separate points_string_to_xy_list().

[pwalczysko]

Retested today again on

• eel, via web-dev-merge
• localhost, (uploaded the script here manually) with a precise copy of outreach database on the Kymograph image used for outreaches

Both tests passed.

Ready to merge FMPOV

[pwalczysko]

Um, the only question was, shoiuld there not be a bump of a version advertised in the script ? see https://github.com/ome/scripts/pull/142/files#diff-a982bdd88ae79e1dbda8de7b27678ab5R33

[will-moore]

@pwalczysko Fixed the version to 5.5.

[pwalczysko]

Version fix works as expected. Ready to merge FMPOV.