search

ome / omero-iviewer

An OMERO.web app allowing to view images
https://www.openmicroscopy.org/omero/iviewer/
Other
19 stars 30 forks source link

Mirroring support #439

Closed barrettMCW closed 1 year ago

barrettMCW commented 1 year ago

We co-register our WSIs against MRIs. Being able to mirror the image is pretty important for that. I did a little CSS trickery then modified pixel values based off what way it's flipped. DragPan, ShapePopup, and Drawing in general work identically when mirrored. (as far as I can tell) Thanks

will-moore commented 1 year ago

Hi, apologies for not reviewing sooner. Can you briefly say what I'm expecting to see when mirroring is enabled? A button or popup option or is the mirroring simply enabled always? Thanks,

barrettMCW commented 1 year ago

No problem! Enabling mirroring adds two buttons to the top left set of buttons, under the plus and minus. These buttons will flip the viewport in the x or y axis with css and modifies pointer event coords as needed. The mirroring buttons themselves are enabled by adding ENABLE_MIRRORING=true to initparams or setting omero.web.iviewer.enable_mirror=true in configs(url) mirrored-drawing

will-moore commented 1 year ago

Ah, yes - it's working now. I though I had the config set but it turned out I didn't....

In general this looks great...

The only significant issue I've seen so far is that when I click a different image thumbnail on the left to open a new Image, that doesn't show the mirror control buttons. It's like those buttons only get shown correctly when the page loads, not when each image loads.

I also noticed that when I drag (pan) the image while the Shape pop-up is showing, the Shape pop-up movement is not reflected, so if the image is reflected then the pop-up flies in the wrong direction. But this is very minor.

barrettMCW commented 1 year ago

Good catch! I figured out how to properly add a control (with globals). I have been using BirdsEye as a guide, but I can't seem to figure out what's going on there. As far as I can tell birdseye is a non-default control, but I can't find where it gets added to the MDI images (or any image that isn't the init_requested.) Any ideas on where that happens or another place I should add the control? I was not able to replicate the shape popup issues. I used to have them (when I recorded the gif) but I added some code to hopefully fix that before my initial commit. Is it persisting on your end? Sorry about the true != 'True' thing, it got me too lol

barrettMCW commented 1 year ago

found it!

barrettMCW commented 1 year ago

Ah I see what you mean. I wasn't able to find an obvious place to fix. I'll make a pr if I find something in the future. Thanks!

pwalczysko commented 1 year ago

I have a problem with a follow-up function which is not taking into account the flipped image.

  1. open an image
  2. flip it
  3. go File > Save Viewport as png
  4. observe that the resulting png is not flipped (the flipping got ignored)

Not sure that this is a blocker, but a user will certainly be surprised.

pwalczysko commented 1 year ago

Similarly to https://github.com/ome/omero-iviewer/pull/439#issuecomment-1381897677 , when a Projection is made out of the flipped image, the Projection is non-flipped (although the Preview of it is flipped)

pwalczysko commented 1 year ago

Thinking and playing further, I think that the Save Viewport... behaviour is not very good (say, worse than the projection) in this sense:

  1. Open and image, and zoom in and rotate it using either the top-left controls for zooming or mouse wheel and the mouse drag for rotation
  2. Then File > Save Viewport...
  3. Observe that the rotation is respected in the saved PNG
  4. Flip the image
  5. File > Save Viewport...
  6. Observe the flipping is ignored

Thus, some controls in the top-left get respected, some ignored... Not explicable in intuitive way...

barrettMCW commented 1 year ago

Great finds! I agree, save viewport should do that. I got saving mirrored canvases to work, so that will be coming. Gonna finish up that then I'll download some z-stack data to look at projections.

barrettMCW commented 1 year ago

Saving mirrored viewport works now! I was looking at the projections and it seems doable but I think maybe we shouldn't save it as mirrored directly. The other controls do not effect how the projection saves. I want to add a "ground truth" option in the future that saves an orientation (zoom, rotation, mirror) to move to when opening that image. ( so once we coreg we view the slides in the proper orientation by default ) I think it would be better to add that and copy over current orientation as ground truth?

pwalczysko commented 1 year ago

Saving mirrored viewport works now! I was looking at the projections and it seems doable but I think maybe we shouldn't save it as mirrored directly. The other controls do not effect how the projection saves. I want to add a "ground truth" option in the future that saves an orientation (zoom, rotation, mirror) to move to when opening that image. ( so once we coreg we view the slides in the proper orientation by default ) I think it would be better to add that and copy over current orientation as ground truth?

Thank you @barrettMCW , this all sounds very reasonable and encouraging to me.

And yes, the Save Viewport... is working as expected. I am suggesting that this great improvement should be merged as is - the bird's eye view is of course good to have, but if it is not soon-to-solve problem, it seems to me that the benefits outweight the drawbacks.

barrettMCW commented 1 year ago

Thank you! Do you mean shape popup? I didn't see any problems with birdseye.

pwalczysko commented 1 year ago

Thank you! Do you mean shape popup? I didn't see any problems with birdseye.

No, I mean bridseye view, bottom-right in central pane of iviewer (on many of the screenshots here in this PR, the birdseye view is collapsed, you just have to click onto the arrow in bottom-right to expand it).

Screenshot 2023-01-16 at 15 43 07

You can see that if I flip the image, the birdseye is still in original orientation. Now, when I use the birdseye to "jump" between places on a large image, the discrepancy between the two views is of course confusing. I have also checked that e.g. rotation of the image rotates the birdseye view accordingly. I was thinking you were discussing it with @will-moore in https://github.com/ome/omero-iviewer/pull/439#issuecomment-1379526368 ?

barrettMCW commented 1 year ago

I was looking at the non-flipped birdseye as a feature rather than a bug, as the red box is correct, although now seeing that rotating the image rotates the entire birdseye makes me feel I should flip it. Shouldn't be hard

barrettMCW commented 1 year ago

Ahh, that was me talking about trying to copy how birdseye was implemented to make sure it's always added. But yeah I'll change that quick, good catch

pwalczysko commented 1 year ago

Ummm, really sorry for not discoverign it earlier. When I rotate&flip horizontally then the panning can become very hard. I can record a video if need be, but maybe you can reproduce ? Just rotate some 45 degrees, then flip horizontally and try to pan please.

barrettMCW commented 1 year ago

ickyyyy, i see it. Birdseye is almost good then I'll fix that. I've got a couple ideas churning so I think it'll be doable

barrettMCW commented 1 year ago

I have working mirrored rotation but I am not happy about it. I unrotate the current and previous coords, mirror new coords and redo the rotation on new coords. It works, barring light stuttering and the occasional jump to a side then jump back. I'm sure there's a faster way to calculate it without unrotating it but I only took trig once in highschool :/ There is an if statement so performance should only suffer while rotated

will-moore commented 1 year ago

@barrettMCW I just found this: https://stackoverflow.com/questions/63638347/inverting-the-y-axis, and the codesandbox link there is working. I wonder if that approach for flipping will address some of these issues you're seeing since OpenLayers may understand at a lower level that the image is inverted/mirrored?

I wouldn't worry about save Projection handling the flipped image - that seems like really edge case.

A couple of other places you may want to handle the mirrored state:

But these are not blockers, so it's up to you (could also come in a follow-up PR)

barrettMCW commented 1 year ago

I remember seeing that codepen from way back when I started this, but I couldn't figure it out. I tried again, but no luck. I think the problem is that I'm doing addCoordinateTransforms(view.getProjection(), new Projection...) instead of (tiles.getProjection(), view.getProjection()), but I cannot find the OmeroImage or Ol3.Tile projections. If our image layer doesn't have a projection I'm not sure that method will work. Cache and Viewport links are good adds, as that's probably the mechanism I'll use for ground truth. I'll get started on that shortly

barrettMCW commented 1 year ago

I think this is it. Cached settings works. ViewportAsUrl probably works. I didn't build and put it on a test server to test the url itself but it recognizes the index-dev.html initparams and builds a url with mirror parameters, which I imagine means it will work. I discovered rotating is a little wonky as well. I think it's much more manageable than the other things I correct for. I think we just leave it for now and I'll take another crack at mirroring with openlayers when I start base truth.

pwalczysko commented 1 year ago

I think this is it. Cached settings works. ViewportAsUrl probably works. I didn't build and put it on a test server to test the url itself but it recognizes the index-dev.html initparams and builds a url with mirror parameters, which I imagine means it will work. I discovered rotating is a little wonky as well. I think it's much more manageable than the other things I correct for. I think we just leave it for now and I'll take another crack at mirroring with openlayers when I start base truth.

Thanks a lot. Testing on our server has shown that:

  1. the cached settings work, as you indicated, the workflow is to open an image, flip it, switch off a channel, click on another thumb in LHP of iviewer and then click back onto the first image thumb -> should still be flipped and channel off, works as expected
  2. the url does not work, https://merge-ci.openmicroscopy.org/web/webclient/img_detail/105665/?dataset=25019&z=0&t=0&x=33059&y=27449&zm=34&c=-1|0:255$FF0000,2|0:255$00FF00,3|0:255$0000FF&m=c&maps=[{"inverted":{"enabled":false},"quantization":{"family":"linear","coefficient":1}},{"inverted":{"enabled":false},"quantization":{"family":"linear","coefficient":1}},{"inverted":{"enabled":false},"quantization":{"family":"linear","coefficient":1}}]&fx=true&fy=false is the link I am producing from the flipped and zoomed-in image, when I paste it in a new tab, I get a non-flipped image
will-moore commented 1 year ago

Testing with the dev build

enableMirror: "True"
initialFlipX: true
initialFlipY: null

and when built, with URL containing &fx=true&fy=false

enableMirror: "True"
initialFlipX: "true"
initialFlipY: "false"

So it looks like you need to convert the "true" string into true boolean

barrettMCW commented 1 year ago

Sorry about that, using a string bool for initparam works. Thanks for your help guys!

pwalczysko commented 1 year ago

Confirming that Get link to viewport now works with the mirroring as expected.

Ready to merge fmpov.

will-moore commented 1 year ago

@barrettMCW Apologies for the delay, but this has now been released in omero-iviewer 0.13.0. Thanks for your contribution!