Staged image sections not correctly displaying (rotation, border detection) [Was: Image upload and breakdown issues]

[juliah3]

I am having a few issues with uploading images, the thumbnail, and the automatic division calculation of staged images. The image breakdown does not show the correct quadrants from the staged image, and frequently cuts off some or all of the information in the images.

My workflow:

Upload new image with 'New image' task. I am uploading JPG images with sizes around 1-3 MB. Dimensions 3024x4032 pixels.

When I choose or drop an image, the first red flag is that the thumbnail of the uploaded image is rotated. My images are usually taller-than-wide, and end up rotated into the thumbnail 90 degrees counter-clockwise. (I have rotated the original taller-than-wide image three times to upload all four orientations in order to compare. Interestingly, the thumbnail has the same orientation no matter the orientation of the original file. (Yes, this means sometimes the thumbnail is a 180degree rotation!) But the image viewer that opens if I click on the thumbnail has the original correct orientation, but set dimensions (wider than tall no matter the original dimensions). See screen shots.)

Then I fill out Author/creator, Licenses, Repository, Preparation, and Taxon determination (pressing add), and the Staged image setup.
I use 'Cross', 'Red', and four quadrants clockwise from upper right: 'Image registration', 'Annotated specimen', 'Identifier', and 'Labels'. My understanding from comparing with an experienced colleague is that the image breakdown should later show 'Annotated specimen', 'Identifier', and 'Labels', but not 'Image registration'. Then I apply attribution and depiction summary.

After uploading, I go to the 'Collection object image breakdown TODO list'. I click on the hexagon, and Recalculate.
Here, if I click on 'Original image' at the top, the original-orientation, original-dimension image is opened in a new tab. That's good.

However, the calculated breakdown does not show the correct quadrants. With a normal image, it shows the 'Image registration', 'Labels', and 'Identifier', instead of the desired 'Annotated specimen', 'Labels', and 'Identifier'.

Remember how I tested it with an image rotated all four directions, and the thumbnail ended up with a consistent orientation? Importantly, for each of the four test images, when the breakdown is calculated, the same image contents are shown, though the original orientations are different each time. So, I see my collection labels, my unique identifier label, and my image registration scalebar/color chip, but never the specimen. See screenshot of the original (upright) orientation (left) and one of the rotations.

** My hypothesis for this is that the quadrant IDs are calculated using the thumbnail (rotated) images, rather than the originals. Then, because my thumbnails are weirdly rotated, the breakdown selects the wrong quadrants.

Additionally, sometimes the breakdown cuts off a small slice of the image, so that the edge of the image is not at the red tape border, but a little further in. Occasionally it cuts off a piece of the label, as well.

Additionally, (though probably as a result of the rotation coordinate issues), the 'drag box to ocr' does not work for me. It shows an entirely different part of the image, not in the grey box.

To summarize. 1) The thumbnail created when I upload images is rotated. I'm not sure why this is. Any help/suggestions would be very welcome. 2) The staged image breakdown mis-identifies the quadrants. I believe the majority of this and downstream issues comes from the thumbnail rotation issue. Maybe could be fixed by assigning quadrant IDs for staged images to the original image, not thumbnail.

I have confirmed this happens in both Chrome and Microsoft Edge (on Windows). Working in a project on https://sfg.taxonworks.org/, not a sandbox.

[mjy]

@juliah3 thanks for the detailed report. What camera is creating the picture? Almost certainly it's an iPhone, newish model?

[juliah3]

Actually no, it's a Samsung Galaxy S8. Just using the in-built phone app.

[LocoDelAssembly]

Could be an EXIF rotation problem @mjy? @juliah3 Does GitHub allow you to upload the images here? Would be nice to see one as originally taken and another rotated with the same method you used.

[mjy]

Almost certainly the issues are rotation related, and yes EXIF related. We will explore, but I fear it might require a pre-rotation save before loading, or, perhaps an option to rotate on load in TW.

[juliah3]

Hmm, yes, it makes sense that it has to do with the orientation the camera registers. In my experimentation above, I was using images rotated on my computer's Microsoft Photos app. But I just tried taking two new photos, one landscape and one portrait on my phone, and they both had the exact same thumbnail issues as the computer-rotated ones.

We've reached the limits of my technical know-how, but in the image Properties menu it says EXIF version 0220 if that's helpful.

I think I can send you the images. Original is red cross with upright orientation, rotated on my computer to be sideways. Both the Green cross ones were taken on my phone, on in landscape, one portrait, and have not been rotated later.

[LocoDelAssembly]

Very varying results:

$ for f in *.jpeg; do echo $f; identify -format "%[EXIF:*]" $f | grep Orientation; echo; done
1.jpeg
exif:Orientation=6
exif:thumbnail:Orientation=6

2.jpeg
exif:Orientation=3
exif:thumbnail:Orientation=6

3.jpeg
exif:Orientation=6
exif:thumbnail:Orientation=6

4.jpeg
exif:Orientation=3
exif:thumbnail:Orientation=3

Not only orientation can be 6 or 3 but also the thumbnail could use a different orientation as well (2.jpeg).

[mjy]

@LocoDelAssembly Probably an issue for Sqed gem then, check exif and pre-rotate prior to processing? Perhaps also thumbail processing on TW side (maybe this is the impetus to move to ActiveStorage).

[tmcelrath]

@juliah3 I have run across this issue. There are two workarounds for Mac and PC respectively. In Mac, you just have to open them in Preview and Save them to get them to save rotation data. In PC, you need to download an EXIF rotation tool. I've had this problem before and can give more specifics.

[juliah3]

@tmcelrath That would be fantastic, thank you! I'm working on a PC.

I'm not familiar with EXIF rotation other than a few quick google searches. So I don't yet understand what TaxonWorks needs or what to change on my images to have them work correctly.

[tmcelrath]

https://savolai.net/software/jpeg-autorotate/

basically when an image is taken and rotated, one of two things happens: 1) all the pixels are rotated 2) EXIF tags are added

TW currently has some sort of issue importing EXIF tags properly. So, use Windows File Explorer to rotate your files to whatever orientation you like (which applies an EXIF tag), then use the above program to auto-rotate the pixels in the image so that TW imports it like you are viewing it in Windows.