Allow to submit the orientation of the image

[cgendreau]

@dshorthouse could you advise on a name and the type of values that we should record regarding the orientation of the primary image. The main goal is to allow to generate derivatives later so the system needs to know the rotation to apply to the primary object.

[dshorthouse]

Just did a quick look around. There's cause for confusion here with respect to the orientation of the image (i.e. ascertained with an image sensor) vs the orientation of the subject in the image (i.e. as expressed in Audubon Core) – the two are not always the same. If we assume the intent here is is to permit a machine/script to rotate the image in its present state to a more desirable orientation (perhaps because there's human-readable labels in the image), the best bet is to make use of the EXIF property called, Orientation. In the case of all the cr2 files from conveyor belt, as near as I can tell, the Orientation value is not populated (= Undefined) if you were to execute an imagick command:

$ identify -verbose 78e160ab-4f21-4a46-accf-ef9b679059b4.cr2

  Base filename: 78e160ab-4f21-4a46-accf-ef9b679059b4.cr2
  Format: CR2 (Canon Digital Camera Raw Image Format)
  Class: DirectClass
  Geometry: 8736x5856+0+0
  Units: Undefined
  Colorspace: sRGB
  Type: TrueColor
  Base type: Undefined
  Endianess: Undefined
  Depth: 16-bit
  Channel depth:
    Red: 16-bit
    Green: 16-bit
    Blue: 16-bit
  Channel statistics:
    Pixels: 51158016
    Red:
      min: 120  (0.00183108)
      max: 64963 (0.991272)
      mean: 46617.3 (0.711334)
      standard deviation: 14037.1 (0.214193)
      kurtosis: -0.441852
      skewness: -0.94607
      entropy: 0.94517
    Green:
      min: 386  (0.00588998)
      max: 63459 (0.968322)
      mean: 40184.6 (0.613177)
      standard deviation: 13629.8 (0.207978)
      kurtosis: -0.685043
      skewness: -0.847488
      entropy: 0.946843
    Blue:
      min: 0  (0)
      max: 65535 (1)
      mean: 38887.8 (0.59339)
      standard deviation: 16598.1 (0.253271)
      kurtosis: -0.958208
      skewness: -0.817879
      entropy: 0.937133
  Image statistics:
    Overall:
      min: 0  (0)
      max: 65535 (1)
      mean: 41896.6 (0.639301)
      standard deviation: 14755 (0.225147)
      kurtosis: -0.563051
      skewness: -0.862854
      entropy: 0.943049
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Matte color: grey74
  Background color: white
  Border color: srgb(223,223,223)
  Transparent color: none
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 8736x5856+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: Zip
  Orientation: Undefined
  Properties:
    date:create: 2021-05-04T14:10:58+00:00
    date:modify: 2021-05-04T14:10:58+00:00
    dng:Aperture: F/8.0
    dng:AspectRatio: 1.491803
    dng:ChannelMultipliers: 2.120117 1.000000 1.805664 1.000000
    dng:CreateID: 1
    dng:Crop: 0 0 8736 5856
    dng:EXIFSource: DCRaw
    dng:ExposureNorm: 32891
    dng:FocalLength: 50.0 mm
    dng:Green: 1.003
    dng:ISOSpeed: 100
    dng:Make: Canon
    dng:MatrixInputProfile: Color matrix
    dng:Model: EOS 5DS R
    dng:Orientation: 0
    dng:Rotation: 0.000000
    dng:Shutter: 1/49.4 s
    dng:sRGBOutputProfile: sRGB
    dng:Temperature: 4894
    dng:Timestamp: Wed Nov  6 10:21:43 2019
    dng:WB: Camera WB
    dng:WBFineTuning: 0
    png:IHDR.bit-depth-orig: 16
    png:IHDR.bit_depth: 16
    png:IHDR.color-type-orig: 2
    png:IHDR.color_type: 2 (Truecolor)
    png:IHDR.interlace_method: 0 (Not interlaced)
    png:IHDR.width,height: 8736, 5856
    png:sRGB: intent=0 (Perceptual Intent)
    png:text: 2 tEXt/zTXt/iTXt chunks were found
    signature: 1d1ceff5ea3025cefff8bdecf55736ebab43a74bfbf9962bb78b05cc3f171767
    Software: UFRaw
    Source: CanonEOS 5DS R
  Artifacts:
    verbose: true
  Tainted: False
  Filesize: 233.771MiB
  Number pixels: 51.158M
  Pixels per second: 20.0126MP
  User time: 2.550u
  Elapsed time: 0:03.556
  Version: ImageMagick 7.0.9-16 Q16 x86_64 2020-01-12 https://imagemagick.org

Ideally, Orientation would be 8 (= LeftBottom) and not Undefined for ALL the conveyor belt images (see the link above), which would make creation of derivatives a breeze because there are libraries that honour and use that flag. However, we should never write any new metadata into those cr2 files for fear of corrupting them.

So...

Orientation is the term, ideally in the EXIF but if if cannot be there then a stand-alone term with integer values to match the intent of what software normally does when it reads that EXIF term as follows:

[dshorthouse]

I suppose the next question is, would you like me to add Orientation: 8 to the sidecar files? The potential annoyance however is that this would only apply to the cr2 and not the highres jpg referenced in the sidecar. That jpg will be rotated prior to submission because OCR needs to be run on them.

[cgendreau]

The goal is indeed to identify the orientation of the subject in the image and not the image itself (e.g. landscape). Not sure how the camera could know that unless entered by the user using the menu I guess.

Anyway, I would prefer to rely on the a value that we set. The fact that it's only for the cr2 is not a problem since we can't add values like that to derivatives anyway.

[dshorthouse]

The goal is indeed to identify the orientation of the subject in the image and not the image itself (e.g. landscape).

Not so sure I agree with that, although it's a hair-splitting exercise. The "subject" in an image, depending on intent, narrowness of its definition, and segmentation techniques might very well be multiple of them in varying orientations. For example, it's not uncommon for labels on a botanical specimen sheet to be in numerous orientations (humans affixed them with glue after all), the specimen itself in other orientations (if there can be one), and the image proper in yet another orientation.

In the case of the mechanically-produced conveyor belt images, they are all precisely (or nearly so) the same dimensions with a long axis and a short axis & the botanical specimen typically aligns with the long axis. Likewise, the labels are usually parallel to the short axis. And so, a landscape image is undesirable because the specimen sheet itself in its entirety is almost always examined by humans in portrait.

What this might beg for is a front-end utility that allows a user to rotate an image in 90 degree increments until they've produced a derivative that they desire. This is particularly true for images of insects. In the short-term & with these particular instances of herbarium sheets, we can hard-code an Orientation: 8 in the metadata.

[As an aside, cameras do in fact record the orientation of the body and do write that in the EXIF. That's why import of those images into photo editing software magically re-orients the image should you have taken a landscape vs a portrait image. But, when the body of the camera is mounted facing downward, all bets on orientation are off.]

[cgendreau]

I would add that the examples in Audubon Core for subject Orientation are: "dorsal", "ventral", "frontal".

The main goal is still "to allow to generate derivatives later so the system needs to know the rotation to apply to the primary object".

Considering this is combined with a requirement to download multiple images in a single request I think we can see the ui for it as a nice to have.

Orientation: 8 is fine as long as we have a source to match the magic number

[dshorthouse]

Indeed, all the reason to have two separate goals here and two separate solutions. Orientation of the image (the present ticket) and orientation of the subject (i.e. Audubon core) are distinct, though partially overlapping. Concretely, if in Orientation we had biologically meaningful values like "dorsal", "ventral", "frontal", that does not help us answer, "Does this image need to be rotated?", especially via script when we cannot make a machine "see" the image and decide for itself. Those integer values for EXIF Orientation do have human-readable counterparts, but am not sure they're much help.

  Orientation: TopLeft
    exif:Orientation: 1
  Orientation: TopRight
    exif:Orientation: 2
  Orientation: BottomRight
    exif:Orientation: 3
  Orientation: BottomLeft
    exif:Orientation: 4
  Orientation: LeftTop
    exif:Orientation: 5
  Orientation: RightTop
    exif:Orientation: 6
  Orientation: RightBottom
    exif:Orientation: 7
  Orientation: LeftBottom
    exif:Orientation: 8

1 = 0 degrees: the correct orientation, no adjustment is required.
2 = 0 degrees, mirrored: image has been flipped back-to-front.
3 = 180 degrees: image is upside down.
4 = 180 degrees, mirrored: image has been flipped back-to-front and is upside down.
5 = 90 degrees: image has been flipped back-to-front and is on its side.
6 = 90 degrees, mirrored: image is on its side.
7 = 270 degrees: image has been flipped back-to-front and is on its far side.
8 = 270 degrees, mirrored: image is on its far side.

[cgendreau]

That's perfect. We just need to decide on the name (Orientation alone?) and a definition. I don't think it should be restricted to images since it could be used for a document for example.

Definition: The orientation of the instrument relative to the scene, when the media was captured. Value between 1 and 8.

[dshorthouse]

Though that definition is a bit obtuse, it's good enough. Somehow, it might be useful to link to the "F" image for humans to understand what the integers mean. Orientation sounds good to me since its use will be exactly like that for EXIF.

[kardecom]

Hi @dshorthouse
Could you please regenerate the test set on the cluster with Orientation in metadata.yml files? I would like to test it

[dshorthouse]

OK, but... I'll have already rotated the jpg, just not the cr2 source file because OCR would otherwise be pointless. So in effect, the orientation: 8 will be in reference to the cr2 & NOT its paired jpg that goes along for the ride. In other words, no action need be taken on any asset until and if a derivative might be generated from the cr2 server-side at some point in the future.

[cgendreau]

yes, it's only to send it in the object-store in the cr2 metadata.

[dshorthouse]

All set @kardecom. orientation: 8 added in the metadata.yml samples on the biocluster.