search

ome / ome-model

OME model (specification, code generator, implementation)
Other
13 stars 25 forks source link

2008-09 to 2009-09: Move CustomAttributes transforms to top level #145

Closed dgault closed 3 years ago

dgault commented 3 years ago

This is a follow up to #142 and should see the unit tests in ome/bioformats#3695 turn green.

The issue appears to have been due to CustomAttributes being able to be located in 4 different locations:

top level under OME under images under images/feature under dataset The non top level CustomAttributes were being transformed into XMLAnnotations but embedded in the original element, be it Image or Dataset, this is an invalid location for it and thus it was not being recognised when converted back to an OME-Model object. This PR aims to move all CustomAnnotations to the top OME level before transforming them. If you have multiple CustomAttributes at different levels then you will get multiple XMLAnnotations inside a single StructuredAnnotations.

Fixes #142

imagesc-bot commented 3 years ago

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/bio-formats-returns-all-metadata-except-ome-meta-data/52295/14

sbesson commented 3 years ago

Quickly tested using a few examples including the one from the original thread and the first impression is positive. For instance using the representative file from the original image.sc thread:

(base) sbesson@ls30630:~ $ diff cells_6.6.1.ome.xml cells_pr145.ome.xml 
13a14,76
>    <StructuredAnnotations>
>       <XMLAnnotation ID="Annotation:1">
>          <Value>
>             <OriginalMetadata ID="OriginalMetadata:0" Name="Acquisition Parameters/Acquisition GUID" Value="{2329CC5F-23E5-4D61-8AB1-526F349A3C9E}"/>
>             <OriginalMetadata ID="OriginalMetadata:1" Name="Acquisition Parameters/Channel 1" Value="Fluorescence                                                    "/>
>             <OriginalMetadata ID="OriginalMetadata:2" Name="Acquisition Parameters/Channel 1 Dye" Value="FITC                                                            "/>
>             <OriginalMetadata ID="OriginalMetadata:3" Name="Acquisition Parameters/Channel 2" Value="Fluorescence                                                    "/>
>             <OriginalMetadata ID="OriginalMetadata:4" Name="Acquisition Parameters/Channel 2 Dye" Value="Cy3                                                             "/>
>             <OriginalMetadata ID="OriginalMetadata:5" Name="Acquisition Parameters/Confocal Aperature-Ch2" Value="95.0�m"/>
>             <OriginalMetadata ID="OriginalMetadata:6" Name="Acquisition Parameters/Confocal Aperature-Ch3" Value="105.0�m"/>
>             <OriginalMetadata ID="OriginalMetadata:7" Name="Acquisition Parameters/Date" Value="02-17-2004"/>
>             <OriginalMetadata ID="OriginalMetadata:8" Name="Acquisition Parameters/DelayToFirstImagePixelInSecs" Value="0.02144"/>
>             <OriginalMetadata ID="OriginalMetadata:9" Name="Acquisition Parameters/Frame Filter" Value="2 frame Kalman Filter"/>
>             <OriginalMetadata ID="OriginalMetadata:10" Name="Acquisition Parameters/Gain Ch2" Value="1.000000"/>
>             <OriginalMetadata ID="OriginalMetadata:11" Name="Acquisition Parameters/Gain Ch3" Value="1.000000"/>
>             <OriginalMetadata ID="OriginalMetadata:12" Name="Acquisition Parameters/Intensity Name" Value="Intensity"/>
>             <OriginalMetadata ID="OriginalMetadata:13" Name="Acquisition Parameters/Intensity Resolution" Value=" 1"/>
>             <OriginalMetadata ID="OriginalMetadata:14" Name="Acquisition Parameters/Intensity Units" Value=" "/>
>             <OriginalMetadata ID="OriginalMetadata:15" Name="Acquisition Parameters/Laser-ND HeNe-G" Value="50.1% transmittance"/>
>             <OriginalMetadata ID="OriginalMetadata:16" Name="Acquisition Parameters/Laser-ND M_Ar2" Value="12.3% transmittance"/>
>             <OriginalMetadata ID="OriginalMetadata:17" Name="Acquisition Parameters/Laser-Shutter HeNe-G" Value="Open"/>
>             <OriginalMetadata ID="OriginalMetadata:18" Name="Acquisition Parameters/Laser-Shutter HeNe-R" Value="Closed"/>
>             <OriginalMetadata ID="OriginalMetadata:19" Name="Acquisition Parameters/Laser-Shutter LD405" Value="Closed"/>
>             <OriginalMetadata ID="OriginalMetadata:20" Name="Acquisition Parameters/Laser-Shutter M_Ar1" Value="Closed"/>
>             <OriginalMetadata ID="OriginalMetadata:21" Name="Acquisition Parameters/Laser-Shutter M_Ar2" Value="Open"/>
>             <OriginalMetadata ID="OriginalMetadata:22" Name="Acquisition Parameters/Laser-Shutter M_Ar3" Value="Closed"/>
>             <OriginalMetadata ID="OriginalMetadata:23" Name="Acquisition Parameters/Magnification" Value="40X"/>
>             <OriginalMetadata ID="OriginalMetadata:24" Name="Acquisition Parameters/Objective Lens" Value="UPLAPO 40XO"/>
>             <OriginalMetadata ID="OriginalMetadata:25" Name="Acquisition Parameters/Offset Ch2" Value="9"/>
>             <OriginalMetadata ID="OriginalMetadata:26" Name="Acquisition Parameters/Offset Ch3" Value="15"/>
>             <OriginalMetadata ID="OriginalMetadata:27" Name="Acquisition Parameters/PMT Voltage Ch2" Value="510"/>
>             <OriginalMetadata ID="OriginalMetadata:28" Name="Acquisition Parameters/PMT Voltage Ch3" Value="595"/>
>             <OriginalMetadata ID="OriginalMetadata:29" Name="Acquisition Parameters/Pan X" Value="76"/>
>             <OriginalMetadata ID="OriginalMetadata:30" Name="Acquisition Parameters/Pan Y" Value="0"/>
>             <OriginalMetadata ID="OriginalMetadata:31" Name="Acquisition Parameters/Scan Speed" Value="8000"/>
>             <OriginalMetadata ID="OriginalMetadata:32" Name="Acquisition Parameters/Scan Start Mode" Value="Scan started immediately"/>
>             <OriginalMetadata ID="OriginalMetadata:33" Name="Acquisition Parameters/ScanMode" Value="Standard Scan"/>
>             <OriginalMetadata ID="OriginalMetadata:34" Name="Acquisition Parameters/ScanRotationAngle" Value="1�"/>
>             <OriginalMetadata ID="OriginalMetadata:35" Name="Acquisition Parameters/SecondsPerScanLine" Value="0.010368"/>
>             <OriginalMetadata ID="OriginalMetadata:36" Name="Acquisition Parameters/Sequential Scanning Mode" Value="Line-sequential"/>
>             <OriginalMetadata ID="OriginalMetadata:37" Name="Acquisition Parameters/SourceOfData" Value="PMT 2 PMT 3 "/>
>             <OriginalMetadata ID="OriginalMetadata:38" Name="Acquisition Parameters/System Configuration" Value="FV500"/>
>             <OriginalMetadata ID="OriginalMetadata:39" Name="Acquisition Parameters/Time" Value="11:54:50"/>
>             <OriginalMetadata ID="OriginalMetadata:40" Name="Acquisition Parameters/XY Resolution-Ch2" Value="0.23�m"/>
>             <OriginalMetadata ID="OriginalMetadata:41" Name="Acquisition Parameters/XY Resolution-Ch3" Value="0.25�m"/>
>             <OriginalMetadata ID="OriginalMetadata:42" Name="Acquisition Parameters/Z Resolution-Ch2" Value="0.57�m"/>
>             <OriginalMetadata ID="OriginalMetadata:43" Name="Acquisition Parameters/Z Resolution-Ch3" Value="0.62�m"/>
>             <OriginalMetadata ID="OriginalMetadata:44" Name="Acquisition Parameters/Zoom Size" Value="3.000000"/>
>             <OriginalMetadata ID="OriginalMetadata:45" Name="Display Settings/DisplayMode" Value=" 545"/>
>             <OriginalMetadata ID="OriginalMetadata:46" Name="Display Settings/DisplayZoom" Value=" 1"/>
>             <OriginalMetadata ID="OriginalMetadata:47" Name="Display Settings/EnabledSourceForView0" Value=" 1 2"/>
>             <OriginalMetadata ID="OriginalMetadata:48" Name="Display Settings/EnabledSourceForView1" Value=" 2"/>
>             <OriginalMetadata ID="OriginalMetadata:49" Name="Display Settings/Gamma 0" Value="1"/>
>             <OriginalMetadata ID="OriginalMetadata:50" Name="Display Settings/Gamma 1" Value="1"/>
>             <OriginalMetadata ID="OriginalMetadata:51" Name="Display Settings/NumberOfViews" Value=" 1"/>
>             <OriginalMetadata ID="OriginalMetadata:52" Name="Display Settings/NumberOfViewsAvailable" Value=" 2"/>
>             <OriginalMetadata ID="OriginalMetadata:53" Name="Intensity Mapping/Map Ch0: Range" Value="00000 to 04095"/>
>             <OriginalMetadata ID="OriginalMetadata:54" Name="Intensity Mapping/Map Ch1: Range" Value="00000 to 04095"/>
>             <OriginalMetadata ID="OriginalMetadata:55" Name="Version Info/FLUOVIEW Version" Value="Version 4.3.53, w/ TIEMPO"/>
>             <OriginalMetadata ID="OriginalMetadata:56" Name="Version Info/File Version" Value="4."/>
>          </Value>
>       </XMLAnnotation>
>    </StructuredAnnotations>

The CustomAttributes previously lost during the transformation is now successfully converted into an XMLAnnotation. I will perform additional testing on the representative samples.

Since the StructuredAnnotation is a top-level element, my only concern at this stage is that the transformation might still be losing the relationship with the parent element, in the use case above this would be the Image.CustomAttributes relationship. Do you think creating aAnnotationRef as part of the transform would be feasible?

sbesson commented 3 years ago

A few failures in https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-test-folder/127065. Additionally testing locally gives the unfortunate error

cvc-pattern-valid: Value 'CustomAttributesAnnotation:1' is not facet-valid with respect to pattern '(urn:lsid:([\w\-\.]+\.[\w\-\.]+)+:Annotation:\S+)|(Annotation:\S+)' for type 'AnnotationID'.

so only LSID-style prefixes seem to be valid