revisions of and around ANNOTATIONS

[lzehl]

I'm creating a joint issue for revisions of an around ANNOTATION schemas so that it is easier to see how they effect each other. This issue joints the discussions: #137 , #174 , #181 and the PR #179 (with #10 in openMINDS_ephys)

Here a summary of the raised issues of an around the ANNOTATION schemas and how they are related:

1) generic vs specific annotations (#137, #181): generally speaking annotations can result in delineations of volumes of interest (3D), regions of interest (2D), landmarks (coordinate point), etc. Although originally meant as a generic schema for all types of annotations, the current annotation schemas only cover metadata for certain VOIs and ROIs. Discussion: we need to either (A) modify the annotations schemas to allow for different types (my preferred option) or (B) split annotations into specific annotation type schemas (my less preferred option, because this might be a bottomless pit and would require a lot of work for EBRAINS)

2) annotation type specific attributes (#174): how to solve this is related to point 1. Different annotation types will require different attributes (e.g. for visualization). Discussion: we need to set up a list what attribute variations are needed / valuable for the specific annotation types and then either set up (A.a) small attribute schemas that can be flexibly linked into an annotation schema (if we go with A above) or (B.b) integrate them into the specific annotation type schemas (if we go with B above).

3) annotation definition in files vs in graph instances (#137, #181): Annotations currently assume that they have to be stored in a file, BUT for annotation services/software smaller user-made annotations should be also allowed to be directly stored within the database (e.g. landmarks). Discussion: we already can point to a CoordinatePoint, but we should also introduce small schemas for other geometrical objects/delineations. This should be combined with the suggested Shape schemas in openMINDS_ephys #8.

4) facilitating usage of annotations (#137, #181): Specifically for custom annotations some flexible properties are missing to make labeling/description of them easier. Discussion: revise the PEV/atlasAnnotation and CAE/customAnnotation relation. Are there properties missing? could the relations be simplified?

5) annotation vs target location (#181, #179 / openMINDS_ephys/#10): An annotation should be used to define/mark WHAT IS within an image, e.g., mark the trajectory and tip location of an electrode within the tissue sample (actual result); Discussion: We are missing a schema that allows to specify what anatomical location WAS AIMED FOR (aimed result). In some cases the latter is all a data provider can offer. Potentially it is worth therefore integrating this new schema already within core.

@UlrikeS91 @xgui3783 @skoehnen @apdavison @Peyman-N I would first like to clarify if we go with option A or B for point 1. We can focus the discussion on the other points then afterwards based on that decision.

[UlrikeS91]

Sorry about the late response to this issue! But thanks a lot for creating the joint issue. Very helpful!

1. generic vs specific annotations (#137, #181): generally speaking annotations can result in delineations of volumes of interest (3D), regions of interest (2D), landmarks (coordinate point), etc. Although originally meant as a generic schema for all types of annotations, the current annotation schemas only cover metadata for certain VOIs and ROIs. Discussion: we need to either (A) modify the annotations schemas to allow for different types (my preferred option) or (B) split annotations into specific annotation type schemas (my less preferred option, because this might be a bottomless pit and would require a lot of work for EBRAINS)

I'm also leaning towards (A). I agree that (B) will be bottomless. Only downside with A is that we need to be cautions how the schema is built up to not have to revise it too often when new use cases come in (but I think we should be able to avoid that).

2. annotation type specific attributes (#174): how to solve this is related to point 1. Different annotation types will require different attributes (e.g. for visualization). Discussion: we need to set up a list what attribute variations are needed / valuable for the specific annotation types and then either set up (A.a) small attribute schemas that can be flexibly linked into an annotation schema (if we go with A above) or (B.b) integrate them into the specific annotation type schemas (if we go with B above).

(A.a) fits also in relation to the attributes that where introduced fort he specimen. Nice and consistent - I like that 😉

3. annotation definition in files vs in graph instances (#137, #181): Annotations currently assume that they have to be stored in a file, BUT for annotation services/software smaller user-made annotations should be also allowed to be directly stored within the database (e.g. landmarks). Discussion: we already can point to a CoordinatePoint, but we should also introduce small schemas for other geometrical objects/delineations. This should be combined with the suggested Shape schemas in openMINDS_ephys #8.

Sounds good!

4. facilitating usage of annotations (#137, #181): Specifically for custom annotations some flexible properties are missing to make labeling/description of them easier. Discussion: revise the PEV/atlasAnnotation and CAE/customAnnotation relation. Are there properties missing? could the relations be simplified?

Only ones that I can see immediately are a lookupLabel and maybe additionalRemarks (at least) for the custom annotation. I'm sure there are more things that could be added. The question is which properties would fit best into the attributes as proposed in your second point and which ones are generic enough to be added on the annotation schemas directly.

5. annotation vs target location (#181, #179 / openMINDS_ephys/#10): An annotation should be used to define/mark WHAT IS within an image, e.g., mark the trajectory and tip location of an electrode within the tissue sample (actual result); Discussion: We are missing a schema that allows to specify what anatomical location WAS AIMED FOR (aimed result). In some cases the latter is all a data provider can offer. Potentially it is worth therefore integrating this new schema already within core.

I don't see why this should be part of the core repo, otherwise I'd be happy to add such a schema immediately. We should also discuss where it can be used, e.g. as a parameter for a protocolExecution? Or rather only within extensions? What use cases would we like to cover with it (and which schemas need to use this schema then)?

[tgbugs]

1. I think that it is possible to create a closed list of types that is manageable, at least in terms of the topology or basic geometry of the annotations. The only thing that is missing from my original document (pre AtOM) is that there can be annotations that have multiple disjoint volumes or surfaces (e.g. if you are looking at a plane of section through the rodent caudoputamen the two major populations would want to be two annotations not one per matrix component). https://github.com/SciCrunch/NIF-Ontology/blob/master/docs/brain-regions.org#tables

2. I think in either A.a or B.b we will want the visualization properties to be orthogonal from the point of view of the data structure. This would also cover things like attaching an ontology term to differentiate parameter vs measurement.

3. I think decoupling the annotations from their exact implementation will be quite useful in the long run, especially as it can facilitate 4. I think what will be needed is a way for people to specify how to load the annotation from whatever format into the target space or type of space (e.g. load an rdf string for a point onto Waxholm Rat v4). Shapes also useful for shanks etc. not having to store a file will be important for electrode metadata.

4. I don't think you need to differentiate those, atlasAnnotation was just what we called it. There might be a distinction between annotations that are to biological structures vs e.g. electrodes, but I don't think that would come in at the type of annotation it would probably come in at its referent?

5. What they are trying to hit should just be a semantic annotation probably? Or if they were trying to hit a stereotaxic coordinate then that would be called the target point which should be a parameter in a protocol as opposed to a result in data.

[lzehl]

Okay. Looks like we are all more or less in agreement / tend to the same direction. Let's become more concrete then. I will provide some details asap

[lzehl]

annotation types we need to handle (discussion with @tgbugs ). looking first at all aspects that define annotations

atomic types:

• point
• contour
• volume
• area
• surface

physical connection between multiple atomic types:

• connected
• disconnected

border definition:

• boolean
• probabilistic
• distance based

shape

• parametric (track only a subset of points and infer the rest)
• non-parametric (track all points)

---

ANNOTATION TYPES (each can contain multiple geometries if they belong to one annotation)

1. point
   • annotation point (2D, 3D)
2. volume
   • annotation parametric volume (3D)
   • annotation non-parametric volume (3D)
3. area
   • annotation parametric area (2D)
   • annotation non-parametric area (2D)
4. surface
   • annotation parametric surface (3D)
   • annotation non-parametric surface (3D)
5. contour
   • annotation non-parametric contour (2D, 3D)
   • annotation parametric contour (2D, 3D)

@UlrikeS91 & @skoehnen & @xgui3783 it would be nice to meet to make a concrete plan for the schema updates

[tgbugs]

A brief elaboration on the dimensions of the space. The two additional ones in addition to those above are coordinate space dimensionality and one/many.

1. coordinate space dimensionality (2d, 3d)
2. topology (point, volume, surface, etc. only one topology per annotation)
3. shape specification (parametric/non-parametric)
4. one/many (the schema should be default many i.e. an array of objects so that the distinction can mostly be ignored)
5. connected/disconnected (in the many case, do we stitch shapes or not? this could be hierarchical, but I hope that case can be avoided)
6. border type (boolean, probabilistic, distance based)

The types of annotations should be fully covered by the cartesian product over these dimensions, with the possible omission of one/many being replaced by connected/disconnected, however there are edge cases where we might actually want one/many in the type declarations.

While the cartesian product does cover the space, there may be more elaboration of subtypes, in particular for parametric contours embedded in a 3d coordinate space, e.g. line, bezier curve, spline, tree, vector, signed distance field, etc.

[xgui3783]

maybe @fsdavid is also interested in the discussion.

[xgui3783]

personally, I am very interested to see /discuss how complex types can be built on top of the atomic types.

For example, I would be very curious to find out how to construct a:

• line segment, which seem to consist of two atomic type point, and a physical connection between the two points
• closed polygons

[fsdavid]

Sure, I'm interested. I'd be happy to attend the meeting.

Also, it is interesting if it is possible to store bounding boxes and spheres in new schemas.

[lzehl]

Let's try concrete suggestions for implementing this.

Annotation concept schema (is extended by AtlasAnnotation and CustomAnnotation, see below)

{
  "required": [
    "borderType",
    "criteriaQualityType",
    "topologyType"
  ],
  "properties": {   
    "borderType": {
      "_instruction": "Add the border type of this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/controlledTerms/AnnotationType"
      ]
    },
    "criteria": {
      "_instruction": "Add the protocol execution defining the criteria that were applied to produce this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/core/ProtocolExecution"
      ]
    },
    "criteriaQualityType": {
      "_instruction": "Add the quality type of the stated criteria used to define this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/controlledTerms/CriteriaQualityType"
      ]
    },
    "inspiredBy": {
      "type": "array",      
      "minItems": 1,
      "uniqueItems": true,
      "_instruction": "Add one or several (source) files that inspired the definition of this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/core/File"
      ]
    },    
    "internalIdentifier": {
      "type": "string",
      "_instruction": "Enter the identifier used for this annotation within the file it is stored in."
    },
    "laterality": {
      "type": "array",
      "maxItems": 2,      
      "minItems": 1,
      "uniqueItems": true,
      "_instruction": "Add one or both sides of the body, bilateral organ or bilateral organ part that this annotation is defined in.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/controlledTerms/Laterality"
      ]
    },
    "preferredVisualization": {
      "_instruction": "Add preferred viewer specifications to visualize this annotation.",
      "_embeddedTypes": [
        "https://openminds.ebrains.eu/sands/ViewerSpecification"
      ]
    },
    "topologyType": {
      "_instruction": "Add the topology type of this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/controlledTerms/AnnotationTopologyType"
      ]
    }
  }
}

AnnotationBorderType instances: "deterministic", "probabilistic" AnnotationTopologyType instances: "volume", "area", "surface", "point", "contour" For Image, Area, Contour, CoordinatePoint, Surface, and Volume schema see below.

AtlasAnnotation schema

{
  "_type": "https://openminds.ebrains.eu/sands/AtlasAnnotation",
  "_extends": "annotation.schema.tpl.json",
  "required": [
    "coordinateSpace"
  ],
  "properties": {   
    "coordinateSpace": {
      "_instruction": "Add the coordinate space for this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/sands/CommonCoordinateSpace"
      ]
    },
    "specification": {
      "_instruction": "Add the non-parametric specification of this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/sands/File"
      ]
    }
  }
}

CustomAnnotation schema

{
  "_type": "https://openminds.ebrains.eu/sands/CustomAnnotation",
  "_extends": "annotation.schema.tpl.json",
  "required": [
    "coordinateSpace",
    "definedIn"
  ],
  "properties": {   
    "coordinateSpace": {
      "_instruction": "Add the coordinate space for this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/sands/CommonCoordinateSpace",
        "https://openminds.ebrains.eu/sands/CustomCoordinateSpace"
      ]
    },
    "specification": {
      "_instruction": "Add the non-parametric or parametric specification of this annotation.",
      "_embeddedTypes": [
        "https://openminds.ebrains.eu/sands/File",
        "https://openminds.ebrains.eu/sands/Point",
        "https://openminds.ebrains.eu/sands/Area",
        "https://openminds.ebrains.eu/sands/Volume",
        "https://openminds.ebrains.eu/sands/Surface",
        "https://openminds.ebrains.eu/sands/Contour"
      ]
    }
  }
}

Example annotation specifications Point schema

{
  "_type": "https://openminds.ebrains.eu/sands/Point",
  "required": [
    "coordinates"
  ],
  "properties": {
    "coordinates": {
      "type": "array",      
      "minItems": 2,
      "maxItems": 3,
      "_instruction": "Add two or three-dimensional coordinates (2D: [x, y] or [[x1, ..., xn], [y1, ...,  yn]] | 3D: [x, y, z] or [[x1, ..., xn], [y1, ...,  yn],  [z1, ...,  zn]]) that define the exact location of this point or this point cloud in the stated coordinate space.",
      "_embeddedTypes": [
        "https://openminds.ebrains.eu/core/QuantitativeValue",
        "https://openminds.ebrains.eu/core/QuantitativeValueArray"
      ]
    }
  }
}

CoordinatePoint will exist in parallel for cases where no (image) annotation was made.

Area schema

{
  "_type": "https://openminds.ebrains.eu/sands/Area",
  "required": [
    "shape",
    "centroidCoordinates"
  ],
  "properties": {
    "shape": {
      "type": "array",      
      "minItems": 1,
      "_instruction": "Add all annotation elements.",
      "_categories": [
        "2D-shapes"
      ]
    },
    "centroidCoordinates": {
      "type": "array",      
      "minItems": 2,
      "maxItems": 2,
      "_instruction": "Add two-dimensional coordinates (2D: [x, y]) that define the exact location of centroid of the area in the annotation associated coordinate space.",
      "_embeddedTypes": [
        "https://openminds.ebrains.eu/core/QuantitativeValue"
      ]
    }
  }
}

Missing: second point to correctly anchor any 2D shape as an Area in space. REQUIRED: Precise definition of "neutral" orientation of ALL Shapes

Volume schema

{
  "_type": "https://openminds.ebrains.eu/sands/Volume",
  "required": [
    "shape",
    "centroidCoordinates",
  ],
  "properties": {
    "shape": {
      "type": "array",      
      "minItems": 1,
      "_instruction": "Add all annotation elements.",
      "_categories": [
        "3D-shapes"
      ]
    },
    "centroidCoordinates": {
      "type": "array",      
      "minItems": 3,
      "maxItems": 3,
      "_instruction": "Add three-dimensional coordinates ([x, y, z]) that define the exact location of centroid of the volume in the annotation associated coordinate space.",
      "_embeddedTypes": [
        "https://openminds.ebrains.eu/core/QuantitativeValue"
      ]
    }
  }
}

Missing: second AND third point to correctly anchor any 3D shape as a Volume in space. REQUIRED: Precise definition of "neutral" orientation of ALL Shapes

@xgui3783 @tgbugs @skoehnen @fsdavid Please have a look at this first ideas, provide feedback and make suggestions :slightly_smiling_face:

[lzehl]

@ehennestad just to pin you on the issue I was telling you about. The last comment from me summarizes the concrete plan for updating the openMINDS model and also lists the open issues

[skoehnen]

@xgui3783 @tgbugs @skoehnen @fsdavid Please have a look at this first ideas, provide feedback and make suggestions slightly_smiling_face

Hi @lzehl,

I took a look, here are some question I had:

• Why is laterality part of the Annotation concept schema?
• What is the difference between Area and Surface/Contour in the CustomAnnotation schema
  • Area is 2D if I've seen this correctly?
• Would it make sense to separate Point into 2D and 3D point?

[lzehl]

@skoehnen please see below my responses to your questions.

Why is liberality part of the Annotation concept schema? Atlas as well as custom annotations can be only true for one organ side (left or right) or both (left and right) or it irrelevant (meaning liberality needs to be an optional property). Does this clarify your question?

What is the difference between Area and Surface/Contour in the CustomAnnotation schema 'Area' would be defined as a 2D filled area in this case as opposite to 'Volume' which would be a filled 3D object in 3D space. 'Contour' would be a the contour line in 2D in this case as opposite to 'Surface' which would be 'a 3D contour' in 3D space.

Would it make sense to separate Point into 2D and 3D point? We didn't think this necessary because the anchoring of a point in 2D and 3D is exactly the same (it does not even need anchoring because it states the coordinates in space already). While for the other spatial objects there is a definition of their shape which then can be anchored to a space. For 2D objects this could mean an anchoring to 2D or 3D space. And the essential properties/parameters to know the anchoring are different for 2D and 3D space , correct? Therefore also the separation into 2D and 3D objects (related to your previous question).

SUMMARY of the purpose of that issue:

• concept schema for atlas and custom annotations (to minimize our maintenance effort)
• for both annotations: introduce first version of a ViewerSpecification schema that adapts to the needs of the annotation type (deterministic vs probabilistic)
• allow simple shape definitions for custom annotations to support annotation tools (that was a request). BUT we can also decide this to be too complicated and of too little interest for instances stored in a graph database (do we need to query for that?)

I would really like to get this issue solved. @ehennestad and @skoehnen could you schedule a meeting with me asap? we should come to an agreement asap.

[lzehl]

ViewerSpecification SCHEMA (annotation -> opt):

• target position (req. -> coordinate point) [instructions: repeat centroid or state other]
• camera position (opt. -> coordinate point)
• preferred coloring (opt. -> SingleColor, ColorMap)
• additional remarks (opt. free text)

SingleColor SCHEMA

• value (req): str (pattern: HEX; "^#[0-9A-Fa-f]{6}$")

ColorMap SCHEMA

• name (req): str
• code source (req): str [example: "import matplotlib as mpl; cmap = mpl.get_cmap(name)"]
• programing language: ControlledTerms

[lzehl]

ANNOTATION CONCEPT

{
  "required": [
    "borderType",
    "criteriaQualityType",
    "topologyType"
  ],
  "properties": {   
    "borderType": {
      "_instruction": "Add the border type of this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/controlledTerms/AnnotationBorderType"
      ]
    },
    "centroid": {
      "type": "array",      
      "minItems": 3,
      "maxItems": 3,
      "_instruction": "Add three-dimensional coordinates ([x, y, z]) that define the geometric center of the annotation in the given coordinate space.",
      "_embeddedTypes": [
        "https://openminds.ebrains.eu/core/QuantitativeValue"
      ]
    },
    "criteria": {
      "_instruction": "Add the protocol execution defining the criteria that were applied to produce this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/core/ProtocolExecution"
      ]
    },
    "criteriaQualityType": {
      "_instruction": "Add the quality type of the stated criteria used to define this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/controlledTerms/CriteriaQualityType"
      ]
    },
    "inspiredBy": {
      "type": "array",      
      "minItems": 1,
      "uniqueItems": true,
      "_instruction": "Add one or several (source) files that inspired the definition of this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/core/File"
      ]
    },    
    "internalIdentifier": {
      "type": "string",
      "_instruction": "Enter the identifier used for this annotation within the file it is stored in."
    },
    "laterality": {
      "type": "array",
      "maxItems": 2,      
      "minItems": 1,
      "uniqueItems": true,
      "_instruction": "Add one or both sides of the body, bilateral organ or bilateral organ part that this annotation is defined in.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/controlledTerms/Laterality"
      ]
    },
    "preferredVisualization": {
      "_instruction": "Add preferred viewer specifications to visualize this annotation.",
      "_embeddedTypes": [
        "https://openminds.ebrains.eu/sands/ViewerSpecification"
      ]
    },
    "type": {
      "_instruction": "Add the topology type of this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/controlledTerms/AnnotationType"
      ]
    }
  }
}

[lzehl]

ATLAS ANNOTATION

{
  "_type": "https://openminds.ebrains.eu/sands/AtlasAnnotation",
  "_extends": "annotation.schema.tpl.json",
  "required": [
    "coordinateSpace"
  ],
  "properties": {   
    "coordinateSpace": {
      "_instruction": "Add the coordinate space for this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/sands/CommonCoordinateSpace"
      ]
    },
    "specification": {
      "_instruction": "Add the non-parametric specification of this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/core/File"
      ]
    }
  }
}

[lzehl]

CUSTOM ANNOTATION

{
  "_type": "https://openminds.ebrains.eu/sands/CustomAnnotation",
  "_extends": "annotation.schema.tpl.json",
  "required": [
    "coordinateSpace",
    "definedIn"
  ],
  "properties": {   
    "coordinateSpace": {
      "_instruction": "Add the coordinate space for this annotation.",
      "_linkedTypes": [
        "https://openminds.ebrains.eu/sands/CommonCoordinateSpace",
        "https://openminds.ebrains.eu/sands/CustomCoordinateSpace"
      ]
    },
    "specification": {
      "_instruction": "Add the non-parametric or parametric specification of this annotation.",
      "_embeddedTypes": [
        "https://openminds.ebrains.eu/core/File",
        "https://openminds.ebrains.eu/core/PropertyValueList"
      ]
    }
  }
}

[lzehl]

old ANNOTATION TYPE -> rename to ANNOTATION BORDER TYPE ANNOTATION TOPOLOGY TYPES -> rename to ANNOTATION TYPE:

• annotation point: is a single point or multiple detached points that are annotated with the same label
• annotation mask: is a filled object masking an area or volume of interest
• annotation surface: 3D boundary of an volume of interest
• annotation contour line: 2D boundary of an area of interest

[lzehl]

@UlrikeS91 @skoehnen @ehennestad You are awesome :) thank you for discussion with me and finding a first solution for this complicated issue!