Error in placement hints: Regex and barrel annotations

[alTeska]

I am not sure about the rest of the pipeline but the regex inside atlas-placement-hints does not work well with barrel annotations: https://github.com/BlueBrain/atlas-placement-hints/blob/d4a57b7d390c14d5f65f3f17ccfab6963f39add8/atlas_placement_hints/layered_atlas.py#L81C[…]81C81

This kind of regex will struggle with the annotations pattern:

SSp-bfd:
    SSp-bfd-C1
        SSp-bfd-C1-1
        SSp-bfd-C1-2
        SSp-bfd-C1-3
...

I attach the comparison of the two [PH]y.nrrd files, with and without barrel annotations. The first one was downloaded from production, the second one was an outcome of a pipeline run with the modified annotations.

Alexis used the following code that made us spot the issue:

from neurocollage import get_layer_annotation
from voxcell.nexus.voxelbrain import Atlas
from region_grower.atlas_helper import AtlasHelper

atlas_path = "/gpfs/bbp.cscs.ch/project/proj100/atlas/mouse/atlas-mouse-barrels"
circuit_path = "/gpfs/bbp.cscs.ch/data/scratch/proj100/mouse-circuits/ssp-bfd/alexis/auxiliary/circuit.somata.h5"
atlas = AtlasHelper(Atlas.open(atlas_path), region_structure_path="region_structure.yaml")

# get layer anotation into an nrrd file to be loaded by itk-snap
layers = get_layer_annotation(
    {"atlas": atlas_path, "structure": "region_structure.yaml"}, "Isocortex"
)
layers["annotation"].save_nrrd("layers.nrrd")

withregion_structure.yaml:

  layers:
  - 1
  - 2
  - 3
  - 4
  - 5
  - 6
  names:
    1: layer 1
    2: layer 2
    3: layer 3
    4: layer 4
    5: layer 5
    6: layer 6
  region_queries:
    1: '@.*1$'
    2: '@.*2[a|b]?$'
    3: '@.*3$'
    4: '@.*4$'
    5: '@.*5$'
    6: '@.*6[a|b]?$'
  thicknesses:
    1: 69
    2: 135
    3: 100
    4: 208
    5: 248
    6: 451

[alTeska]

path for the two[PH]y.nrrdfiles :

1. with artifacts /gpfs/bbp.cscs.ch/project/proj100/atlas/mouse/atlas-release-barrels-may2023/placement_hints_ccfv3_l23split_barrel/[PH]y.nrrd

2. without /gpfs/bbp.cscs.ch/project/proj148/scratch/test_circuit/.atlas/[PH]y.nrrd

[arnaudon]

Here are more plots: PHy with proper values, we see the barrel, but we should not: PHlayer_1 (bottom value, aka [:, :, :, 0]), we also see the barrels: and if I zoom out the same PHlayer_1, I'm not convinced it is so 'good', we see a red spot top right, and not a uniform increase in top left pprojection

I hope this helps improving the qualities of PH files!

[arnaudon]

Also the cell_densities files are wrong:

[arnaudon]

Also orientations fields:

[mgeplf]

I am not sure about the rest of the pipeline but the regex inside atlas-placement-hints does not work > This kind of regex will struggle with the annotations pattern:

SSp-bfd:
    SSp-bfd-C1
        SSp-bfd-C1-1
        SSp-bfd-C1-2
        SSp-bfd-C1-3
...

Which part is of the regex is suspicious? AFAIK, the part highlighted won't matter ([ab]?$) as ab are optional. I'm more concerned by the 1 at the end of the region name: SSp-bfd-C1; since this "looks" like it belongs to layer one.

I think it would be useful to work with @lecriste to isolate the step within the pipeline where the results start to differ, so it can be narrowed down to this regex, or potentially something else.

The use of regular expressions in these atlases has always been fraught (ie: [and] Now You Have Two Problems), but I'm not sure when/if we'll be able to tackle that, so, in the meantime, tracking down to make sure that this is the problem would be helpful.

[arnaudon]

@lecriste here is a PHy of another atlas I was working on this morning (which we use for striatum). I'm not sure which one is 'correct' isocortex, I quickly loose track of the latest atlas folder on gpfs. in any case, these PH are distances to smooth boundaries (or meshes), hence they should be smooth inside the atlas as well.

@mgeplf probably reruning the pipeline with a regex that does not confuses the entire barrel columns with layers would work. I'm not good enough with regex to know how to update it, but aleksandra had a try before she left and said it should be possible.

[lecriste]

The only input (other than hierarchy and annotation volume) to calculate placement hints is direction-vectors. If the direction-vectors volume looks fine, than the different regex can be tested with the command described in the ReadMe: atlas-placement-hints isocortex --hierarchy-path <hierarchy-path> --annotation-path <annotation-path> --direction-vectors-path <direction_vectors_path> --algorithm voxel-based where <hierarchy-path> and <annotation-path> are the hierarchy and annotation volume linked to the current atlas release in Nexus Staging.

[alTeska]

hi, yes @mgeplf the problem is, as you worried, the numbered names and numbered layers. The 1 at the end of the barrel is assumed to be layer 1, etc. and then the whole barrel becomes part of a layer.

I think the issue is propagated in the pipeline that's in staging, as the files from there seem to all have corrupted inputs as shown by Alexis.

@lecriste can we re-run the things with new regex and I can examine if the files are all in check?

[lecriste]

Which new regex? I would need a PR in this repo to fetch and re-run the pipeline.

[mgeplf]

@lecriste > Which new regex?

Maybe it's this one: https://github.com/BlueBrain/atlas-placement-hints/pull/14

@alTeska what needs to be done to get this PR moved forward?

[alTeska]

I test-run, the two sections of the atlas pipeline, which are directly changed by the PR.

The missing part is a full re-run of the pipeline and inspection of the complete atlas files.

[mgeplf]

The missing part is a full re-run of the pipeline and inspection of the complete atlas files.

Can you help with that @lecriste? I'd like to get this PR closed, if possible.

[lecriste]

Yes but not before February.

[lecriste]

Fixed by https://github.com/BlueBrain/atlas-placement-hints/pull/14