BlueBrain / morphology-workflows

Workflows used for morphology processing.
https://morphology-workflows.readthedocs.io/en/stable/
Apache License 2.0
5 stars 2 forks source link

[Bug report] Applying luigi.cfg to CA1 #106

Closed KeremKurban closed 1 year ago

KeremKurban commented 1 year ago

Describe the bug

I was running the workflow for non-cortical region i.e. CA1 with layers SP, SO, SR, SLM so i set the mtype_regex as the following:

[MType] 
mtype_regex = ["S*"]

I assumed * would take care of all the rest but perhaps i had to be more specific if the code expects it such as this one

mtype_regex = ["L[1-6]_[A-Z]+$", "L[1-6]_[A-Z]+:[A-C]$", "L23_[A-Z]+$", "L23_[A-Z]+:[A-C]$", "L[1-6]_[A-Z]+-[A-Z]+$"]

I also added manually layer and mtype columns to dataset.csv after initialization. (Note in the documentation layer was not mentioned but it seems to be a mandatory field since i got an error asking for that column)

After pipeline was run, what i see in neurondb.dat is something like

                <morphology>
                        <name>mpg150304_A_idA</name>
                        <mtype>S</mtype>
                        <msubtype></msubtype>
                        <layer>SP</layer>
                        <repair>
                                <use_axon>true</use_axon>
                                <use_dendrites>true</use_dendrites>
                                <axon_repair>true</axon_repair>
                                <dendrite_repair>true</dendrite_repair>
                                <basal_dendrite_repair>true</basal_dendrite_repair>
                                <tuft_dendrite_repair>true</tuft_dendrite_repair>
                                <oblique_dendrite_repair>true</oblique_dendrite_repair>
                                <unravel>true</unravel>
                                <use_for_stats>true</use_for_stats>
                                <axon_sources>
                                </axon_sources>
                        </repair>
                </morphology>

where you can see mtype is S which is similar to S* field i put in the luigi.cfg

Is there a specific way of formatting we need to put the regex ? I see for every use case like L2-X , L23-Y L5:TTPC:A you put a different regex. in our case the mtypes are

SP_PC, SP_PVBC, SOOLM .. (i.e. {layer}{celltype} ) but it doesnt mean in the future for other regions/mtypes it will be like this as the nomenclature in literature will be set in such way.

Optional link from https://morphology-workflows.readthedocs.io which documents the behavior that is expected

No response

To Reproduce

# code is the same , im putting the luigi.cfg

[core]
logging_conf_file = logging.conf
autoload_range = false

# Curation tasks

[Curate]
tag_output = False
dataset_df = dataset.csv
result_path = out_curated
report_path = report_curation.pdf
nb_processes = 4

[ExtractMarkers]
skip = False

[PlotMarkers]
with_plotly = True
skip = False

[CheckNeurites]
# for cells with no axon, add 1 dummy section
mock_soma_type = spherical
ensure_stub_axon = True

[Recenter]
skip = False

[DetectErrors]
skip = False
min_range = 50

[PlotErrors]
with_plotly = True
skip = False

[Orient]
pia_direction = y

[Align]
skip = True

[EnsureNeuritesOutsideSoma]
skip = True

[PlotMorphologies]
skip = False
with_realistic_diameters = False

# Annotation tasks

[Annotate]
tag_output = False
dataset_df = out_curated/curated_dataset.csv
result_path = out_annotated
report_path = report_annotation.pdf
nb_processes = 4

[MType]
mtype_regex = ["S*"]

[HardLimit]
skip_axon = False
dendrite_hard_limit = L1_hard_limit
axon_hard_limit = L1_axon_hard_limit

[PlotHardLimit]
with_plotly = True
skip = False

[ApicalPoint]
tuft_percent = 20
skip = False

[PlotApicalPoint]
with_plotly = True
skip = False

[CutLeaves]
bin_width = 20
percentile_threshold = 60

[PlotCutLeaves]
with_plotly = True
skip = False

# Repair tasks

[Repair]
tag_output = False
dataset_df = out_annotated/annotated_dataset.csv
result_path = out_repaired
report_path = report_repair.pdf
make_release = True
nb_processes = 4

[Unravel]
window_half_length = 5

[RepairNeurites]
with_plot = False

[PlotRepair]
with_plotly = False
skip = False

[MakeRelease]
repair_path = repair_release
unravel_path = unravel_release
zero_diameter_path = zero_diameter_release

Error

No error in this tool, but later on synthesis-workflow does not work.

Additional context

sample dataset.csv first column looks like

morph_name,morph_path,mtype,layer
mpg141124_A_idA,/gpfs/bbp.cscs.ch/project/proj142/home/kurban/MMBHIP-38/mmb-hippocampus-morphologies/regions/ca1/lnmc_mouse_hipp/mpg141124_A_idA.swc,SP_PC,SP

morphology_workflows Version in Use

0.7.0

Python Version

3.10.8

Operating system

Linux

arnaudon commented 1 year ago

Hello, just use ' = []' for the regex thing, so it will not check anything. I tried locally, and S* creates your bug for some weird regex reason. This regex I added to be able to check that mtype names are of specific format, but it is not very useful, I should make it optional