NeuroML / pyNeuroML

A single package in Python unifying scripts and modules for reading, writing, simulating and analysing NeuroML2/LEMS models.
https://docs.neuroml.org/Userdocs/Software/pyNeuroML.html
GNU Lesser General Public License v3.0
34 stars 30 forks source link

[Bug] ExportSWC: exceptions are wrong #330

Open sanjayankur31 opened 2 months ago

sanjayankur31 commented 2 months ago

Using the ExportSWC class to export NeuroML2 to SWC works, but it has some checks that aren't necessarily correct for all cell models.

For example, on valid cells like the HL23PYR cell, it throws this exception:

$ pynml -swc HL23PYR.cell.nml 
pyNeuroML >>> INFO - Trying to convert HL23PYR.cell.nml to swc format...
pyNeuroML >>> INFO - Loading NeuroML2 file: HL23PYR.cell.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/CaDynamics_E2_NML2.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/pas.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/Ih.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/SK.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/K_T.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/K_P.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/Kv3_1.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/Im.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/NaTg/NaTg.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/Ca_HVA.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/Ca_LVA.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/L23-example/channels/Nap.channel.nml
Traceback (most recent call last):
  File "/home/asinha/.local/share/virtualenvs/neuroml-311-dev/bin/pynml", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/pyneuroml/pynml.py", line 971, in main
    _evaluate_arguments(args)
  File "/home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/pyneuroml/pynml.py", line 866, in _evaluate_arguments
    convert_count += 1 if convert_to_swc(f) else 0
                          ^^^^^^^^^^^^^^^^^
  File "/home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/pyneuroml/swc/ExportSWC.py", line 174, in convert_to_swc
    raise Exception(
Exception: The numbers of the segments in groups: soma_group+dendrite_group+axon_group (5818), is not the same as total number of segments (5840)! All bets are off!

I need to check why these numbers don't add up---where are the 22 segments? But I also see it with other cells, like this one from the Allen cells: https://github.com/OpenSourceBrain/AllenInstituteNeuroML/blob/master/CellTypesDatabase/models/NeuroML2/Cell_497232312.cell.nml

$ pynml -swc Cell_497232312.cell.nml 
pyNeuroML >>> INFO - Trying to convert Cell_497232312.cell.nml to swc format...
pyNeuroML >>> INFO - Loading NeuroML2 file: Cell_497232312.cell.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/pas.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/NaV.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/K_T.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Kd.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Kv2like.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Kv3_1.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/SK.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Ih.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Im_v2.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Ca_HVA.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/Ca_LVA.channel.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/CaDynamics_all.nml
pyNeuroML >>> INFO - Loading NeuroML2 file: /home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/tests/plot/CaDynamics.nml
Traceback (most recent call last):
  File "/home/asinha/.local/share/virtualenvs/neuroml-311-dev/bin/pynml", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/pyneuroml/pynml.py", line 971, in main
    _evaluate_arguments(args)
  File "/home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/pyneuroml/pynml.py", line 866, in _evaluate_arguments
    convert_count += 1 if convert_to_swc(f) else 0
                          ^^^^^^^^^^^^^^^^^
  File "/home/asinha/Documents/02_Code/00_mine/NeuroML/software/pyNeuroML/pyneuroml/swc/ExportSWC.py", line 174, in convert_to_swc
    raise Exception(
Exception: The numbers of the segments in groups: soma_group+dendrite_group+axon_group (1161), is not the same as total number of segments (3489)! All bets are off!
sanjayankur31 commented 2 months ago

In the allen inst cell, the apical dendrites aren't included in the "dendrite_group":

a = cell.get_ordered_segments_in_groups(["all", "soma_group", "dendrite_group", "axon_group", "dend", "apic"])

Checking:

In [39]: len(a["apic"]) + len(a["dendrite_group"]) + len(a["soma_group"]) + len(a["axon_group"])
Out[39]: 3489

In [40]: len(a["all"])
Out[40]: 3489

In [41]: len(a["dendrite_group"]) + len(a["soma_group"]) + len(a["axon_group"])
Out[41]: 1161

@pgleeson : I guess the apical dendrites segment group apic should also be included in the dendrite_group here?

In the HL23PYR cell, there are 22 segments in the myelin_group that aren't in the conventional groups:

a = cell.get_ordered_segments_in_groups(["all", "soma_group", "dendrite_group", "axon_group", "basal_dendrite_group", "apical_dendrite_group", "myelin_group"])

here's verification:

(a["myelin_group"])
Out[12]: 
[<Segment|5819|Seg0_myelin_0>,
 <Segment|5820|Seg1_myelin_0>,
 <Segment|5821|Seg2_myelin_0>,
 <Segment|5822|Seg3_myelin_0>,
 <Segment|5823|Seg4_myelin_0>,
 <Segment|5824|Seg5_myelin_0>,
 <Segment|5825|Seg6_myelin_0>,
 <Segment|5826|Seg7_myelin_0>,
 <Segment|5827|Seg8_myelin_0>,
 <Segment|5828|Seg9_myelin_0>,
 <Segment|5829|Seg10_myelin_0>,
 <Segment|5830|Seg11_myelin_0>,
 <Segment|5831|Seg12_myelin_0>,
 <Segment|5832|Seg13_myelin_0>,
 <Segment|5833|Seg14_myelin_0>,
 <Segment|5834|Seg15_myelin_0>,
 <Segment|5835|Seg16_myelin_0>,
 <Segment|5836|Seg17_myelin_0>,
 <Segment|5837|Seg18_myelin_0>,
 <Segment|5838|Seg19_myelin_0>,
 <Segment|5839|Seg20_myelin_0>,
 <Segment|5840|Seg21_myelin_0>]

In [7]: len(a["dendrite_group"]) + len(a["soma_group"]) + len(a["axon_group"])
Out[7]: 5818

In [8]: len(a["myelin_group"])
Out[8]: 22

In [9]: len(a["all"])
Out[9]: 5840

I don't think the myelin segments are meant to be in any of the standard groups, so this seems correct?

sanjayankur31 commented 2 months ago

This is not actually an exception bug. The cells we tested here are valid but not completely. We just don't have a test to check that the sum of dendrite_group + axon_group + soma_group = all. For correctness, this should hold, so we'll add another validation test for this: if one is using the default segment groups, this check should hold.