KeyError: 'Filename' in DWI processing

[jcohenadad]

Version: e3cc60adec6aff45e8f9b716aaa58fd8860effbd

Traceback (most recent call last):
  File "/home/GRAMES.POLYMTL.CA/p101317/code/spine-park/venv/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3653, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Filename'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/GRAMES.POLYMTL.CA/p101317/code/spine-park/aggregate_chunks.py", line 66, in <module>
    main()
  File "/home/GRAMES.POLYMTL.CA/p101317/code/spine-park/aggregate_chunks.py", line 62, in main
    aggregate_data(input_csv, output_csv)
  File "/home/GRAMES.POLYMTL.CA/p101317/code/spine-park/aggregate_chunks.py", line 19, in aggregate_data
    df['Subject'] = df['Filename'].str.extract(r'(sub-[^_/]+)')
  File "/home/GRAMES.POLYMTL.CA/p101317/code/spine-park/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 3761, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/GRAMES.POLYMTL.CA/p101317/code/spine-park/venv/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3655, in get_loc
    raise KeyError(key) from err
KeyError: 'Filename'

[jcohenadad]

Weirdly, the output CSV, only for the DWI files, did not include a header:

[jcohenadad]

When re-running the command, fresh, the header is here:

Re-running, on another subject, the header is kept intact:

So, at some point, for some reasons, the header is removed 🤔

[jcohenadad]

For LD-214 sct_extract_metric did not complete:

Spinal Cord Toolbox (git-jca/4527-register-template-step0-517cc66596db50a310ddc0bdf8985fbbe54405a4*)

sct_extract_metric -i sub-LD214_chunk-1_DWI_moco_FA.nii.gz -f label_sub-LD214_chunk-1_DWI_moco/atlas -l 32,33 -combine 1 -vert 2:12 -vertfile label_sub-LD214_chunk-1_DWI_moco/template/PAM50_levels.nii.gz -perlevel 1 -method map -o /home/GRAMES.POLYMTL.CA/p101317/lydia/results_20240702_140359/results/DWI_FA_32-33.csv -append 1
--

Load metric image...
Estimation for label: 32,33
Traceback (most recent call last):
  File "/home/GRAMES.POLYMTL.CA/p101317/code/spinalcordtoolbox/spinalcordtoolbox/scripts/sct_extract_metric.py", line 400, in <module>
    main(sys.argv[1:])
  File "/home/GRAMES.POLYMTL.CA/p101317/code/spinalcordtoolbox/spinalcordtoolbox/scripts/sct_extract_metric.py", line 393, in main
    save_as_csv(agg_metric, fname_output, fname_in=fname_data, append=append_csv)
  File "/home/GRAMES.POLYMTL.CA/p101317/code/spinalcordtoolbox/spinalcordtoolbox/aggregate_slicewise.py", line 570, in save_as_csv
    agg_metric_key = [v for i, (k, v) in enumerate(agg_metric.items())][0]
IndexError: list index out of range

Is this the cause of the issue?

[jcohenadad]

Even more weirdly, when re-running the exact same command on the same data, it completes:

(base) p101317@joplin:~/lydia/results_20240702_140359/data_processed/sub-LD214/dwi$ sct_extract_metric -i sub-LD214_chunk-1_DWI_moco_FA.nii.gz -f label_sub-LD214_chunk-1_DWI_moco/atlas -l 32,33 -combine 1 -vert 2:12 -vertfile label_sub-LD214_chunk-1_DWI_moco/template/PAM50_levels.nii.gz -perlevel 1 -method map -o /home/GRAMES.POLYMTL.CA/p101317/lydia/results_20240702_140359/results/DWI_FA_32-33.csv -append 1

--
Spinal Cord Toolbox (git-jca/4527-register-template-step0-517cc66596db50a310ddc0bdf8985fbbe54405a4*)

sct_extract_metric -i sub-LD214_chunk-1_DWI_moco_FA.nii.gz -f label_sub-LD214_chunk-1_DWI_moco/atlas -l 32,33 -combine 1 -vert 2:12 -vertfile label_sub-LD214_chunk-1_DWI_moco/template/PAM50_levels.nii.gz -perlevel 1 -method map -o /home/GRAMES.POLYMTL.CA/p101317/lydia/results_20240702_140359/results/DWI_FA_32-33.csv -append 1
--

Load metric image...
Estimation for label: 32,33

Done! To view results, type:
xdg-open /home/GRAMES.POLYMTL.CA/p101317/lydia/results_20240702_140359/results/DWI_FA_32-33.csv

[jcohenadad]

Note that vertebral labeling is completely off for this subject, so hopefully the issue is related to that:

I'll re-run the whole pipeline, only on subjects with correct vertebral labeling.

[jcohenadad]

Re-running without this subject, the pipeline completed without error. We can close then.