Introduce contrast-agnostic segmentation in the pipeline

[jcohenadad]

Since it contrast-agnostic segmentation is more performant than sct_deepseg_sc, it would be a good idea to use it instead.

[jcohenadad]

Example of comparison sct_deepseg_sc vs. sct_deepseg in 9 subjects (I suggest you order by 'Subject' so you can compare the two algorithms for each subject): qc.zip. Notice sub-LM166 where sct_deepseg clearly performs better.

@naga-karthik @sandrinebedard you might also find this comparison interesting.

[!NOTE]
To be able to use this model, SCT's latest version needs to be installed (v6.3, or from master).

[!WARNING]
sct_deepseg is more "RAM hungry", so if RAM is a limitation, I suggest processing less subjects in parallel (with -j flag)

[naga-karthik]

Thank you for uploading the QC! Indeed for sub-LM166 contrast-agnostic did a good job at the compression! :)

[Kaonashi22]

Indeed, the segmentations are more accurate with _sctdeepseg. However, I do have a problem of RAM; I was able able to run only two subjects in parallel (-jobs 2). Do you think it will be fine with _sctdeepseg.?

[Kaonashi22]

I have this error immediately when I run the new script: _sct_deepseg: error: unrecognized arguments: -qc /dagher/dagher11/lydia11/RESULTS010524/qc -qc-subject sub-BJ170

[jcohenadad]

I have this error immediately when I run the new script: sct_deepseg: error: unrecognized arguments: -qc /dagher/dagher11/lydia11/RESULTS_010524/qc -qc-subject sub-BJ170

you likely don't have the latest SCT version-- can you pls copy/paste the entire terminal output (not just the error message)-- thx

[jcohenadad]

I was able able to run only two subjects in parallel (-jobs 2). Do you think it will be fine with sct_deepseg.?

wow, only 2? how much RAM do you have? can you please copy/paste the output of sct_check_dependencies that you run on your server?

[Kaonashi22]

wow, only 2? how much RAM do you have? can you please copy/paste the output of sct_check_dependencies that you run on your server?

Yes, only 2 subjects in parallel. I keep having errors when I run more subjects (it took me ~22h to run 60 subjects, with 6 that failed for memory issues...). I have 12 CPUs The output of the command:

Spinal Cord Toolbox (git-master-365651dbd7d31b909b7ec85844b0e6492a4649ca)

sct_check_dependencies 
--
_SYSTEM INFORMATION
------------------
SCT info:
- version: git-master-365651dbd7d31b909b7ec85844b0e6492a4649ca
- path: /export02/data/lydiac/spinalcordtoolbox
OS: linux (Linux-5.15.115-i686-64-smp-x86_64-with-glibc2.31)
CPU cores: Available: 12, Used by ITK functions: 12
RAM: Total: 31991MB, Used: 4219MB, Available: 27024MB

OPTIONAL DEPENDENCIES
---------------------
Check FSLeyes version...............................[OK] (1.10.4)

MANDATORY DEPENDENCIES
----------------------
Check Python executable.............................[OK]
  Using bundled python 3.9.18 (main, Sep 11 2023, 13:41:44) 
[GCC 11.2.0] at /export02/data/lydiac/spinalcordtoolbox/python/envs/venv_sct/bin/python
Check if data are installed.........................[OK]
Check if dipy is installed..........................[OK] (1.5.0)
Check if ivadomed is installed......................[OK] (2.9.8)
Check if matplotlib is installed....................[OK] (3.8.2)
Check if nibabel is installed.......................[OK] (3.2.2)
Check if nilearn is installed.......................[OK] (0.10.2)
Check if numpy is installed.........................[OK] (1.23.5)
Check if onnxruntime is installed...................[OK] (1.16.3)
Check if pandas is installed........................[OK] (1.4.4)
Check if portalocker is installed...................[OK] (2.8.2)
Check if psutil is installed........................[OK] (5.9.6)
Check if pyqt5 (5.12.3) is installed................[OK] (5.12.3)
Check if pytest is installed........................[OK] (7.4.3)
Check if pytest-cov is installed....................[OK] (4.1.0)
Check if requests is installed......................[OK] (2.31.0)
Check if requirements-parser is installed...........[OK]
Check if scipy is installed.........................[OK] (1.11.4)
Check if scikit-image is installed..................[OK] (0.22.0)
Check if scikit-learn is installed..................[OK] (1.3.2)
Check if xlwt is installed..........................[OK] (1.3.0)
Check if tqdm is installed..........................[OK] (4.66.1)
Check if transforms3d is installed..................[OK] (0.4.1)
Check if urllib3 is installed.......................[OK] (2.1.0)
Check if pytest_console_scripts is installed........[OK]
Check if pyyaml is installed........................[OK] (6.0.1)
Check if voxelmorph is installed....................[OK] (0.2)
Check if wquantiles is installed....................[OK] (0.4)
Check if xlsxwriter is installed....................[OK] (3.1.9)
Check if spinalcordtoolbox is installed.............[OK]
Check ANTs compatibility with OS ...................[OK]
Check PropSeg compatibility with OS ................[OK]
Check if figure can be opened with PyQt.............[OK]
Check if figure can be opened with matplotlib.......[OK] (Using GUI backend: 'QtAgg')_

[Kaonashi22]

I have this error immediately when I run the new script: sct_deepseg: error: unrecognized arguments: -qc /dagher/dagher11/lydia11/RESULTS_010524/qc -qc-subject sub-BJ170

you likely don't have the latest SCT version-- can you pls copy/paste the entire terminal output (not just the error message)-- thx

err.batch_processing_sub-BB277.log

[jcohenadad]

~It says you have "Available: 27024MB", which is more than enough for doing 12 subjects in parallel (you have 12 cores). Somethings not right. Can you pls ask your McGill IT person to run some profiling while you run sct_run_batch and report in this thread the result of the profiling?~

Moved thread here: https://github.com/sct-pipeline/spine-park/issues/30

[jcohenadad]

from your log, it shows you are running SCT version from 6 months ago:

pls go to your SCT install folder and run:

git pull
./install_sct

~and then~

sct_deepseg -install seg_sc_contrast_agnostic

EDIT 20240501_114341 Update from https://github.com/sct-pipeline/spine-park/issues/28#issuecomment-2088628841: command should be:

sct_deepseg -install-task seg_sc_contrast_agnostic

[Kaonashi22]

OK, I'm gonna reinstall it

---

From: Julien Cohen-Adad @.> Sent: May 1, 2024 08:34 To: sct-pipeline/spine-park @.> Cc: Lydia Chougar, Dr @.>; Comment @.> Subject: Re: [sct-pipeline/spine-park] Introduce contrast-agnostic segmentation in the pipeline (Issue #28)

from your log, it shows you are running SCT version from 6 months ago: image.png (view on web)https://github.com/sct-pipeline/spine-park/assets/2482071/b7ce1e8c-ba28-43c3-8a33-05535d7dbb9a

pls go to your SCT install folder and run:

git pull ./install_sct

and then

sct_deepseg -install seg_sc_contrast_agnostic

— Reply to this email directly, view it on GitHubhttps://github.com/sct-pipeline/spine-park/issues/28#issuecomment-2088403512, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BFCFJYX7YWZWGSDXS4BQ4LDZADOPBAVCNFSM6AAAAABG5C3GMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBYGQYDGNJRGI. You are receiving this because you commented.Message ID: @.***>

[Kaonashi22]

when I do git pull, it returns "Already up to date". And when I run sct_deepseg -install seg_sc_contrast_agnostic, it returns: _sctdeepseg: error: ambiguous option: -install could match -i, -install-task

[jcohenadad]

when I do git pull, it returns "Already up to date".

@joshuacwnewton can you advise? Should we just recommend to delete the folder and do a fresh git clone?

[Kaonashi22]

I also tried to reinstall the sct with git clone https://github.com/spinalcordtoolbox/spinalcordtoolbox, them git pull ./install_sct, but I still have the same error

[jcohenadad]

I also tried to reinstall the sct with git clone https://github.com/spinalcordtoolbox/spinalcordtoolbox, them git pull ./install_sct, but I still have the same error

if you reinstalled it at another location, it is possible that you are still calling the 'old' SCT commands. At this point, with many installations on your system, it is difficult for us to understand and debug. unfortunately i do not have the time to debug this-- hopefully @joshuacwnewton or @mguaypaq you can spare some time to help Lydia-- thanks

[sandrinebedard]

I noticed that the command to install the contrast agnostic model is

sct_deepseg -install-task seg_sc_contrast_agnostic

rather than just -install as mentionned here

[jcohenadad]

good point @sandrinebedard, i thought https://github.com/spinalcordtoolbox/spinalcordtoolbox/issues/4435 was already fixed 😅

[Kaonashi22]

Thanks, it's working with this new command!

[joshuacwnewton]

good point @sandrinebedard, i thought spinalcordtoolbox/spinalcordtoolbox#4435 was already fixed 😅

Oops, yes! Very sorry about not getting to this prior to the v6.3 release. I'll add it to the v6.4 milestone so it isn't missed.