Open jcohenadad opened 6 months ago
After testing image-based and segmentation-based registration, I think the segmentation-based registration is better suited and probably more robust. Example in CAN-03-RRM-092-M0
: qc.zip
Compare:
Example of registered lesion:
It is far from perfect. The main issue is likely that the lesion mask comes from a sagittal image (with 3mm thick slices) to be mapped onto an axial image (with 0.9 mm in-plane resolution). Unfortunately there is nothing we can do about it.
@leelisae I still need to add code to exclude the lesion mask from the calculation of MTR, but could you already give the lesion registration a try? You just need to 'git checkout' that branch:
git checkout jca/91-register-lesion-mask
Also make sure to upgrade to SCT 6.3 as indicated above.
@jcohenadad Yes, I upgraded to SCT 6.3 then, ran your code for an example subject - lesion registration looks good! Thank you (and also in advance for the code to calculate MTR for ROIs excluding lesions)!
Update: upsampling the PSIR to 0.7mm iso before running the segmentation and registration seems to improve the registration quality. See below (comparing: PSIR_resampled_reg, PSIR_reg, mt_t1):
Comparing the new registration method (632bdab) vs. the old one :
sliiiiightly better
Next step: test new contrast-agnostic model
69e9183 now implements code to estimate MTR excluding lesions. See example below the left and right CST (green) that is subtracted from the registered lesion mask (yellow):
And the resulting output CSV in the whole spinal cord without the lesion mask (first row) with with the lesion mask (2nd row):
Timestamp | SCT Version | Filename | Slice (I->S) | VertLevel | DistancePMJ | Label | Size [vox] | MAP() | STD() |
---|---|---|---|---|---|---|---|---|---|
2024-05-31 14:59:23 | git-nk/4481-add-monai-postprocessing-aadf57aa587f6cd921f0946514abb2c3a19e9d71 | /Users/julien/Desktop/canproco2/data/CAN-03-RRM-092-M0/mt/mtr.nii.gz | 8:16 | 2:4 | spinal cord | 717.0782670044073 | 48.01110572968546 | 5.3898473009545445 | |
2024-05-31 13:41:44 | git-nk/4481-add-monai-postprocessing-aadf57aa587f6cd921f0946514abb2c3a19e9d71 | /Users/julien/Desktop/canproco2/data/CAN-03-RRM-092-M0/mt/mtr.nii.gz | 8:16 | 2:4 | spinal cord | 735.8115947564245 | 47.50436930652403 | 5.4287692925640725 |
@lisaeylee can you give it a try?
@lisaeylee If you haven't tried it already, please make sure to update to SCT's latest branch to benefit from the latest contrast-agnostic model that works better for PSIR/STIR data (https://github.com/spinalcordtoolbox/spinalcordtoolbox/commit/bb479d82ea1e2076dd50343177056a61bd17e260)
@jcohenadad I had a few other tasks to complete earlier this week for a deadline, but this is on my agenda for today and tomorrow! To clarify, is this feature implemented in the SCT v6.3 that I recently updated to, or is it a separate update? If the latter, where can I find the exact steps to follow?
To clarify, is this feature implemented in the SCT v6.3 that I recently updated to, or is it a separate update? If the latter, where can I find the exact steps to follow?
Separate update. Step to follow (just do steps 1 and 3): https://spinalcordtoolbox.com/user_section/installation/mac.html#option-2-install-from-github-development
@jcohenadad I just realized that the update you suggested above failed to install. Here is the install_sct_log.txt. Do you know why this might be? Thank you!
@joshuacwnewton would you mind helping @lisaeylee with this? thanks!
Reading the install log, there isn't much information to work with?
'sct_check_dependencies' is what throws the error, but there are no details in the error message, plus all of the packages and dependencies seem to have installed OK in the earlier steps.
I would suggest to try using the SCT installation as-is (due to the correctly installed packages), and if something else fails in further processing, please let us know and I can help to debug further. :)
@joshuacwnewton - First and foremost, thank you for your help! I was advised to (1) upgrade to SCT v6.3 (previously was using v5.7) and (2) update SCT's latest branch to benefit from the latest contrast-agnostic model that works better for PSIR/STIR data (refer here)
I've done both but the latter (2) with the message, "installation validation failed! installation failed!".
To clarify - are you suggesting that I just proceed with the processing anyway?
@joshuacwnewton - I think the failed installation does matter. I am unable to run Julien's code - it gets stuck at the below step.
sct_deepseg -i psir_r.nii.gz -task seg_sc_contrast_agnostic -qc /Users/msresearch/Downloads/canproco/data/qc
--
Spinal Cord Toolbox (git-master-65ffa7524dc145a23086127be6d9f03ccc6b0778)
sct_deepseg -i psir_r.nii.gz -task seg_sc_contrast_agnostic -qc /Users/msresearch/Downloads/canproco/data/qc
--
Model 'model_seg_sc_contrast_agnostic_softseg_monai' is up to date (Source: https://github.com/sct-pipeline/contrast-agnostic-softseg-spinalcord/releases/download/v2.4/model_soft_bin_20240425-170840.zip)
Thank you for the update! (I was hoping that the processing would provide a more useful error message, but it seems we are getting no useful output once more.)
I will try to dig into why this might be right away. :)
@joshuacwnewton - Thank you! If it helps, here is the full report after running Julien's code.
Thank you for sharing the detailed log!
Could you please try running the following debugging commands? (The last command may not work or stall/freeze, since the problem seems to be with import monai
, but I think it's still worth trying.)
export TORCH_CPP_LOG_LEVEL=0
export TORCH_SHOW_CPP_STACKTRACES=1
export MONAI_DEBUG=True
$SCT_DIR/python/envs/venv_sct/bin/python3 -c "import monai; monai.config.deviceconfig.print_debug_info()"
@joshuacwnewton - Below is what I entered in Terminal. Should I try anything else?
(base) msresearch@MSs-MacBook-Pro ~ % export TORCH_CPP_LOG_LEVEL=0
(base) msresearch@MSs-MacBook-Pro ~ % export TORCH_SHOW_CPP_STACKTRACES=1
(base) msresearch@MSs-MacBook-Pro ~ % export MONAI_DEBUG=True
(base) msresearch@MSs-MacBook-Pro ~ % $SCT_DIR/python/envs/venv_sct/bin/python3 -c "import monai; monai.config.deviceconfig.print_debug_info()"
zsh: segmentation fault $SCT_DIR/python/envs/venv_sct/bin/python3 -c
I am very surprised to see a segmentation fault occur when simply importing monai! But, that at least tells us what error is occurring, which is helpful.
Could you please check Memory usage in Activity Monitor and share a screenshot? It's possible that there are other processes running that are using up a lot of memory.
Also, could you please try the following commands?
ulimit -s # This will return a number, e.g. 1024
ulimit -s <new_value> # enter double the number from the first command, e.g. 2048
Then re-run the commands from my previous reply? (If you still get a segmentation fault, then try doubling the number again and try again.)
Is this what you mean?
(base) msresearch@MSs-MacBook-Pro ~ % ulimit -s
8176
(base) msresearch@MSs-MacBook-Pro ~ % ulimit -s 16352
(base) msresearch@MSs-MacBook-Pro ~ % export TORCH_CPP_LOG_LEVEL=0
export TORCH_SHOW_CPP_STACKTRACES=1
export MONAI_DEBUG=True
$SCT_DIR/python/envs/venv_sct/bin/python3 -c "import monai; monai.config.deviceconfig.print_debug_info()"
zsh: segmentation fault $SCT_DIR/python/envs/venv_sct/bin/python3 -c
(base) msresearch@MSs-MacBook-Pro ~ % ulimit -s 32704
(base) msresearch@MSs-MacBook-Pro ~ % export TORCH_CPP_LOG_LEVEL=0
export TORCH_SHOW_CPP_STACKTRACES=1
export MONAI_DEBUG=True
$SCT_DIR/python/envs/venv_sct/bin/python3 -c "import monai; monai.config.deviceconfig.print_debug_info()"
zsh: segmentation fault $SCT_DIR/python/envs/venv_sct/bin/python3 -c
(base) msresearch@MSs-MacBook-Pro ~ % ulimit -s 65408
(base) msresearch@MSs-MacBook-Pro ~ % export TORCH_CPP_LOG_LEVEL=0
export TORCH_SHOW_CPP_STACKTRACES=1
export MONAI_DEBUG=True
$SCT_DIR/python/envs/venv_sct/bin/python3 -c "import monai; monai.config.deviceconfig.print_debug_info()"
zsh: segmentation fault $SCT_DIR/python/envs/venv_sct/bin/python3 -c
Hmm... I'm not seeing anything out of the ordinary! That does rule out some things, though, so thank you for trying.
I'll continue to try to brainstorm/debug on my end, and hopefully get a fix for you. :)
Just to double check: 6.3 was working for you here, right? And it's only after re-installing that you've begun encountering these issues? :thinking:
It's possible that this is Big Sur-specific? macOS 10.16/11.0 hit "end of life" last year, and so we're not able to test it on GitHub anymore, so an issue may have slipped through (e.g. the lazy loading code).
I'm setting up a virtual machine to test on Big Sur right now, though. :)
@joshuacwnewton Yes, if you're referring to my comment from 3 weeks ago, SCT v6.3 worked but at this time, there was no 'contrast-agnostic model' part in the script.
It was after this time that Julien revised the script to include, for example, sct_deepseg -i psir_r.nii.gz -task seg_sc_contrast_agnostic -qc $qc
and he suggested that I update to SCT's latest branch and advised me to go to this link and follow Option 2 - Steps 1 & 3, but the "installation failed".
And that particular command line and all else that follows after that for the PSIR (lesion segmentation) part failed likely because of the "installation failed".
I'm currently using macOS Big Sur v11.5.2.
I tried freeing up more space on this Mac, and removing spinalcordtoolbox and re-installing this again; however, installation failed again and similarly to before, sct_check_dependencies looked okay.
Thank you so much for all your help in troubleshooting this!
Aha! I was able to reproduce the issue on my Big Sur virtual machine:
This should make it much easier to debug. I'll update when I have a fix. :)
I've isolated the issue to onnx
(which monai
depends on, hence the crash when importing monai
):
If I reinstall from 6.3
, onnx==1.16.0
gets installed, and there is no crash. However, if I install from the master
branch, onnx==1.16.1
gets installed, and there is a crash. (This update to the onnx
package happened on May 23rd, which is why we're only now seeing this crash.)
I have reported this issue to the onnx
developers upstream:
For now, I will make sure we avoid this buggy version in SCT.
To quickly fix your installation, you can run this command to downgrade the version of onnx
back to 1.16.0
:
cd $SCT_DIR
./python/envs/venv_sct/bin/pip3 install onnx==1.16.0
(I've also minimized the prior debugging we did to keep this PR neat and tidy.)
@joshuacwnewton - I tried 1) running the command you provided above, 2) remove current spinalcordtoolbox, 3) re-install spinalcordtoolbox. But, I still get the same error message (see below). Did I misunderstand anything? What else can I try? I am very grateful for your generous help and time on this!
Ah, yes! There has been a misunderstanding. There is no need to re-run install_sct
a second time -- once you run the "install onnx 1.16.0" command, you are done. :)
(you can rerun sct_check_dependencies after installing onnx, and it should no longer crash, indicating that the broken installation has been fixed)
@joshuacwnewton - It works now! I successfully ran the full script and got results very similar, though not exactly the same, as Julien's above. The slight differences in the numbers might be due to the different SCT versions we're using.
I will run the script for the rest of the subjects and do QC. Hopefully, we can submit the revised manuscript soon! Thank you and your team for your very kind help!
Yay! I'm so glad things are working now. Thank you so much for your patience, and for providing lots of useful debugging information to help narrow down the issue. :relaxed:
This PR implements the following features:
TODO:
Fixes #91