search

CBICA / CaPTk

Cancer Imaging Phenomics Toolkit (CaPTk) is a software platform to perform image analysis and predictive modeling tasks. Documentation: https://cbica.github.io/CaPTk
https://www.cbica.upenn.edu/captk
Other
175 stars 63 forks source link

SurvivalPredictor doesn't run on cluster #1234

Open chiharusako opened 4 years ago

chiharusako commented 4 years ago

Describe the bug SurvivalPredictor doesn't run on the cluster

To Reproduce Testing:

$ dir=/cbica/home/sakoc/comp_space/lab/captk/180b/survival
$ /cbica/software/lab/captk/centos7/1.8.0.Beta/bin/SurvivalPredictor \
        -t 1 \
        -i $dir/input_2subjects \
        -m /cbica/home/sakoc/CaPTk/PretrainedModels/SurvivalPredictor_PretrainedModel_1.8.0 \
        -o $dir/output_test"
Input directory name:/cbica/home/sakoc/comp_space/lab/captk/180b/survival/input_2subjects
Output directory name:/cbica/home/sakoc/comp_space/lab/captk/180b/survival/output_test
Model directory name:/cbica/home/sakoc/CaPTk/PretrainedModels/SurvivalPredictor_PretrainedModel_1.8.0
Module loaded: Survival Prediction on Existing Model:
Number of subjects with required input: 2
Segmentation fault

Training (same error submitting interactively):

$ dir=/cbica/home/sakoc/comp_space/lab/captk/180b/survival
$ qsub -j y -b y -l h_vmem=128G -o $dir/captk_train_\$JOB_ID.log \
        /cbica/software/lab/captk/centos7/1.8.0.Beta/bin/SurvivalPredictor \
        -t 0 \
        -i $dir/input \
        -o $dir/output_train

$ head $dir/captk_train_8280295.log
Input directory name:/cbica/home/sakoc/comp_space/lab/captk/180b/survival/input
Output directory name:/cbica/home/sakoc/comp_space/lab/captk/180b/survival/output_train
Module loaded: Prepare Survival Prediction Model.
Number of subjects with required input: 102
Patient's data loaded:1
*** Error in `/cbica/software/lab/captk/centos7/1.8.0.Beta/bin/SurvivalPredictor': free(): invalid pointer: 0x00007f4c58295010 ***
======= Backtrace: =========
/usr/lib64/libc.so.6(+0x81299)[0x7f4c8e0ba299]
/cbica/software/lab/captk/centos7/1.8.0.Beta/bin/SurvivalPredictor[0x4cc028]
/cbica/software/lab/captk/centos7/1.8.0.Beta/bin/SurvivalPredictor[0x4cea97]

Expected behavior Testing outputs scores and Training outputs a model

CaPTk Version 1.8.0.Beta on cbica-cluster

OS centos7

saimarathore commented 4 years ago

@chiharusako seems like in the 2 testing cases, segmentation has been warped to atlas space, but survivalpredictor module expects atlas to be warped to the patients space. Can you try warping atlas to patient space and then run again?

chiharusako commented 4 years ago

I was not aware this changed. In previous versions, it required the segmentation labels 1,2,4 in atlas space instead and sample data was in that format as well. I do confirm that the new sample data includes an atlas warped to patient data as you noted, but this needs to be updated in the documentation as well.

Can you please point me to which atlas should be used? Where is it downloadable for users?

Also, for the segmentation file - please clarify whether you still need the ventricle label (7) or not, and whether all other labels other than 1,2,4 (or 1,2,4,7) will be ignored or not. In the sample data, the segmentation file has label 7 but it's not only the ventricle but also includes CSF, and there are many more labels than 1,2,4.

saimarathore commented 4 years ago

updated the documentation accordingly. atlas will come with captk's installation in the following folder: ..\CaPTk install directory\data\survival\jakob_labels_ChrisClark_to_glistr.nii.gz

the segmentation file should have the following 4 labels: 1=non-enhancing 2=edema 4=enhancing 7=ventricles

chiharusako commented 3 years ago

@saimarathore I tried testing with the atlas label warped to patient space, as you described.

  1. It still ends with a Segmentation fault
  2. The atlas label file you provided is a label file with values 0-255. How are you recommending users to use this? In order to warp the atlas label, one will also need an atlas brain image (like t1) that goes with it so that you can register to the patient image to get the transformation.
  3. Also the atlas label has names in the filename. Would it make sense to remove the name?

Logs below, tested on cubic-login1

$ module load captk/1.8.0.Beta
$ dir=/cbica/home/sakoc/comp_space/lab/captk/180b/survival2
$ SurvivalPredictor -t 1 \
      -i $dir/input_2subjects \
      -o $dir/output_2subjects \
      -m /cbica/home/sakoc/CaPTk/PretrainedModels/SurvivalPredictor_PretrainedModel_1.8.0/
Input directory name:/cbica/home/sakoc/comp_space/lab/captk/180b/survival2/input_2subjects
Output directory name:/cbica/home/sakoc/comp_space/lab/captk/180b/survival2/output_2subjects
Model directory name:/cbica/home/sakoc/CaPTk/PretrainedModels/SurvivalPredictor_PretrainedModel_1.8.0/
Module loaded: Survival Prediction on Existing Model:
Number of subjects with required input: 2
Segmentation fault
saimarathore commented 3 years ago

@chiharusako can you try on local machine? perhaps the issue is on cluster (we will later figure out why if that is the case). i just ran on my local machine and it worked fine for me.

Input directory name:\cbica-cifs\cbica\home\sakoc\comp_space\lab\captk\180b\sur vival2\input_2subjects Output directory name:E:/SoftwareDevelopmentProjects/testing_1.8.2Alpha/survival /output_2subjects Model directory name:\cbica-cifs\cbica\home\sakoc\CaPTk\PretrainedModels\Surviv alPredictor_PretrainedModel_1.8.0 Module loaded: Survival Prediction on Existing Model: Number of subjects with required input: 2 done Subject No:0 Subject No:1

SPI indices: AAAA_2007.01.21: -1.02645 AAAB_2006.10.28: -5.27295 Finished successfully Press any key to continue . . .

chiharusako commented 3 years ago

@saimarathore Is your version the same as 1.8.0.Beta? Which version should I use?

chiharusako commented 3 years ago

I just tried on Mac version CaPTk_1.8.0.nonRelease.20200826.249b89b6 and it gave a Segmentation fault: 11, using the same input

AlexanderGetka-cbica commented 3 years ago

Hi @chiharusako, could you please try this on the latest mac artifact?

Thanks for the continued testing, it is of tremendous help!

chiharusako commented 3 years ago

@AlexanderGetka-cbica Same error

$ /Applications/CaPTk_1.8.0.nonRelease.20200915.108d5d65.app/Contents/Resources/bin/SurvivalPredictor \
   -t 1 \
   -i input_2subjects \
   -o output_2subjects_mac_1.8.0.nonRelease.20200915.108d5d65 \
  -m /Volumes/cbica/home/sakoc/CaPTk/PretrainedModels/SurvivalPredictor_PretrainedModel_1.8.0/
Input directory name:input_2subjects/
Output directory name:output_2subjects_mac_1.8.0.nonRelease.20200915.108d5d65/
Model directory name:/Volumes/cbica/home/sakoc/CaPTk/PretrainedModels/SurvivalPredictor_PretrainedModel_1.8.0/
Module loaded: Survival Prediction on Existing Model:
Number of subjects with required input: 2
Segmentation fault: 11
sarthakpati commented 3 years ago

There are no changes when I am comparing the files Saima sent in email and the current master. Also, I am able to replicate this on Linux (non-cluster) using the subjects in @chiharusako's directory and the latest pre-trained model.

However, it worked on Windows. There might be something very trivial going on regarding how the paths are checked (for example) which is Windows-specific. Probably need to put in a bunch of std::cout statements to check what is going on in SurvivalPredictor.cpp before L556 (which is never hit on Linux/macOS).

sarthakpati commented 3 years ago

I think I have found the issue. I will push a fix and check it will work.

sarthakpati commented 3 years ago

Alright, so the only change was removing the extra , in the pre-trained model's Survival_ZScore_Mean file. I have uploaded the new model (same link) and I am able to run it from the new builds on the cluster (@chiharusako the version you tested on your macOS should also work).

chiharusako commented 3 years ago

@sarthakpati Thank you - now it produces an output for testing on cluster. I will test on more subjects and also test the training part, which will both take a little more time as I need to prepare the input