preprocessing luna gets wrong

Bonsen commented 7 years ago

starting preprocessing luna 252 278 556 523 746 435 231 MetaImage: Read: Cannot open data file 166 608 409 722 179 276 788 547 807 549 659 383 575 182 706 Traceback (most recent call last): File "prepare.py", line 378, in preprocess_luna() File "prepare.py", line 287, in preprocessluna =pool.map(partial_savenpy_luna,range(N)) File "/usr/lib64/python2.7/multiprocessing/pool.py", line 250, in map return self.map_async(func, iterable, chunksize).get() File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get raise self._value RuntimeError: Exception thrown in SimpleITK ReadImage: /tmp/SimpleITK-build/ITK/Modules/IO/Meta/src/itkMetaImageIO.cxx:483: itk::ERROR: MetaImageIO(0x6f83b30): File cannot be read: /data/tmp/luna_data/15.mhd for reading. Reason: Success ^Z[4] 退出 1 nohup python -u prepare.py 2>&1

lfz commented 7 years ago

I don't know the reason because it's in a pool function and I haven't faced it before, any more detail?

Bonsen commented 7 years ago

@lfz system: centos7.1 python 2.7, CUDA 8.0, cudnn 5.1, h5py (2.6.0), SimpleITK (0.10.0), numpy (1.11.3), nvidia-ml-py (7.352.0), matplotlib (2.0.0), scikit-image (0.12.3), scipy (0.18.1), pyparsing (2.1.4), pytorch (0.1.10+ac9245a)

config = {'stage1_data_path':'/home/stage1/', //// 7za x stage1.7z -r -o./home/stage1/

      'luna_raw':'/home/luna16/data/original_lungs/',  ///original_lungs/subset(0-9)/

// 7za x /data/images/CT/luna16data/subset*.zip -r -o./original_lungs/ //in ./original_lungs/subset0/ like : //1.3.6.1.4.1.14519.5.2.1.6279.6001.979083010707182900091062408058.mhd //1.3.6.1.4.1.14519.5.2.1.6279.6001.979083010707182900091062408058.raw

      'luna_segment':'/home/luna16/seg-lungs-LUNA16/',    ///like:

// inflating: 1.3.6.1.4.1.14519.5.2.1.6279.6001.162845309248822193437735868939.mhd // inflating: 1.3.6.1.4.1.14519.5.2.1.6279.6001.162845309248822193437735868939.zraw

      'luna_data':'/home/dsb2017/data/tmp/luna_data',
      'preprocess_result_path':'/home/dsb2017/data/tmp/preprocess_result_path/',

      'luna_abbr':'./detector/labels/shorter.csv',
      'luna_label':'./detector/labels/lunaqualified.csv',
      'stage1_annos_path':['./detector/labels/label_job5.csv',
            './detector/labels/label_job4_2.csv',
            './detector/labels/label_job4_1.csv',
            './detector/labels/label_job0.csv',
            './detector/labels/label_qualified.csv'],
      'bbox_path':'../detector/results/res18/bbox/',
      'preprocessing_backend':'python'
     }

After unzip them ,I run prepare.py and get these.

mileyan commented 7 years ago

I have find the bug. In the Luna_data folder, we do not add zero prefix to the file name when the length of file name less than 3, such as 32.mhd, 54.mhd. However, in the luna segmentation folder, we add the zero prefix, such as 032.mhd. And we use the file names in Luna_data folder as index. So we cannot find the file in segmentation folder when file name less than 3.

lfz commented 7 years ago

Thank you for the feedback, I have fixed this bug

fd24b36e6c30c60a0d488716c175ae267a0d5760

lfz / DSB2017

preprocessing luna gets wrong #5