Processing error

[cjacobs1]

The log shows this. I've put the test data in

Digest: sha256:224954b28ad6a5805c6796fde1127fd3a066f9165f16a835da4a5ba8dcf4fa13
Status: Downloaded newer image for oni:11500/uoks/bodyct-kaggle-grt123:processor
oni:11500/uoks/bodyct-kaggle-grt123:processor
Invoking: ./start-job.py --user=colin --ticket=8483 --ngpus=1 --mem=30g --cpus=8 --create-only --run=oni:11500/uoks/bodyct-kaggle-grt123:processor --volume /mnt/netcache/bodyct/experiments/itr_cad_evaluation_t8071/grt123-p3-archive:/input:ro --volume /mnt/netcache/bodyct/experiments/itr_cad_evaluation_t8071/grt123-p3-archive-output-all:/output --parent_pid=83373 --
Using gpus: GPU-556014e4-1e63-afc2-449c-ff862419b5b7
Extra arguments:
7a38b9050492c1d0d5d81674d049286bd7406b87bc9619086d7451c20891bc38
Created container 7a38b9050492c1d0d5d81674d049286bd7406b87bc9619086d7451c20891bc38; starting...
7a38b9050492c1d0d5d81674d049286bd7406b87bc9619086d7451c20891bc38
----------
{'output_prep_dir': '/output/1.2.276.0.7230010.3.1.3.921054970.9700.1555410110.633/prep', 'outputfile': None, 'skip_preprocessing': False, 'detector_param': './model/detector.ckpt', 'detector_model': 'net_detector', 'use_existing_preprocessing': True, 'n_gpu': 1, 'classifier_model': 'net_classifier', 'datapath': '/input/1.2.276.0.7230010.3.1.3.921054970.9700.1555410110.633', 'classifier_num_nodules_for_cancer_decision': 5, 'crop_rects_outputfile': None, 'n_worker_preprocessing': 8, 'output_convert_debug_file': None, 'skip_detect': False, 'output_bbox_dir': '/output/1.2.276.0.7230010.3.1.3.921054970.9700.1555410110.633/bbox', 'classifier_max_nodules_to_include': None, 'outputdir': '/output/1.2.276.0.7230010.3.1.3.921054970.9700.1555410110.633', 'classifier_param': './model/classifier.ckpt'}
Processing 1 files with 8 workers
multiprocessing.Pool error:  
(Re-)computing preprocessing results with a single thread...
  Loading /input/1.2.276.0.7230010.3.1.3.921054970.9700.1555410110.633
Traceback (most recent call last):
  File "main.py", line 238, in <module>
    main(**config_submit)
  File "main.py", line 58, in main
    use_existing=use_existing_preprocessing)
  File "/home/user/DSB2017/preprocessing/full_prep.py", line 240, in full_prep
    savenpy(i, filelist, prep_folder, data_path, use_existing=use_existing)
  File "/home/user/DSB2017/preprocessing/full_prep.py", line 129, in savenpy
    box = np.array([[np.min(xx), np.max(xx)], [np.min(yy), np.max(yy)],
  File "/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 2420, in amin
case_path=/input/1.2.276.0.7230010.3.1.3.921054970.9700.1555410110.633, data_path=/input, name=1.2.276.0.7230010.3.1.3.921054970.9700.1555410110.633
binarize... 1.9556350708
slicing... 10.6087679863
fill_hole... 2698.74687195
two_lung_only... 2712.07392287
end step1 2775.86559391
bug in 1.2.276.0.7230010.3.1.3.921054970.9700.1555410110.633
    out=out, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/numpy/core/_methods.py", line 29, in _amin
    return umr_minimum(a, axis, None, out, keepdims)
ValueError: zero-size array to reduction operation minimum which has no identity
Traceback (most recent call last):
  File "apply.py", line 107, in <module>
    cwd=os.path.join(THIS_DIR, "DSB2017"))
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
input dir = /input/1.2.276.0.7230010.3.1.3.921054970.9700.1555410110.633, output dir = /output/1.2.276.0.7230010.3.1.3.921054970.9700.1555410110.633
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python2', 'main.py']' returned non-zero exit status 1

[silvandeleemput]

I just reran the case and I identify this as an error in the preprocessing, just before saving the preprocessed data. Sadly it takes over an hour to get to the error. But at least I got a little bit more information that isn't truncated by the apply.py in the output trace above:

Traceback (most recent call last):
  File "/mnt/synology/neuro/sil/bodyct-kaggle-grt123/main.py", line 251, in <module>
bug in 1.2.276.0.7230010.3.1.3.921054970.9700.1555410110.633
    main(**config_submit)
  File "/mnt/synology/neuro/sil/bodyct-kaggle-grt123/main.py", line 59, in main
    use_existing=use_existing_preprocessing)
  File "/mnt/synology/neuro/sil/bodyct-kaggle-grt123/preprocessing/full_prep.py", line 240, in full_prep
    savenpy(i, filelist, prep_folder, data_path, use_existing=use_existing)
  File "/mnt/synology/neuro/sil/bodyct-kaggle-grt123/preprocessing/full_prep.py", line 129, in savenpy
    box = np.array([[np.min(xx), np.max(xx)], [np.min(yy), np.max(yy)],
  File "/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 2420, in amin
    out=out, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/numpy/core/_methods.py", line 29, in _amin
    return umr_minimum(a, axis, None, out, keepdims)
ValueError: zero-size array to reduction operation minimum which has no identity

The responsible lines in the savenpy method:

        newshape = np.round(np.array(Mask.shape) * spacing / resolution)
        xx, yy, zz = np.where(Mask)
        box = np.array([[np.min(xx), np.max(xx)], [np.min(yy), np.max(yy)],
                        [np.min(zz), np.max(zz)]])

My guess is that an invalid bounding box (probably empty) is extracted at that location, which will yield an error. I am currently retrying with the debugger at the point of error to investigate what the preprocessing mask looks like.

[silvandeleemput]

Ok, I just figured it out, the case is completely filled with a single value: -32768. The lung mask then becomes an empty image. Then the np.where(Mask) returns empty lists. Then it crashes on np.min. We might need some exception handling for this kind of case.

[cjacobs1]

OK, thanks! I wonder what's going on. Perhaps an error in the dicom reading because it should not just contain -32768 values.. What dicom reader is used here? the diag dicom loader?

[silvandeleemput]

It uses the diag dataloader. It seems to happen both there and in Mevislab using the directdicomimport module.

[cjacobs1]

OK, then there is something wrong with the dicom data. I will check tomorrow

[cjacobs1]

@silvandeleemput I made a mistake when creating the input folders (the dicom loader was loading asynchronously, so it start writing out the new dicom before the pixeldata was actually loaded). I am recreating the data now.