search

NifTK / NiftyNet

[unmaintained] An open-source convolutional neural networks platform for research in medical image analysis and image-guided therapy
http://niftynet.io
Apache License 2.0
1.37k stars 404 forks source link

Compressed file ended before the end-of-stream marker was reached #148

Closed JunMa11 closed 6 years ago

JunMa11 commented 6 years ago

Dear nifytnet organizers, When I try to inference (segmentation application), only two files are segmented in my image folder (there are 66 cases totally).

EOFError: Compressed file ended before the end-of-stream marker was reached

I'm really confused on this error. By any chance, could you give me some insights about the error?

Following is the error detail information :

Number of subjects 66, input section names: ['subject_id', 'Brain']
-- using all subjects (without data partitioning).
2018-07-08 22:23:17.514130: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1423] Adding visible gpu devices: 0
2018-07-08 22:23:17.521093: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-08 22:23:17.536887: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:917]      0
2018-07-08 22:23:17.547346: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:930] 0:   N
2018-07-08 22:23:17.551358: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1041] Created TensorFlow device (/device:GPU:0 with 3084 MB memory) -> physical GPU (device: 0, name: GeForce GTX 950M, pci bus id: 0000:01:00.0, compute capability: 5.0)
INFO:niftynet: starting from iter 4000
INFO:niftynet: Accessing E:\Data\Brats2018\Brats18_Code\DDBrFlairWhole\models\model.ckpt-4000 ...
INFO:niftynet: Restoring parameters from E:\Data\Brats2018\Brats18_Code\DDBrFlairWhole\models\model.ckpt-4000
Traceback (most recent call last):
  File "E:\DLCode\NiftyNet\niftynet\engine\image_window_buffer.py", line 148, in _push
    for output_dict in self():
  File "E:\DLCode\NiftyNet\niftynet\engine\sampler_resize.py", line 73, in layer_op
    image_id, data, interp_orders = self.reader(shuffle=self.shuffle)
  File "E:\DLCode\NiftyNet\niftynet\layer\base_layer.py", line 34, in __call__
    return self._op(*args, **kwargs)
  File "D:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\template.py", line 455, in __call__
    result = self._call_func(args, kwargs)
  File "D:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\template.py", line 400, in _call_func
    result = self._func(*args, **kwargs)
  File "E:\DLCode\NiftyNet\niftynet\io\image_reader.py", line 230, in layer_op
    {field: image.get_data() for (field, image) in image_dict.items()}
  File "E:\DLCode\NiftyNet\niftynet\io\image_reader.py", line 230, in <dictcomp>
    {field: image.get_data() for (field, image) in image_dict.items()}
  File "E:\DLCode\NiftyNet\niftynet\io\image_type.py", line 431, in get_data
    self.file_path[0], self.loader[0], self.dtype[0])
  File "E:\DLCode\NiftyNet\niftynet\io\image_type.py", line 510, in _load_single_file
    file_path, loader, dtype)
  File "E:\DLCode\NiftyNet\niftynet\io\image_type.py", line 412, in _load_single_file
    image_data = image_obj.get_data()  # new API: get_fdata()
  File "D:\ProgramData\Anaconda3\lib\site-packages\nibabel\dataobj_images.py", line 202, in get_data
    data = np.asanyarray(self._dataobj)
  File "D:\ProgramData\Anaconda3\lib\site-packages\numpy\core\numeric.py", line 544, in asanyarray
    return array(a, dtype, copy=False, order=order, subok=True)
  File "D:\ProgramData\Anaconda3\lib\site-packages\nibabel\arrayproxy.py", line 290, in __array__
    raw_data = self.get_unscaled()
  File "D:\ProgramData\Anaconda3\lib\site-packages\nibabel\arrayproxy.py", line 285, in get_unscaled
    mmap=self._mmap)
  File "D:\ProgramData\Anaconda3\lib\site-packages\nibabel\volumeutils.py", line 524, in array_from_file
    n_read = infile.readinto(data_bytes)
  File "D:\ProgramData\Anaconda3\lib\gzip.py", line 276, in read
    return self._buffer.read(size)
  File "D:\ProgramData\Anaconda3\lib\_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "D:\ProgramData\Anaconda3\lib\gzip.py", line 482, in read
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached

originally defined at:
  File ".\net_run.py", line 8, in <module>
    sys.exit(main())
  File "E:\DLCode\NiftyNet\niftynet\__init__.py", line 138, in main
    app_driver.initialise_application(system_param, input_data_param)
  File "E:\DLCode\NiftyNet\niftynet\engine\application_driver.py", line 201, in initialise_application
    data_param, app_param, data_partitioner)
  File "E:\DLCode\NiftyNet\niftynet\application\segmentation_application.py", line 80, in initialise_dataset_loader
    inference_reader = ImageReader({'image'})
  File "E:\DLCode\NiftyNet\niftynet\io\image_reader.py", line 82, in __init__
    super(ImageReader, self).__init__(name='image_reader')
  File "E:\DLCode\NiftyNet\niftynet\layer\base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "D:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\template.py", line 153, in make_template
    **kwargs)

Exception in thread Thread-1:
Traceback (most recent call last):
  File "D:\ProgramData\Anaconda3\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "D:\ProgramData\Anaconda3\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "E:\DLCode\NiftyNet\niftynet\engine\image_window_buffer.py", line 148, in _push
    for output_dict in self():
  File "E:\DLCode\NiftyNet\niftynet\engine\sampler_resize.py", line 73, in layer_op
    image_id, data, interp_orders = self.reader(shuffle=self.shuffle)
  File "E:\DLCode\NiftyNet\niftynet\layer\base_layer.py", line 34, in __call__
    return self._op(*args, **kwargs)
  File "D:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\template.py", line 455, in __call__
    result = self._call_func(args, kwargs)
  File "D:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\template.py", line 400, in _call_func
    result = self._func(*args, **kwargs)
  File "E:\DLCode\NiftyNet\niftynet\io\image_reader.py", line 230, in layer_op
    {field: image.get_data() for (field, image) in image_dict.items()}
  File "E:\DLCode\NiftyNet\niftynet\io\image_reader.py", line 230, in <dictcomp>
    {field: image.get_data() for (field, image) in image_dict.items()}
  File "E:\DLCode\NiftyNet\niftynet\io\image_type.py", line 431, in get_data
    self.file_path[0], self.loader[0], self.dtype[0])
  File "E:\DLCode\NiftyNet\niftynet\io\image_type.py", line 510, in _load_single_file
    file_path, loader, dtype)
  File "E:\DLCode\NiftyNet\niftynet\io\image_type.py", line 412, in _load_single_file
    image_data = image_obj.get_data()  # new API: get_fdata()
  File "D:\ProgramData\Anaconda3\lib\site-packages\nibabel\dataobj_images.py", line 202, in get_data
    data = np.asanyarray(self._dataobj)
  File "D:\ProgramData\Anaconda3\lib\site-packages\numpy\core\numeric.py", line 544, in asanyarray
    return array(a, dtype, copy=False, order=order, subok=True)
  File "D:\ProgramData\Anaconda3\lib\site-packages\nibabel\arrayproxy.py", line 290, in __array__
    raw_data = self.get_unscaled()
  File "D:\ProgramData\Anaconda3\lib\site-packages\nibabel\arrayproxy.py", line 285, in get_unscaled
    mmap=self._mmap)
  File "D:\ProgramData\Anaconda3\lib\site-packages\nibabel\volumeutils.py", line 524, in array_from_file
    n_read = infile.readinto(data_bytes)
  File "D:\ProgramData\Anaconda3\lib\gzip.py", line 276, in read
    return self._buffer.read(size)
  File "D:\ProgramData\Anaconda3\lib\_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "D:\ProgramData\Anaconda3\lib\gzip.py", line 482, in read
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached

originally defined at:
  File ".\net_run.py", line 8, in <module>
    sys.exit(main())
  File "E:\DLCode\NiftyNet\niftynet\__init__.py", line 138, in main
    app_driver.initialise_application(system_param, input_data_param)
  File "E:\DLCode\NiftyNet\niftynet\engine\application_driver.py", line 201, in initialise_application
    data_param, app_param, data_partitioner)
  File "E:\DLCode\NiftyNet\niftynet\application\segmentation_application.py", line 80, in initialise_dataset_loader
    inference_reader = ImageReader({'image'})
  File "E:\DLCode\NiftyNet\niftynet\io\image_reader.py", line 82, in __init__
    super(ImageReader, self).__init__(name='image_reader')
  File "E:\DLCode\NiftyNet\niftynet\layer\base_layer.py", line 26, in __init__
    self._op = tf.make_template(name, self.layer_op, create_scope_now_=True)
  File "D:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\ops\template.py", line 153, in make_template
    **kwargs)

WARNING:niftynet: File E:\Data\Brats2018\Pre_Brats18_Validation\Seg_V_BrainFlair4k\Brats18_CBICA_AAM_1_niftynet_out.nii.gz exists, overwriting the file.
Saved E:\Data\Brats2018\Pre_Brats18_Validation\Seg_V_BrainFlair4k\Brats18_CBICA_AAM_1_niftynet_out.nii.gz
INFO:niftynet: Inference iter 0,  (3.361948s)
INFO:niftynet: Cleaning up...
WARNING:niftynet: stopped early, incomplete loops
INFO:niftynet: stopping sampling threads
INFO:niftynet: DVNApplication stopped (time in second 5.12).
wyli commented 6 years ago

Hi @EdwardMa12593, looks like at least one of your 'nii.gz' files is corrupted. You can quickly double check the files by:

import nibabel
for f_name in my_files:
    print(f_name)
    nibabel.load(f_name).get_data()
JunMa11 commented 6 years ago

Dear @wyli , thanks for your reply very much. You are right, One of the nii.gz files is corruped.