Compressed file ended before the end-of-stream marker was reached

LucaGherardini commented 2 years ago

Hello, I'm using this handy pipeline with many heterogeneous DWI images. In my dataset, I have some ex multi-shells which have been extracted into single shells (keeping only 0, 1000 as bvals, and their corresponding bvecs) but I have this error EOFError: Compressed file ended before the end-of-stream marker was reached I made the single shell extraction by hand, and it is identical to the outputs of other commands (I.e. dwiextract of mrtrix3). It's not raising complaints either from the eddy correction tool in FSL suite (which usually is quite precise with bvals and bvecs specification). Original single shell files are ok, so I suppose it's something I didn't care about during single shell extraction, but I would like to have more precise information on this error.

Below you can find the traceback:

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/net/people/plglucaghera/miniconda3/envs/harmonization/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/net/pr1/plgrid/plggsano/Luca_Gherardini/Dataset/Harmonization/dMRIharmonization/lib/preprocess.py", line 93, in preprocessing
    lowResImg = lowRes.get_data().astype('float')
  File "/net/people/plglucaghera/miniconda3/envs/harmonization/lib/python3.9/site-packages/nibabel/deprecator.py", line 183, in deprecated_func
    return func(*args, **kwargs)
  File "/net/people/plglucaghera/miniconda3/envs/harmonization/lib/python3.9/site-packages/nibabel/dataobj_images.py", line 207, in get_data
    data = np.asanyarray(self._dataobj)
  File "/net/people/plglucaghera/miniconda3/envs/harmonization/lib/python3.9/site-packages/nibabel/arrayproxy.py", line 391, in __array__
    arr = self._get_scaled(dtype=dtype, slicer=())
  File "/net/people/plglucaghera/miniconda3/envs/harmonization/lib/python3.9/site-packages/nibabel/arrayproxy.py", line 358, in _get_scaled
    scaled = apply_read_scaling(self._get_unscaled(slicer=slicer), scl_slope, scl_inter)
  File "/net/people/plglucaghera/miniconda3/envs/harmonization/lib/python3.9/site-packages/nibabel/arrayproxy.py", line 332, in _get_unscaled
    return array_from_file(self._shape,
  File "/net/people/plglucaghera/miniconda3/envs/harmonization/lib/python3.9/site-packages/nibabel/volumeutils.py", line 522, in array_from_file
    n_read = infile.readinto(data_bytes)
  File "/net/people/plglucaghera/miniconda3/envs/harmonization/lib/python3.9/gzip.py", line 300, in read
    return self._buffer.read(size)
  File "/net/people/plglucaghera/miniconda3/envs/harmonization/lib/python3.9/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/net/people/plglucaghera/miniconda3/envs/harmonization/lib/python3.9/gzip.py", line 506, in read
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/net/pr1/plgrid/plggsano/Luca_Gherardini/Dataset/Harmonization/dMRIharmonization/lib/harmonization.py", line 538, in <module>
    pipeline.run()
  File "/net/people/plglucaghera/miniconda3/envs/harmonization/lib/python3.9/site-packages/plumbum/cli/application.py", line 624, in run
    retcode = inst.main(*tailargs)
  File "/net/pr1/plgrid/plggsano/Luca_Gherardini/Dataset/Harmonization/dMRIharmonization/lib/harmonization.py", line 513, in main
    self.createTemplate()
  File "/net/pr1/plgrid/plggsano/Luca_Gherardini/Dataset/Harmonization/dMRIharmonization/lib/harmonization.py", line 203, in createTemplate
    targetImgs, targetMasks= common_processing(self.tar_unproc_csv)
  File "/net/pr1/plgrid/plggsano/Luca_Gherardini/Dataset/Harmonization/dMRIharmonization/lib/preprocess.py", line 182, in common_processing
    attributes= [r.get() for r in res]
  File "/net/pr1/plgrid/plggsano/Luca_Gherardini/Dataset/Harmonization/dMRIharmonization/lib/preprocess.py", line 182, in <listcomp>
    attributes= [r.get() for r in res]
  File "/net/people/plglucaghera/miniconda3/envs/harmonization/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
EOFError: Compressed file ended before the end-of-stream marker was reached

tashrifbillah commented 2 years ago

Markdowns are enclosed within triple backticks, not single--note for you for future.

tashrifbillah commented 2 years ago

EOFError: Compressed file ended before the end-of-stream marker was reached

I believe this means your .gz files were not written properly which I can only pin on the method of your splitting.

Here's our reliable program for multi shell harmonization: https://github.com/pnlbwh/multi-shell-dMRIharmonization

If you need shell-splitting, you can use this script: https://github.com/pnlbwh/multi-shell-dMRIharmonization/blob/master/lib/separateBshells.py

For the --ref_bshell_file, I believe you need only:

0
1000

in a text file. But I would let you figure otherwise.

LucaGherardini commented 2 years ago

@tashrifbillah thank you very much for your fast response and for the suggestion about Markdown. I will try to shift my method as you suggested. Have a nice day

pnlbwh / dMRIharmonization

Compressed file ended before the end-of-stream marker was reached #82