daviddmc / NeSVoR

NeSVoR is a package for GPU-accelerated slice-to-volume reconstruction.
MIT License
69 stars 16 forks source link

"SVD did not converge" error when using nesvor segment-stack #15

Closed GerardMJuan closed 11 months ago

GerardMJuan commented 11 months ago

I have an issue when using nesvor segment-stack. It fails, giving a SVD error when trying to save the . I have tried with different images, so it is not related to the input. I am using the docker image, version 0.5.0. I have also tried it in a HPC environment (converting the image to a singularity container), getting the same error.

This is the full error stack:

WARNING:root:Fail to load tinycudann. Will use pytorch implementation.
/usr/local/lib/python3.10/dist-packages/nibabel/nifti1.py:1088: RuntimeWarning: invalid value encountered in divide
  R = RZS / zooms
/usr/local/lib/python3.10/dist-packages/numpy/linalg/linalg.py:2139: RuntimeWarning: invalid value encountered in det
  r = _umath_linalg.det(a, signature=signature)
ERROR:root:Unhandled exception:
Traceback (most recent call last):
  File "/usr/local/bin/nesvor", line 33, in <module>
    sys.exit(load_entry_point('nesvor', 'console_scripts', 'nesvor')())
  File "/usr/local/NeSVoR/nesvor/cli/main.py", line 23, in main
    run(args)
  File "/usr/local/NeSVoR/nesvor/cli/main.py", line 48, in run
    getattr(commands, command_class)(args).main()
  File "/usr/local/NeSVoR/nesvor/cli/commands.py", line 75, in main
    self.exec()
  File "/usr/local/NeSVoR/nesvor/cli/commands.py", line 257, in exec
    outputs(
  File "/usr/local/NeSVoR/nesvor/cli/io.py", line 93, in outputs
    m.save(p)
  File "/usr/local/NeSVoR/nesvor/image/image.py", line 187, in save
    save_nii_volume(path, output_volume, affine)
  File "/usr/local/NeSVoR/nesvor/image/image_utils.py", line 110, in save_nii_volume
    img = nib.nifti1.Nifti1Image(volume, affine)
  File "/usr/local/lib/python3.10/dist-packages/nibabel/nifti1.py", line 1848, in __init__
    super().__init__(dataobj, affine, header, extra, file_map, dtype)
  File "/usr/local/lib/python3.10/dist-packages/nibabel/analyze.py", line 909, in __init__
    super().__init__(dataobj, affine, header, extra, file_map)
  File "/usr/local/lib/python3.10/dist-packages/nibabel/spatialimages.py", line 531, in __init__
    self.update_header()
  File "/usr/local/lib/python3.10/dist-packages/nibabel/nifti1.py", line 2273, in update_header
    super().update_header()
  File "/usr/local/lib/python3.10/dist-packages/nibabel/nifti1.py", line 1884, in update_header
    super().update_header()
  File "/usr/local/lib/python3.10/dist-packages/nibabel/spatialimages.py", line 565, in update_header
    self._affine2header()
  File "/usr/local/lib/python3.10/dist-packages/nibabel/nifti1.py", line 1894, in _affine2header
    hdr.set_qform(self._affine, code='unknown')
  File "/usr/local/lib/python3.10/dist-packages/nibabel/nifti1.py", line 1100, in set_qform
    P, S, Qs = npl.svd(R)
  File "<__array_function__ internals>", line 200, in svd
  File "/usr/local/lib/python3.10/dist-packages/numpy/linalg/linalg.py", line 1642, in svd
    u, s, vh = gufunc(a, signature=signature, extobj=extobj)
  File "/usr/local/lib/python3.10/dist-packages/numpy/linalg/linalg.py", line 98, in _raise_linalgerror_svd_nonconvergence
    raise LinAlgError("SVD did not converge")
numpy.linalg.LinAlgError: SVD did not converge

Note that the rest of the tool works great: when using Monaifbs separately and providing NeSVoR with the masks, everything works smoothly. So a workaround right now is to compute the masks outside of the docker.

It is not a memory issue, and I have manually checked the images and they have no issues (Monaifbs works well with the same images when used in my own installation).

daviddmc commented 11 months ago

Hi @GerardMJuan, it seems to be a problem related to the affine matrix in nii file. Could share me with one stack of your data to test? If you are not able to share the data, could you share with me the affine matrix as follows?

import nibabel as nib
img = nib.load(file_path)
print(img.header["pixdim"][1:4])
print(img.affine)
print(img.get_qform())
print(img.get_sform())
GerardMJuan commented 11 months ago

Hi,

I am not able to share the data, and did not find any issues with the affine matrix. However, I just tried it now with same command, same image, same data and it seems to work now. The most plausible explanation for this change, I believe, is that I was running it on a HPC environment and the GPU/node I was selecting last week had some type of weird incompatibility, and the one I am selecting now is not.

Anyway, now it works perfectly well. Sorry for bothering you. Closing this issue. If in the future I am able to reproduce it and have more information I will let you know.

Best, Gerard