About Unable to open file Error in Implementation of Variational Network

[fharman]

Hello,

In my implementation of Variational Network, I have been always facing OSError: Unable to open file (file signature not found). (Especially retrieving and reading the metadata in mri_data.py). I used it for getting metadata and have never used an error like this. I don't understand what was updated there last days.

Have anyone encountered an error like this?

If anyone has any idea, could it be shared with me, too?

Best,

[mmuckley]

Hello @fharman, could you post the full stack trace? I'd like to know the calling function.

[fharman]

Sorry if I am wrong. I have been trying to understand the error.

Traceback (most recent call last): File "run_pretrained_varnet_inference.py", line 150, in torch.device(args.device), File "run_pretrained_varnet_inference.py", line 78, in run_inference root=data_path, transform=data_transform, challenge="multicoil" File "/content/fastMRI/fastmri/data/mri_data.py", line 268, in init metadata, num_slices = self._retrieve_metadata(fname) File "/content/fastMRI/fastmri/data/mri_data.py", line 305, in _retrieve_metadata with h5py.File(fname, "r") as hf: File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/files.py", line 427, in init swmr=swmr) File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/files.py", line 190, in make_fid fid = h5f.open(name, flags, fapl=fapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 96, in h5py.h5f.open OSError: Unable to open file (file signature not found)

[mmuckley]

Based on this it seems the file could possibly corrupted. Are you running this on the fastMRI data? Did you verify the sha hashes to make sure the download completed successfully?

[fharman]

Yes, I used multicoil FastMRI knee validation dataset. I extracted the tar.gz of the validation dataset and saved it. Upon seeing the dimension, I could not face a problem like this ( kx,kx,coil, and kz). I will check it again. Thank you for your return.

Matthew Muckley @.***>, 10 Ağu 2021 Sal, 15:33 tarihinde şunu yazdı:

Based on this https://stackoverflow.com/questions/38089950/error-opening-file-in-h5py-file-signature-not-found/43607837 it seems the file could possibly corrupted. Are you running this on the fastMRI data? Did you verify the sha hashes to make sure the download completed successfully?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/facebookresearch/fastMRI/issues/155#issuecomment-895989838, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO4IQ2RVYFKBZTCL4XKW7RLT4EMCZANCNFSM5B365LOA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[adithyaOvGu]

Sorry if I am wrong. I have been trying to understand the error.

Traceback (most recent call last): File "run_pretrained_varnet_inference.py", line 150, in torch.device(args.device), File "run_pretrained_varnet_inference.py", line 78, in run_inference root=data_path, transform=data_transform, challenge="multicoil" File "/content/fastMRI/fastmri/data/mri_data.py", line 268, in init metadata, num_slices = self._retrieve_metadata(fname) File "/content/fastMRI/fastmri/data/mri_data.py", line 305, in _retrieve_metadata with h5py.File(fname, "r") as hf: File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/files.py", line 427, in init swmr=swmr) File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/files.py", line 190, in make_fid fid = h5f.open(name, flags, fapl=fapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 96, in h5py.h5f.open OSError: Unable to open file (file signature not found)

Hi, This error usually occurs when the .h5 files are corrupted or if there are other hidden files such as ._DSstore. files inside the data folder. The latter occurs usually in macOS. Therefore, I would suggest you check if there are hidden files or files that are not of .h5 format in the folder and make sure the data you're trying to pull from the file is actually present.

[fharman]

Sorry if I am wrong. I have been trying to understand the error. Traceback (most recent call last): File "run_pretrained_varnet_inference.py", line 150, in torch.device(args.device), File "run_pretrained_varnet_inference.py", line 78, in run_inference root=data_path, transform=data_transform, challenge="multicoil" File "/content/fastMRI/fastmri/data/mri_data.py", line 268, in init metadata, num_slices = self._retrieve_metadata(fname) File "/content/fastMRI/fastmri/data/mri_data.py", line 305, in _retrieve_metadata with h5py.File(fname, "r") as hf: File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/files.py", line 427, in init swmr=swmr) File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/files.py", line 190, in make_fid fid = h5f.open(name, flags, fapl=fapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 96, in h5py.h5f.open OSError: Unable to open file (file signature not found)

Hi, This error usually occurs when the .h5 files are corrupted or if there are other hidden files such as ._DSstore. files inside the data folder. The latter occurs usually in macOS. Therefore, I would suggest you check if there are hidden files or files that are not of .h5 format in the folder and make sure the data you're trying to pull from the file is actually present.

Thank you for your return, too.

I could see the kspace. According to your advise, there is a crashed problem in metadata downloading and extracting.

I will check it again. Thank you!

[fharman]

Hello,

Firstly, sorry for my easy faults but I have been trying to overcome it. I redownloaded the multicoil_val again and extracted it. I have faced another error named as

File "/content/fastMRI/fastmri/data/mri_data.py", line 358, in getitem sample = self.transform(kspace, mask, target, attrs, fname.name, dataslice) File "/content/fastMRI/fastmri/data/transforms.py", line 434, in call mask = torch.from_numpy(mask.reshape(*mask_shape).astype(np.float32)) AttributeError: 'NoneType' object has no attribute 'reshape'

I have used https://github.com/facebookresearch/fastMRI/tree/f87e7a084f3a9b9df3d95721cc8c9b933c035c2a/data , before. and I have never faced an error like this. I don't know how to overcome it. Is there anyone who give me any suggestion about this?

Best,

[mmuckley]

Hello @fharman, what git hash are you on?

[fharman]

Hello, Sorry for misconnection. I used for Variational Network https://github.com/facebookresearch/fastMRI/tree/master/fastmri/data as you recommended in the Readme. Now I used it for Unet again, too. I have faced the same error. (There is no crashed data because I used the same data for the another network). I have been trying to understand where I am stucked in and why I am not going on.

Thank you for your support,

Best,

[mmuckley]

@fharman, my question was about the version of the software, not the model. How did you install the software? If via pip, what is the output of fastmri.__version__? If you installed from source, then what is the git hash from git log?

[fharman]

Hi,

I used pip and Name: fastmri, Version: 0.1.2a20210721. I installed from the source and the result of git log -1 --format="%H" is cfdb4784a4ea5d99a82a96a077345e02b9e668c8.

Thank you again.

[NayeeC]

Hello, I encounter a similar problem: I download the multi-coil brain data through NYU's link, and I just use the code below to do some tests:

files = list(Path(args.data_path).iterdir())
    for f in files:
        with h5py.File(f, 'r') as hf:
            print('h5 open successfully')

and there exits the same error:

Traceback (most recent call last):
  File "ch_test.py", line 33, in <module>
    main(args)
  File "ch_test.py", line 16, in main
    with h5py.File(f, 'r') as hf:
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\h5py\_hl\files.py", line 455, in __init__
    fid = make_fid(name, mode, userblock_size,
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\h5py\_hl\files.py", line 199, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5f.pyx", line 100, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

I tested these codes on a remote device, but when I test them on a local device, the error doesn't occur. I make the python version and h5py version the same between these two devices. I use pip install fastmri, and the fastmri version is 0.1.1.

BTW, I copy the date from the remote device to the local device, so I guess the data on the remote device is complete.

[NayeeC]

Hello, I encounter a similar problem: I download the multi-coil brain data through NYU's link, and I just use the code below to do some tests:

files = list(Path(args.data_path).iterdir())
    for f in files:
        with h5py.File(f, 'r') as hf:
            print('h5 open successfully')

and there exits the same error:

Traceback (most recent call last):
  File "ch_test.py", line 33, in <module>
    main(args)
  File "ch_test.py", line 16, in main
    with h5py.File(f, 'r') as hf:
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\h5py\_hl\files.py", line 455, in __init__
    fid = make_fid(name, mode, userblock_size,
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\h5py\_hl\files.py", line 199, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5f.pyx", line 100, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

I tested these codes on a remote device, but when I test them on a local device, the error doesn't occur. I make the python version and h5py version the same between these two devices. I use pip install fastmri, and the fastmri version is 0.1.1.

BTW, I copy the date from the remote device to the local device, so I guess the data on the remote device is complete.

I solve the problem. According to the @adithyaOvGu , I find some files, which are not .h5, I just delete them, and everything work.

[fharman]

Hello, I encounter a similar problem: I download the multi-coil brain data through NYU's link, and I just use the code below to do some tests:

files = list(Path(args.data_path).iterdir())
    for f in files:
        with h5py.File(f, 'r') as hf:
            print('h5 open successfully')

and there exits the same error:

Traceback (most recent call last):
  File "ch_test.py", line 33, in <module>
    main(args)
  File "ch_test.py", line 16, in main
    with h5py.File(f, 'r') as hf:
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\h5py\_hl\files.py", line 455, in __init__
    fid = make_fid(name, mode, userblock_size,
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\h5py\_hl\files.py", line 199, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5f.pyx", line 100, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

I tested these codes on a remote device, but when I test them on a local device, the error doesn't occur. I make the python version and h5py version the same between these two devices. I use pip install fastmri, and the fastmri version is 0.1.1.

BTW, I copy the date from the remote device to the local device, so I guess the data on the remote device is complete.

Hi,

I used jupyter notebook (not Pycharm) and implemented it in Google Colab Pro. After this error, I redownloaded the multicoil_val again for crashed meta data possibility. But I have had similar errors. I have 199 volume in multicoil_val that has the known custom number (not having some files to delete).

[NayeeC]

Hi @fharman, I used to do the following steps to test where exist the error:

1. copy just one .h5 file to a new directory
2. try to write a simple code to see whether you can just open this .h5 file.

if you can open it, then the multi-coil data is supposed to be complete. BTW, the folder where you store .h5 files should only contain .h5, no .ipynb, .py .txt etc.

[fharman]

Hello again,

I have divided volumes in Google Drive as small subsets ( to overcome Cuda Memory error in Google Colab Pro) and run another network. It worked. As I said before,I could see the kspace. Now I added a colab file to show you.

https://colab.research.google.com/drive/1A0o8FkTGGoqYPKSA3VpMxs9UAtaEcAj_#scrollTo=ZU35ykxXH9HG

Thank you for your return!

[NayeeC]

Hi, I notice that there is an error message in your traceback:

error message = 'Is a directory'

I am not sure, but maybe there is another directory in your data folder, you can give it another check. Or, you can try to output every name that the code got, fasmri load everything under the folder into a list, if the data is not corrupted, then I guess there may exist something wrong in that list.

[fharman]

Hi,

I have changed the full path and faced the similar error (another similar error). I am stuck in this and not overcoming it.

Thanks for support!

[mmuckley]

@fharman could you post a small, reproducible example of your code isolating the dataloader and without any NN model?

[fharman]

Hello,

I have just implemented i-RIM for small multicoil val dataset lastly. Colab link is https://colab.research.google.com/drive/1XyJed5FtWVpIWD7gsqa7-7jdBaZxSi7S#scrollTo=ZU35ykxXH9HG

Thanks,

[mmuckley]

Hello @fharman, did you fix this code? I just got a chance to look at it and it looks like it ran successfully.

[fharman]

Hello,

I have just run https://colab.research.google.com/drive/1A0o8FkTGGoqYPKSA3VpMxs9UAtaEcAj_#scrollTo=ZU35ykxXH9HG right now. But I have still have the same OSError. It did not work and I did not fix it. But i-RIM for multicoil validation dataset works with submodule path 'external/fastMRI' (https://github.com/facebookresearch/fastMRI.git) as you see above colab link. I know they use different hash f87e7a084f3a9b9df3d95721cc8c9b933c035c2a. I checked it.

Thank you for your return!

[mmuckley]

Hello @fharman, I'm wondering if the data directory is incorrect. Usually --data_path will be something like path/multicoil_val or something like that.

In this case there is a folder in the directory you specified (which is not a .h5 file), and that folder is causing the error since it is not a file.

[fharman]

Hello,

I revised the colab and changed the path. https://colab.research.google.com/drive/1A0o8FkTGGoqYPKSA3VpMxs9UAtaEcAj_#scrollTo=oClqPXbXsjXL

Hello,

Firstly, sorry for my easy faults but I have been trying to overcome it. I redownloaded the multicoil_val again and extracted it. I have faced another error named as

File "/content/fastMRI/fastmri/data/mri_data.py", line 358, in getitem sample = self.transform(kspace, mask, target, attrs, fname.name, dataslice) File "/content/fastMRI/fastmri/data/transforms.py", line 434, in call mask = torch.from_numpy(mask.reshape(*mask_shape).astype(np.float32)) AttributeError: 'NoneType' object has no attribute 'reshape'

I have seen another problem as before. I am missing something out. Sorry for then and Thanks for return!

Best,

[mmuckley]

@fharman, now it is erroring because no mask was loaded. The validation data does not have a mask saved to file because it is the fully-sampled data. If you want to run the inference script, you can run it on the test data (which has been subsampled and has a mask saved to the file). Otherwise, you need to subsample the data yourself and provide the sampling mask to the model.

[fharman]

Ohh, ok! I thought it was like i-RIM.

Thank you Matt!

1 Eyl 2021 Çar 22:30 tarihinde Matthew Muckley @.***> şunu yazdı:

@fharman https://github.com/fharman, now it is erroring because no mask was loaded. The validation data does not have a mask saved to file because it is the fully-sampled data. If you want to run the inference script, you can run it on the test data (which has been subsampled and has a mask saved to the file). Otherwise, you need to subsample the data yourself and provide the sampling mask to the model.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/facebookresearch/fastMRI/issues/155#issuecomment-910629528, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO4IQ2UIFUW6TUFEW3SURFTT7Z5N3ANCNFSM5B365LOA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[mattnsoleil]

Hi,

In test dataset, It gives TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'NoneType'>. There is a problem in dataset loading images with the PIL library and does not convert them into necessary types like list or Tensor. Is there anyone who overcome this problem? Because I could not overcome with editting transform = transforms.Compose([ transforms.ToTensor() ]).

[mmuckley]

Closing this as it seems the issue has been resolved.