search

zaccharieramzi / fastmri-reproducible-benchmark

Try several methods for MRI reconstruction on the fastmri dataset. Home to the XPDNet, runner-up of the 2020 fastMRI challenge.
https://fastmri.org/leaderboards
MIT License
151 stars 50 forks source link

Problem too many files open #40

Closed zaccharieramzi closed 4 years ago

zaccharieramzi commented 4 years ago

When training the pdnet, I hit an error after 24 iterations (~4 hours so not very reproducible) that says: OSError: [Errno 24] Too many open files.

I checked the number of processes created when fitting, it is very high for the second epoch (above 35 specified, double that).

I need to reproduce this to create a tf issue.

I think this has to do with sequences so trying right now with datasets and it looks ok, also the speed is equivalent so early try-outs were not representative of the real use case.

zaccharieramzi commented 4 years ago

This problem seems to be resolved with the use of datasets instead of sequences. Even if those datasets use under the hood the python hdf5 opener and not the tailored HDF5IO function of tensorflow-io.

Will update all training scripts and notebooks with this.

zaccharieramzi commented 4 years ago

Solved by #43