GPU->CPU Memcpy failed

[Jacksonm75]

when I run paper_eval.py with cpu, it works but too slow. However, with gpu it reports errors like ‘GPU->CPU Memcpy failed’, how can I handle it?

[zaccharieramzi]

Hi @Jacksonm75 ,

Sorry for answering so late and thank you for your interest in this repo. Indeed the eval surely will be slow on CPU.

How big is your GPU's memory? You might be facing a hidden out-of-memory error: https://github.com/tensorflow/tensorflow/issues/8710

One solution, but a bit involved, might be to reduce the batch size by changing these lines:

• https://github.com/zaccharieramzi/fastmri-reproducible-benchmark/blob/9d699f49531baeffef5b83b550fc471c0ec257d6/fastmri_recon/evaluate/reconstruction/unet_reconstruction.py#L5
• https://github.com/zaccharieramzi/fastmri-reproducible-benchmark/blob/9d699f4953/fastmri_recon/evaluate/reconstruction/cross_domain_reconstruction.py#L5

to use predict instead of predict_on_batch with this time a smaller batch size.

[zaccharieramzi]

Hi @Jacksonm75 ,

Did you manage to fix this issue? If so can we close it?

[zaccharieramzi]

Closing this for clarity, but feel free to reopen if needed.