search

NiftyPET / NIPET

High-throughput PET image reconstruction with high quantitative accuracy and precision
Apache License 2.0
29 stars 7 forks source link

BLAS : Program is Terminated. Because you tried to allocate too many memory regions. #21

Open casperdcl opened 4 years ago

casperdcl commented 4 years ago

during nipet.mmrchain, unpredictably (could be after any number of iterations), the program often crashes despite having ample free memory.

i> using CUDA device #0
i> put gaps in and reorder sino...DONE in 0.001838s.
BLAS : Program is Terminated. Because you tried to allocate too many memory regions.
...
BLAS : Program is Terminated. Because you tried to allocate too many memory regions.

This seems to be fixed by running export OMP_NUM_THREADS=1 before launch. Unfortunately running os.environ["OMP_NUM_THREADS"] = "1" does not work as an alternative.

Easiest fix is to add this to NiftyPET:

if os.getenv("OMP_NUM_THREADS", None) != "1":
    raise EnvironmentError("should run `export OMP_NUM_THREADS=1` before launch")

Otherwise (proper fix) would involve passing in that environ flag somehow when calling the crashing libraries.