gmargo11
commented
1 year ago Hi @thisisnotahuman ,
How much GPU memory do you have available? (You can check this by running nvidia-smi from the command line)
If your GPU is smaller than 10GB, you can try reducing the number of environments (Cfg.env.num_envs) from 4000 to 2000 or fewer.
-Gabe
hi guys, i ran into this problem when i ran: python train.py. i already set the num_envs to 1 but it doesn't work, any idea how to handle this?
PxgCudaDeviceMemoryAllocator fail to allocate memory 339738624 bytes!! Result = 2 PxgCudaDeviceMemoryAllocator fail to allocate memory 339738624 bytes!! Result = 2 PxgCudaDeviceMemoryAllocator fail to allocate memory 339738624 bytes!! Result = 2 PxgCudaDeviceMemoryAllocator fail to allocate memory 339738624 bytes!! Result = 2 /buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2155) : internal error : memcpy failed fail! 2 700
/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2165) : internal error : GPU cudaMainGjkEpa or prepareLostFoundPairs kernel fail! 700
/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (54) : internal error : SynchronizeStreams cuEventRecord failed
/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (60) : internal error : SynchronizeStreams cuStreamWaitEvent failed
/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2408) : internal error : memcpy failed fail! 700
/buildAgent/work/99bede84aa0a52c2/source/gpucommon/src/PxgCudaMemoryAllocator.cpp (59) : warning : Failed to allocate pinned memory.