search

isaac-sim / IsaacGymEnvs

Isaac Gym Reinforcement Learning Environments
Other
1.91k stars 410 forks source link

RuntimeError: CUDA error: an illegal memory access was encountered #45

Open maniast opened 2 years ago

maniast commented 2 years ago

I keep getting this at random time periods during my training. The specific error:

/buildAgent/work/45f70df4210b2e3e/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (9908) : internal error : GPU compressContactStage1 fail to launch kernel stage 1!!

/buildAgent/work/45f70df4210b2e3e/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (9945) : internal error : GPU compressContactStage2 fail to launch kernel stage 1!!

yilin-wu98 commented 2 years ago

I may have encountered a similar issue. Have you found any way to solve it? Which task are you training?

MyelinsheathXD commented 2 years ago

having the same issue. All AMP samples crashing on start

some common errors

ed: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 5993 [Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 937 [Error] [carb.gym.plugin] Failed to fill root state tenso

MyelinsheathXD commented 1 year ago

Solution is to decrease batch size , contact nums, env_nums. Since these vatiables' memory allocation consuming required pytorch's memory allocation

explore-code-world7 commented 2 months ago

I find the problem in a while circle. image

So I stopped the circle after first implement,then I got another previous error: image This is caused by implementgym.simulate(sim) before gym.prepare_sim(sim)，I added it before the while circle containing gym.simulate(sim)，the problem was resolved