search

motional / nuplan-devkit

The devkit of the nuPlan dataset.
https://www.nuplan.org
Other
662 stars 126 forks source link

my ML model was not running on GPU in closed-loop simulation #294

Closed Yisten closed 1 year ago

Yisten commented 1 year ago

GPU utilization rate was keeping 0%. What should I do to make it on GPU to accelerate the simulation?

patk-motional commented 1 year ago

Hi @Yisten,

Let's try to isolate the issue first. Can you try switching to the sequential worker by setting worker=sequential?

Yisten commented 1 year ago

Thanks for your immediate reply!

It seems that GPU works when I conduct my simulations using Sequential. But I am still confused why it does not when using RayDistributed. What is the difference between these two modes and how should I choose either of them.

gianmarco-motional commented 1 year ago

Hi @Yisten,

Ray distributed uses multi-processing (the simulation runs on multiple CPU cores in parallel). In order to give GPU access to each process, fractional GPU access would need to be set up with Docker. I'm not even sure that is possible at the moment. To avoid any kind of race conditions to resource access each simulation-submission pair will have a dedicated GPU.