search

gorkaydemir / ADAPT

Official implementation of the ICCV 2023 paper "ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptation"
MIT License
70 stars 6 forks source link

cpu and gpu #7

Closed penglo closed 2 months ago

penglo commented 4 months ago

Hello,

First of all, thank you for your outstanding contributions. During the training process, I monitored the usage of the CPU and GPU. While the CPU's 64GB of memory was being utilized, the GPU was only using a small amount, around 6GB. Have you encountered this issue before?

gorkaydemir commented 4 months ago

Hi,

I suspect the bottleneck in the model is due to high CPU usage during the data loading step, while the actual model processing on the GPU is relatively minimal. I attempted to mitigate this by loading data samples directly from memory, rather than processing them during loading via a dataset class. However, this approach actually increased training time due to latency in storage I/O. Therefore, it seems this behavior is to be expected.

Best