search

ZikangZhou / HiVT

[CVPR 2022] HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction
https://openaccess.thecvf.com/content/CVPR2022/papers/Zhou_HiVT_Hierarchical_Vector_Transformer_for_Multi-Agent_Motion_Prediction_CVPR_2022_paper.pdf
Apache License 2.0
577 stars 115 forks source link

torch.load bottleneck? #33

Closed moonseokha closed 1 year ago

moonseokha commented 1 year ago

Hi. I'm interested in your research, so I'm looking at it hard. But when I start training code, at first, gpu is used as a high utilization as data enters well through torch.load, as steps progress, data loading time of the torch.load becomes longer and gpu utill 0, a bottleneck occurs. Have you ever had a problem like this?

It is the estimated time required and gpu util before the bottleneck.

스크린샷 2023-01-23 오후 4 38 06 스크린샷 2023-01-23 오후 4 38 26

After bottleneck

스크린샷 2023-01-23 오후 4 37 58 스크린샷 2023-01-23 오후 4 38 17

Thank U!

ggosjw commented 1 year ago

Hi, i have the same issue. May I how do you solve this problem?

Liwen-Xiao commented 1 week ago

I have the same problem. This seems to be the bottleneck that comes with mechanical hard drives image