search

Henry1iu / TNT-Trajectory-Prediction

A Pytorch Implementation of TNT: Target-driveN Trajectory Prediction
487 stars 92 forks source link

Process is killed when transforming to Graph Data #36

Closed alfredgu001324 closed 1 year ago

alfredgu001324 commented 1 year ago

Hi,

Thanks for sharing this great repo. I have finished the data pre-processing. However, when I tried to run the train_tnt.py using interm_data, the "Transforming the data to GraphData..." process always stopped at 77% and the program got killed. I am wondering what might be the cause of this? I have checked that I still have 100GB of free space available so the disk space should not be the issue? When I run the train_tnt.py using the interm_data_small, everything is just fine.

Thank you so much for your help and looking forward to your reply.

Henry1iu commented 1 year ago

Hi,

Thanks for your appreciation. I think it may be caused by the limitation of memory. What's your memory size (memory + swap)? The transformation consumes a large memory space (128G+).

Henry1iu commented 1 year ago

If you don't have enough memory. You can try to use the InDiskLoader. However, this will make the training take longer time.

alfredgu001324 commented 1 year ago

Hi Henry, thanks for the reply. By typing in 'free -h' into terminal, I found that the Mem is 31G and Swap is 18G. So I guess indeed it's due to the limitation of memory. If I were to use the 'InDiskLoader', I would just change lines 21&22 in the train_tnt.py to use 'ArgoverseInDisk', is that correct?

Thank you so much for your reply and advice and really appreciate it.

Wei233666 commented 1 month ago

Hello! I use the server memory 250G+, why run to about 50% will stop