八卡3090训练的问题

wzzheng / TPVFormer

[CVPR 2023] An academic alternative to Tesla's occupancy network for autonomous driving.

https://wzzheng.net/TPVFormer/

Apache License 2.0

1.16k stars 105 forks source link

八卡3090训练的问题 #28

Closed w-yibo closed 1 year ago

w-yibo commented 1 year ago

首先感谢您非常好的工作。我复现的过程中，按照Train TPVFormer for lidar segmentation task on 3090 with 24G GPU memory.的步骤，四卡GPU训练时可以正常训练，但是使用八卡3090时会出现“_pickle.UnpicklingError: pickle data was truncated”的问题，请问是哪里出了问题呢

huang-yh commented 1 year ago

We run into the same error when we set num_workers for dataloader to a value larger than one. And when we used 0 or 1 for num_workers, this problem went away.

zhangzaibin commented 1 year ago

请问八卡3090需要跑多久？