Training time is super long for MatrixVT

[TmacTmac1992]

Hi,

I am training MatrixVT on 4A100, and it takes 12 hours to train one epoch. I am setting batch_size 8 for training. Do you know how to improve the training time?

python bevdepth/exps/nuscenes/MatrixVT/matrixvt_bev_depth_lss_r50_256x704_128x128_24e_ema.py --amp_backend native -b 8 --gpus 4

[ZRandomize]

Any detailed information? Are there any function-wise latency data? The get_proj_mat() function do slower the training a little bit, but it shouldn't be that slow

[zhangzaibin]

Have you solved this problem？ I encounter the same problem too.

[ChirsHuanhuan]

你好，你们的速度慢怎么解决