search

shubham-goel / 4D-Humans

4DHumans: Reconstructing and Tracking Humans with Transformers
https://shubham-goel.github.io/4dhumans/
MIT License
1.23k stars 118 forks source link

DDP has slower training speed #124

Open jyuntins opened 5 months ago

jyuntins commented 5 months ago

Hi, I am trying to Finetune HMR2.0 on 3 RTX3090. It works fine when I only train it on 1 gpu. When I set trainer.devices=3, I received the error: ValueError: ctypes objects containing pointers cannot be pickled

I found a workaround to solve this is to use ddp strategy just as you did in the ddp trainer config file. However, when I set the trainer to ddp, the speed drops from 2.5 it/s to 7s/it.

I wonder which config file is used when you trained hmr2.0? Is there anyway to accelerate the training speed in this configuration?

wtx9527 commented 3 months ago

Hi, I have the same problem, have you solved it?

jyuntins commented 3 months ago

Hi, @wtx9527 No, I didn't solve it.

wangsen1312 commented 1 month ago

I met the same problem when I use the multi GPU training