search

akanazawa / hmr

Project page for End-to-end Recovery of Human Shape and Pose
Other
1.54k stars 395 forks source link

Are the amount of data important? #72

Closed gsygsy96 closed 5 years ago

gsygsy96 commented 5 years ago

Hi, @akanazawa , your work is very fantastic! But I have a question on your dataset choice. So many datasets are employed in your experiment, so have you test the performance when some dataset are ablated?

akanazawa commented 5 years ago

Hi @mehameha998,

Thank you! This is a good question. We didn't do this, but I believe this paper retrained a model that's only trained with Human3.6M. As you can see from there and also from my initial experiments, the performance goes down significantly when it's just Human3.6M. I have not ablated with/without LSP, but to be honest it's only 10k images, as opposed to there are 60k COCO. I think having LSP definitely does help in recovering upside down people that happen during sports, because COCO is mostly of people standing up. But I think the most important data is COCO.

Hope this helps.

Angjoo

monajalal commented 4 years ago

Hi Angjoo,

  1. Do you mean that COCO is more important than the H3.6M dataset also or just more important than LSP?

  2. Does it mean that H3.6M with much more datapoint than COCO is less effective due to lack of diversity in data? Or are there other potential reasons than diversity?

  3. Just to make sure I understood correctly if I train a model with only COCO, should I expect to get more accurate results than training a model with only H3.6M?

Thank you, Mona

akanazawa commented 4 years ago

Hi Mona,

  1. For generalization to in-the-wild images, it's very important to train on COCO. LSP is not bad but it's not that big in comparison.

  2. Yes training on H3.6M alone will not work on COCO like images because the H36M is captured in a controlled environment.

  3. This depends on the dataset you're using to evaluate on -- if you don't train on H36M, the model will not do as well on H36M and the model that's trained on H36M will do well on it. I think it's best to evaluate on models on a dataset like 3DPW. I suggest you look at more recent papers like SPIN, which I like a lot. There it looks like H36M is always used also, I don't think we have seen a systematic evaluation on 3DPW with/without H36M. I don't think it will help that much, it would be interesting to see, but I think the idea here has been why not use data that's available (unless ofc it hurts).

Best,

A