search

dchen236 / FairFace

465 stars 92 forks source link

code to train #11

Open jlim13 opened 3 years ago

jlim13 commented 3 years ago

is there code to train and reproduce results?

cpshaheen commented 3 years ago

@dchen236 @Bernardo1998 - can you guys please provide the following for experimental validity:

  1. Model parameters and architecture.
  2. Code showing the training process.
  3. Code showing the evaluation process.

@kylemcdonald has opened an issue detailing a discrepancy between testing and the results published in the paper. I also have had the same results and would appreciate some clarity regarding this. It seems like the repo has been updated within the last month but neither this issue or the one Kyle raised were addressed? Don't mean to bother you two too much but it would be great if you could follow up with the addition of the code used for training and testing so that we can benefit from your findings or help refine the findings if it is the case that the reported findings were significantly off from the true metrics for some unknown reason.

Thanks and best wishes!

jlim13 commented 3 years ago

Can we get some follow on this please? It doesn't really make sense to release a dataset paper without a script to reproduce your baselines.

joojs commented 3 years ago

Our whole codebase is very big and uses other data which we can't release, so it will be hard to release the whole thing. We may be able to clear up the part separately, but not sure when. We provided the dataset, inference code and pretrained models, which should be sufficient for most use cases.

For the evaluation accuracy in the other issue: The result in Table 6 in arxiv (table 8 in wacv paper) was measured on the "external validation datasets". The paper explains how they were collected and evaluated in detail. We are not able to release these datasets because these are not under CC license. The pre-trained model is the one used in our experiments in the paper.

cpshaheen commented 3 years ago

Our whole codebase is very big and uses other data which we can't release, so it will be hard to release the whole thing. We may be able to clear up the part separately, but not sure when. We provided the dataset, inference code and pretrained models, which should be sufficient for most use cases.

For the evaluation accuracy in the other issue: The result in Table 6 in arxiv (table 8 in wacv paper) was measured on the "external validation datasets". The paper explains how they were collected and evaluated in detail. We are not able to release these datasets because these are not under CC license. The pre-trained model is the one used in our experiments in the paper.

Appreciate the response. Thanks for getting back!

anderleich commented 2 years ago

Any updates on this?