search

pumpikano / tf-dann

Domain-Adversarial Neural Network in Tensorflow
MIT License
628 stars 224 forks source link

Theoretical question - Domain accuarcy #22

Open jecaJeca opened 6 years ago

jecaJeca commented 6 years ago

Hello,

Thanks for sharing this code! I have couple of question, more theoretical then practical :)

When I run MNIST example for 86000 steps (10 times more then original), I got very poor results (around 20% acc on source and target sets), so I am wondering what is a problem. I run couple of times more and got ok results, but I am wondering what leads to poor results in some executions.

First, I am not sure what should I expect for domain accuracy - should it be around 50% or it is not important. I think that is better to look at domain predictions, because I should expect probabilities around 50% (domain discriminator is not sure what is source what is target example), so maybe accuracy itself is not important. I think that problem with poor results become in situation where domain discriminator predicts only one class with probabilities 1 (loss is very huge so in that case it ruins feature extractor and classification accuracy).

Another thing which confuses me is could DANN gets good results even if domain accuracy is 1. So, domain discriminator can decide which is source which is target example (with huge probabilities), but results on target data are good.

Generally, I am wondering how can I track if my DANN works ok, which metrics to use.

I hope that you can help me, maybe I totally wrong understand concept :)

Thanks a lot!

zhasan49 commented 6 years ago

I observe something similar with quite a lot of variability between results. It appears that during training the domain loss can suddenly explode. Sometimes training can recover other times it diverges completely and you get the very poor results. I do not know the cause. The above is only noticeable if trained for a >10000 steps.

@pumpikano Any ideas on what it could be? Increasing batch size to 128 seems to reduce the problem.

I am currently trying to get the code working on the SVHN dataset in order to recreate the results from the Ganin paper but currently am unable to due despite using the same architecture as in the paper. Even with a slightly simpler architecture from the Bousmalis 2016 paper I encounter this exploding domain loss issue. I'm not sure what is causing it. This is with a batch size of 128 as per the paper.

avital commented 5 years ago

@zhasan49 I'm also been trying unsuccessfully to reproduce the SVHN -> MNIST results from the original Ganin paper. Have you (or anyone else) had luck on this?