Domain Adaptation with MMD in subsection 3.2.2

[deep0learning]

You did additional experiments with MMD loss instead of associative loss. Can you explain a bit more how you implemented MMD loss in your architecture?

[deep0learning]

This is the graphs for SVHN → MNIST that I got. So what is the accuracy for SVHN → MNIST? Is it 99.99%?

[haeusser]

Hi @deep0learning,

Here is the implementation of the MMD loss.

For further details (in particular the source), please have a look at Section 3.2.2 of the paper.

With regard to your plots, there is no evaluation data. Please run eval.py with the respective flags in order to evaluate the model.

I hope this is helpful. Philip

[deep0learning]

@haeusser Thank you so much.

mkdir /home/ubuntu/datasets/; mkdir /home/ubuntu/datasets/svhn/; mkdir /home/ubuntu/datasets/mnist/; wget http://ufldl.stanford.edu/housenumbers/test_32x32.mat -O /home/ubuntu/datasets/svhn/test_32x32.mat; wget http://ufldl.stanford.edu/housenumbers/train_32x32.mat -O /home/ubuntu/datasets/svhn/train_32x32.mat; wget http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz -O /home/ubuntu/datasets/mnist/train-images-idx3-ubyte.gz; wget http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz -O /home/ubuntu/datasets/mnist/train-labels-idx1-ubyte.gz; wget http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz -O /home/ubuntu/datasets/mnist/t10k-images-idx3-ubyte.gz; wget http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz -O /home/ubuntu/datasets/mnist/t10k-labels-idx1-ubyte.gz;

python semisup/train.py \ --target_dataset="mnist3" \ --walker_weight_envelope_delay=500 \ --new_size=32 \ --dataset="svhn" \ --sup_per_batch=100 \ --decay_steps=9000 \ --unsup_batch_size=1000 \ --sup_per_class=-1 \ --walker_weight_envelope_steps=1 \ --walker_weight_envelope="linear" \ --visit_weight_envelope="linear" \ --architecture="svhn_model" \ --visit_weight=0.2 \ --max_steps=12000 \ --logdir=./log/svhn_to_mnist/reproduce

And eval script:

python semisup/eval.py \ --target_dataset="mnist3" \ --walker_weight_envelope_delay=500 \ --new_size=32 \ --dataset="svhn" \ --sup_per_batch=100 \ --decay_steps=9000 \ --unsup_batch_size=1000 \ --sup_per_class=-1 \ --walker_weight_envelope_steps=1 \ --walker_weight_envelope="linear" \ --visit_weight_envelope="linear" \ --architecture="svhn_model" \ --visit_weight=0.2 \ --max_steps=12000 \ --logdir=./log/svhn_to_mnist/reproduce

But get the results. Am I doing anything wrong?

When I run evaluate script it gives me this error:

INFO:tensorflow:Timed-out waiting for a checkpoint.

Another thing I would like to know from you that after running this file : mnist_train_eval.py I am getting test error. Is it the script for SVHN-> MNIST transfer task ?

[haeusser]

The script does domain adaptation once you define the flag target_dataset.

Regarding the evaluation problem, please see my response to your post in issue 3.

[deep0learning]

By running this code, I am getting the following results: Test error: 90.20 % for SVHN -> MNSIT. So what is the accuracy for SVHN -> MNSIT ?? Is it 100 - 90.20 = 9.80 % ???

python semisup/mnist_train_eval.py \ --target_dataset="mnist3" \ --walker_weight_envelope_delay=500 \ --new_size=32 \ --dataset="svhn" \ --sup_per_batch=100 \ --decay_steps=9000 \ --unsup_batch_size=1000 \ --sup_per_class=-1 \ --walker_weight_envelope_steps=1 \ --walker_weight_envelope="linear" \ --visit_weight_envelope="linear" \ --architecture="svhn_model" \ --visit_weight=0.2 \ --max_steps=12000 \ --logdir=/media/mahfuj/DATA/New_CODE/Working/learning_by_association/log/svhn_to_mnist/reproduce1

[deep0learning]

Thank you so much. Now It's working now.

[deep0learning]

I am running the scrpts and get the results. It seems that SVHN->MNIST: 99.87% accuracy. Is it?? In your paper it is: 93.71%.

python semisup/train.py --target_dataset="mnist3" --walker_weight_envelope_delay=500 --new_size=32 --dataset="svhn" --sup_per_batch=100 --decay_steps=9000 --unsup_batch_size=1000 --sup_per_class=-1 --walker_weight_envelope_steps=1 --walker_weight_envelope="linear" --visit_weight_envelope="linear" --architecture="svhn_model" --visit_weight=0.2 --max_steps=12000 --logdir=./log/svhn_to_mnist/reproduce

And eval script:

python semisup/eval.py --target_dataset="mnist3" --walker_weight_envelope_delay=500 --new_size=32 --dataset="svhn" --sup_per_batch=100 --decay_steps=9000 --unsup_batch_size=1000 --sup_per_class=-1 --walker_weight_envelope_steps=1 --walker_weight_envelope="linear" --visit_weight_envelope="linear" --architecture="svhn_model" --visit_weight=0.2 --max_steps=12000 --logdir=./log/svhn_to_mnist/reproduce

[haeusser]

Our reported accuracy is 97.6%. The number in parentheses is the improvement over the „source only“ trained model (cf. table caption for a longer explanation).

Depending on the initialization, you might obtain better results than what we reported in the paper, which is the mean of 10 runs. If you run the same experiment a few more times, you should get a similar number.

If not, congrats - you got the new state of the art :-)

[deep0learning]

@haeusser Thank you so much for your kind reply.

Actually, when I was run the code with MMD loss I got also 99.90 Accuracy. I am sure that I did some mistake. That's why I would like to know is there anything that I need to change in the code in eval.py or train.py script.

In eval.py, In line 39: Is is ok? Or I have to write down mnist3 as my target data is mnist3 in this experiment. Please give me answer. Thanks again for your kind help.

flags.DEFINE_string('dataset', 'svhn', 'Which dataset to work on.')

And in train.py [Line 40], Should I define target_dataset as 'mnist3' instead of None?

flags.DEFINE_string('target_dataset', None, 'If specified, perform domain adaptation using dataset as ' 'source domain and target_dataset as target domain.')