Getting what appears to be noise on Imagenette + XResnet

[sshleifer]

I am getting outputs that look completely random when I try to run distillation on a subset of imagenet with a XResnet 18 model.

I have only tried one set of command line args and was wondering whether you had any intuition for what I might obviously be doing wrong or had tried this before.

My command is:

python main.py --mode distill_basic --dataset Imagenette --arch DXResNet18 --batch_size 64 \
    --distill_steps 3 --train_nets_type known_init --n_nets 1 \
    --test_nets_type same_as_train

I made my own DXResnet18 class and Imagenette dataloader.

Thanks in advance!

[ssnl]

Fixed init should give you images that look like noise, as expected. :)

[sshleifer]

They look way worse than teaser.png. Were those created in a different way?

[ssnl]

I wouldn't be surprised. You have a much larger network with a much larger image space. There could be a lot more ways of using high freq signals to overfit a particular init.

[sshleifer]

I cut to only 10 classes, 13K examples. Would you suggest also cutting to smaller images/smaller network?

[ssnl]

Number of classes doesn’t matter much. I do think lower res would help. However, these said, it is fixed init, so random noise looks normal to me.

[sshleifer]

It did! Thank you for being so responsive!

Is the code to train a model using the images in results.pth invoked by running with --phase test?

I am trying to see whether can train good models on the distilled images with slightly different settings/ whether if you run hparam search on distilled images you get the same result as if you ran it on the orginal images.

[ssnl]

Is the code to train a model using the images in results.pth invoked by running with --phase test?

Yes. You can set different test lr settings, but for flexible and other hyperparameter tuning, you may want to implement your own :)

[ssnl]

I’m closing this one for now. Let me know if you need more help!