search

NUS-HPC-AI-Lab / Neural-Network-Parameter-Diffusion

We introduce a novel approach for parameter generation, named neural network parameter diffusion (p-diff), which employs a standard latent diffusion model to synthesize a new set of parameters
828 stars 42 forks source link

Training and diffusion using vit and convnext #9

Open Weigaa opened 6 months ago

Weigaa commented 6 months ago

I read your neural network diffusion paper and it was great and had a lot to say about us. I would like to try to reproduce them and have reproduced the case of resnet18 so far. I see that your paper also reports the experimental results of vit and convnext for cifar and imagent. I see that in the models folder you provided, there seems to be no vit and convnext models, and all the models seem to be tuned specifically for the cifar input (32*32). I tried training the model parameters directly using the vit model provided by torchvision, but it doesn't seem to integrate directly with hydra? Can you please provide the vit and convnext models you used for the experiment? Or can you teach me how to convert the torchvision model into a nnd usable model?

Thank you very much for your help.

1zeryu commented 6 months ago

Thanks for your feedback. Could you provide more details about your code and experiment? I'm more than willing to help you deal with them

Weigaa commented 6 months ago

Thank you, I want to use NND to diffuse vit model parameters, but I can not find the model in this git repository/models. Could you please provide the model file which used in you paper? Thx

1zeryu commented 6 months ago

You can use vit models and checkpoints in torchvision/models for pre-trained models on the ImageNet dataset. And finetune it to another dataset. reference code repo: https://github.com/lucidrains/vit-pytorch.

Weigaa commented 6 months ago

This github repository doesn't seem to be the official vit in torchvision/models, which seems to have a much simpler implementation. However, this repository doesn't seem to provide a checkpoint, and it looks like I won't be able to use the checkpoint provided by torchvision/models with the vit modeling code in this repository.

The ref repo you gave me seems much more concise than torchvision's vit model, but that may mean I have to train this model from scratch myself using ImageNet.

Are you saying that I need to modify the vit model in this "repo: https://github.com/lucidrains/vit-pytorch." and put it in the models directory of your NND repo so that it can be called correctly by Hydra?

Thank you.