search

tristandeleu / pytorch-meta

A collection of extensions and data-loaders for few-shot learning & meta-learning in PyTorch
https://tristandeleu.github.io/pytorch-meta/
MIT License
1.98k stars 256 forks source link

Implementing Meta-Curvature #77

Open sudarshan1994 opened 4 years ago

sudarshan1994 commented 4 years ago

Right now Meta-Curvature seems to be the best performing MAML variant for various few-shot learning tasks. In this technique, they learn precondition matrices for the gradients in the inner loop. I feel it would be useful to have a general framework where we could have a transform called inner gradient transform, which can modulate the inner loop gradients. So in this framework Meta-Curvature would be a special case. I would be happy to contribute if possible.

tristandeleu commented 4 years ago

This would be a great contribution indeed! You can have a look at the gradient_update_parameters for inspiration. Having a general transform sounds like an excellent idea, I would also suggest adding some utility functions (as syntactic sugar) for some standard methods (MAML or Meta-Curvature).

sudarshan1994 commented 4 years ago

Hey thanks for getting back, I went through the code base--its a super useful contribution, thank the whole team for this ! I think a class wrapping the gradient_update_parameters which takes in gradient transform as an argument in its init would be a neat way to implement meta-curvature, plus other gradient transformations could be seamlessly integrated into the system this way.

sudarshan1994 commented 4 years ago

Hey, so we have an implementation of metacurvature using pytorch-meta and these are the results on miniImageNet: 52 % for 1-shot and 63.8 % for 5-shot. The meta-curvature paper implements a couple of regularization tricks : cut-out and a data augmentation trick, plus they use a much wider CNN (128 filters), so the numbers we have cant be directly compared to the original paper. However this paper, has an implementation of meta-curvature with the standard CNN (32 filters) and does not apply any regularization tricks: Our 5-shot result on miniImageNet matches here (unfortunately they dont report 1-shot results :/)

qbouniot commented 4 years ago

Hi ! Could you share your implementation of Meta-Curvature ? I would be really interested in trying it out. Thank you !

sudarshan1994 commented 4 years ago

Sure I ll clean up the code, give me a couple of weeks

mfischer-ucl commented 3 years ago

Hi @sudarshan1994 , any updates on this? I'd be interested in your implementation of Meta-Curvature as well! Thank you :)