learnables / learn2learn

A PyTorch Library for Meta-learning Research
http://learn2learn.net
MIT License
2.66k stars 353 forks source link

Add functorch improvements to inner-loop algorithms #331

Open DubiousCactus opened 2 years ago

DubiousCactus commented 2 years ago

With PyTorch's latest update, the introduction of the functorch library allows, via composable function transforms, to "efficiently batching together tasks in the inner-loop of MAML". There seems to be a new way to vectorise gradient computation on a batch of tasks during the adaptation phase. I will try and look into this when I have some time, and maybe submit a PR. Unless someone knows how to do this? :)

seba-1511 commented 1 year ago

I'm not sure batching tasks will be very helpful because we're already running OOM when looping through individual tasks with MAML. I'll leave this open for now and we can see if there's more interest in the future.