Transfer learning / fine tuning with ActionFormer

[christian-matroid]

Hello, thank you for all your work! I am wondering if there has been any attempt to use transfer learning in ActionFormer. For example, training on EpicKitchens and fine-tuning to ActivityNet, or another more limited dataset.

Any insights, methods or results would be greatly appreciated. Thanks!

[tzzcl]

Sorry for the late reply, We haven't tried transfer learning with ActionFormer. The common practice is that, with strong backbone features, you do not need to perform transfer learning with ActionFormer.

[happyharrycn]

A limiting factor for transfer learning is the significant domain shift between many existing video datasets. Take the two datasets mentioned in your question for example. Epic-Kitchens focuses on first person videos of meal preparation tasks, while ActivityNet contains YouTube videos of daily activities. The duration / frequency / coverage of the actions in these two datasets also differs drastically. Due to these major gaps, I don't think training on one and fine-tuning on the other can be helpful.

On the other hand, if you are thinking about two datasets that are "similar", say Ego4D and Epic-Kitchens. This transfer learning using fine-tuning might be beneficial.

[christian-matroid]

Thank you for your responses. I have been able to implement fine-tuning across existing datasets. As you mentioned the results on any individual dataset are limited by the drastic domain shift between training and fine-tuning data, however it has been more successful than training from scratch when attempting to learn the task on a very limited fine-tuning dataset.

[happyharrycn]

Great to hear that. I am going to mark this as closed.

[EmreOzkose]

Hi,

@christian-matroid, If you don't mind, can you share your fine-tunning code ?

[qingqianqiana]

谢谢您的回复。我已经能够对现有数据集进行调整。正如您所提到的，任何单个数据集的结果都受到训练和数据集之间的模拟域转换的限制，但是当尝试在非常有限的时扭矩数据集上学习任务时，它比从头开始训练更成功。

Can you please share your fine-tuning code with me?