search

open-mmlab / mmaction2

OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark
https://mmaction2.readthedocs.io
Apache License 2.0
4.04k stars 1.2k forks source link

[Docs] How to train a model with different inputs from different model feature and videos #2805

Open zeynepgokce opened 3 months ago

zeynepgokce commented 3 months ago

The doc issue

Hello,

I need a model training code which takes two different inputs, first one comes from pre-trained another model feature and second input from video inputs. I need to train the my main model using these two different input. I could not handle this problem in mmaction2. I dont know how to start it? Is there any sample code for this purpose? Please help me.

Thank you.

Suggest a potential alternative/fix

Is there any sample code for this purpose? If it does not exist, could you please provide a documentation about the customization of a model with various inputs or traning codes?

valentin-fngr commented 3 months ago

Not an active contributor but here is how I would deal with it for now (as you can see, it can take up to weeks before getting an answer from the devs ...).

I would suggest to make a new class using https://github.com/open-mmlab/mmaction2/blob/4d6c93474730cad2f25e51109adcf96824efc7a3/mmaction/models/recognizers/recognizer2d.py#L11

as template.

Modify this class such that it take your input feature as well. Additionally, look into models that also deal with optical flow as they use the optical flow as additional features.