openai / Video-Pre-Training

Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos
MIT License
1.36k stars 146 forks source link

loading IDM weights without VPT's lib code? #19

Closed verbose-void closed 2 years ago

verbose-void commented 2 years ago

i'm having a lot of trouble trying to get the inverse dynamics model weights to work in a separate custom torch nn module. all the custom model code is spread out through VPT's lib modules and the complexity is a bit overwhelming for me.

do you have any bare-bones loading methods for the weights? or do you think the complexity is just the nature of the beast

i'd like to basically just use IDM's latent representation (before it goes to the policy head) and add my own custom policy head to it. but doing so ruptures a lot of the functionality.

Miffyli commented 2 years ago

Hmm unfortunately I think the code is mostly as barebones it can get; you could remove some features that are not used and skip parts here and there, but not by much. Alternativaly you could simplify things, but our first aim was to share code asap and as close to original format as possible to avoid breaking anything.

You are right the model code itself could probably be smaller, but shaping the inputs for the model in correct way are also important: otherwise you can not use the model right.

verbose-void commented 2 years ago

@Miffyli right, so is your recommendation to copy the code and add it to my submission?

Miffyli commented 2 years ago

Yes, for the BASALT competition, you should copy this code as part of your submission.

Small protip: default .gitignore templates include lib as one of the ignored names, which leads to the lib directory being ignored by default 🙃

verbose-void commented 2 years ago

@Miffyli copy that, thanks king