search

nicklashansen / tdmpc

Code for "Temporal Difference Learning for Model Predictive Control"
MIT License
346 stars 55 forks source link

Multimodal data as input to the model #10

Closed SergioArnaud closed 1 year ago

SergioArnaud commented 1 year ago

Hi, congratulations on the amazing work!

I wanted to ask a question, the paper mentions that multimodal data [RGB + proprioception] can be used as input of the model

In the code, the observations are sent to an encoder that process them in different ways depending if it's pixels or another modality, nevertheless I'm not sure that any of those options apply to multimodal data containing both pixels and state information. Given the experiments you made in the paper, how would you recommend processing such data in the encoder?

nicklashansen commented 1 year ago

Hi, thank you for your interest. We recently open-sourced an extension to the TD-MPC algorithm which takes multimodal data (pixels + state) inputs by default. It is available here: https://github.com/facebookresearch/modem/blob/6cee94def92b910a3fe122a10dcec1330c3519c3/algorithm/tdmpc.py#L37 Modalities are fused by projecting features from each modality into a low-dim space and summing them. Feel free to re-open if you have additional questions!