search

allenai / allenact

An open source framework for research in Embodied-AI from AI2.
https://www.allenact.org
Other
313 stars 50 forks source link

Adding CLIP encoders #329

Closed apoorvkh closed 2 years ago

apoorvkh commented 2 years ago

Adding visual and text encoders from CLIP for use in RoboTHOR ObjectNav via a new clip_plugin. Can install CLIP via the clip_plugin extra requirements.

Can invoke in "zeroshot" mode (where objects are split into seen/unseen sets and their names are encoded with CLIP's text encoder). Or, can just replace the visual encoder with CLIP's ResNet.

Training example

PYTHONPATH=. allenact -b projects/objectnav_baselines/experiments/robothor/clip objectnav_robothor_zeroshot_rgb_clipgru_ddppo
lgtm-com[bot] commented 2 years ago

This pull request introduces 5 alerts when merging e0c2060300ab79bdcea8cd917232424d837d3620 into 9da8674e7781370b4c257eab707a613e953c002f - view on LGTM.com

new alerts: