Add an argument that can be passed to deepspeed in the pretrain and finetune launch scripts. This argument will be used to pick which VE class to use.
Types needed: CLIP, OCLIP, timm
For OCLIP models it is also necessary to pass the config of the VE to enable local models (see load_model in open_clip.py), and if the model uses Timm in OpenClip
Working directory: robin/model/multimodal_encoder/
Additional objectives:
Add an argument that can be passed to deepspeed in the pretrain and finetune launch scripts. This argument will be used to pick which VE class to use. Types needed: CLIP, OCLIP, timm For OCLIP models it is also necessary to pass the config of the VE to enable local models (see load_model in open_clip.py), and if the model uses Timm in OpenClip Working directory: robin/model/multimodal_encoder/ Additional objectives:
model_name == ...