model architectures and pretrained models to support

[dmarx]

installable

• [ ] https://github.com/salesforce/LAVIS
  • https://github.com/salesforce/BLIP
  • https://github.com/salesforce/ALBEF
• [ ] https://github.com/facebookresearch/multimodal
  • FLAVA
  • LateFusion
  • ALBEF
  • MDETR
  • OMNIVORE
  • video-gpt
• [ ] https://github.com/ai-forever/ru-clip
• [ ] https://github.com/FreddeFrallan/Multilingual-CLIP
• [ ] https://github.com/lucidrains/x-clip
  • see citations for supported pre-trained checkpoints
• [ ] https://github.com/ai-forever/ru-dolph
• [ ] https://github.com/optas/artemis
• [ ] https://github.com/CompVis/net2net
• [ ] LiT - https://github.com/google-research/vision_transformer/
• [ ] (old) - DAMSM - https://github.com/taoxugit/AttnGAN
• [ ] https://github.com/facebookresearch/vilbert-multi-task
• [ ] https://github.com/Aleph-Alpha/magma
• [ ] https://github.com/sallymmx/ActionCLIP
• [ ] https://github.com/facebookresearch/mmf
• [ ] https://github.com/rinnakk/japanese-clip
• [ ] https://github.com/j-min/clip-caption-reward
• [ ] https://github.com/facebookresearch/omnivore
• [ ] https://github.com/microsoft/GLIP
• [ ] https://github.com/microsoft/RegionCLIP
  • setup.py is actually for detectron2
• [ ] https://github.com/yusufani/TrCLIP
• [ ] https://github.com/facebookresearch/detectron2

installable with extra effort

• [ ] https://github.com/Sense-GVT/DeCLIP
  • installs into a generic "prototype" package that could conflict with other packages using the same "spring" starter template
  • dataclasses==0.8 in requirements.txt throws version conflict

Not installable

• [ ] https://github.com/Kaushalya/medclip
• [ ] https://github.com/clip-italian/clip-italian
• [ ] https://github.com/j-min/VL-T5
• [ ] https://github.com/ashkamath/mdetr
• [ ] https://github.com/drboog/Lafite
• [ ] https://github.com/pbaylies/Augmented_CLIP
• [ ] (old) - https://github.com/sidward14/Style-AttnGAN
• [ ] https://github.com/luogen1996/MCN
• [ ] https://github.com/luogen1996/SimREC
• [ ] https://github.com/facebookresearch/Detic
• [ ] https://github.com/henghuiding/Vision-Language-Transformer
• [ ] https://github.com/allenai/reclip
• [ ] https://github.com/ChenRocks/UNITER
• [ ] https://github.com/raoyongming/DenseCLIP
• [ ] https://github.com/ttlmh/Bridge-Prompt
• [ ] https://github.com/sonoisa/clip-japanese
• [ ] https://github.com/facebookresearch/OTTER
• [ ] https://socraticmodels.github.io/
• [ ] https://github.com/yxuansu/MAGIC
• [ ] https://github.com/mindspore-ai/models/tree/master/research/mm/wukong | https://wukong-dataset.github.io/wukong-dataset/benchmark.html
• [ ] https://github.com/microsoft/UniCL
• [ ] https://github.com/goel-shashank/CyCLIP
• [ ] https://github.com/FacePerceiver/FaRL#use-farl-as-faceclip
• [ ] https://github.com/Lednik7/CLIP-ONNX/tree/main/clip_onnx
• [ ] https://github.com/OFA-Sys/OFA

Not released

• [ ] ALIGN - https://arxiv.org/abs/2102.05918
  • [ ] https://github.com/kakaobrain/coyo-align
• [ ] NUWA
  • [ ] https://github.com/microsoft/NUWA
  • [ ] https://github.com/lucidrains/nuwa-pytorch
• [ ] CoCa - https://arxiv.org/abs/2205.01917

---

• [x] CLIP

  • [x] OpenAI/CLIP
  • [x] mlfoundations/open_clip

• [x] CLOOB

  • [x] crowsonkb/cloob-training
  • [ ] pretrained model from CLOOB authors: https://github.com/ml-jku/cloob/blob/master/src/notebooks/zeroshot.ipynb

• [x] SLIP

  • [x] facebookresearch/SLIP

• [ ] CLIP-JAX

• [ ] AudioCLIP

  • https://github.com/AndreyGuzhov/AudioCLIP
  • https://github.com/NotNANtoN/AudioCLIP

• [x] CLIPfa (farsi) - https://github.com/sajjjadayobi/CLIPfa

• [ ] CLIP pretrained on FOOD101 by PASSL? - https://github.com/PaddlePaddle/PASSL/blob/main/docs/Train_CLIP_model.md

• [x] SBERT Multilingual CLIP - https://www.sbert.net/docs/pretrained_models.html#image-text-models

References for more variants:

https://paperswithcode.com/paper/learning-transferable-visual-models-from

Potentially in scope, lower priority

• https://github.com/j-min/VL-T5
• https://github.com/ashkamath/mdetr
• https://github.com/sberbank-ai/ru-dolph
• https://github.com/drboog/Lafite
• https://github.com/pbaylies/Augmented_CLIP
• ALIGN (is this even public? or just big?)
  • https://ai.googleblog.com/2021/05/align-scaling-up-visual-and-vision.html
  • https://arxiv.org/abs/2102.05918
• ALBEF
  • https://arxiv.org/abs/2102.05918
  • https://github.com/salesforce/ALBEF
  • https://github.com/salesforce/ALBEF/blob/main/visualization.ipynb
• ArtEmis - Affective language for Visual Art
  • https://github.com/optas/artemis
• https://github.com/CompVis/net2net
• LiT (not sure if there are public pre-trained models) - https://arxiv.org/abs/2111.07991
  • https://colab.research.google.com/github/google-research/vision_transformer/blob/main/lit.ipynb
• NUWA (very unlikely they'll release a reptrained model)
  • https://github.com/microsoft/NUWA
  • https://github.com/lucidrains/nuwa-pytorch

Older stuff

• DAMSM - https://github.com/taoxugit/AttnGAN
• https://github.com/sidward14/Style-AttnGAN
• https://github.com/luogen1996/MCN
• ViLBERT - https://github.com/facebookresearch/vilbert-multi-task
• VL-BERT - https://github.com/jackroos/VL-BERT

VQA is sort of a generalization of vision language co-training... TBD.

• https://github.com/facebookresearch/Detic
• https://github.com/henghuiding/Vision-Language-Transformer

MAGMA could be another useful approach to promote multi-lingual support

https://github.com/Aleph-Alpha/magma

[dmarx]

https://github.com/navervision/KELIP

[apolinario]

LiT: they released some models last week https://github.com/google-research/vision_transformer#lit-models

Audio: besides AudioCLIP there's also wav2clip with a different approach: https://github.com/descriptinc/lyrebird-wav2clip

[dmarx]

https://github.com/allenai/reclip

[dmarx]

https://github.com/facebookresearch/Detic

[dmarx]

https://github.com/sallymmx/ActionCLIP

[dmarx]

https://github.com/ChenRocks/UNITER

[dmarx]

https://github.com/raoyongming/DenseCLIP

[dmarx]

https://github.com/ttlmh/Bridge-Prompt

[apolinario]

https://github.com/sonoisa/clip-japanese

[dmarx]

• https://mmf.sh/docs/notes/projects
  • older stuff, e.g. ViLBET
• https://github.com/facebookresearch/multimodal
  • Newer stuff, e.g. FLAVA
  • https://github.com/facebookresearch/multimodal/tree/main/examples/flava
  • helloooooooo.... https://github.com/facebookresearch/multimodal/blob/668eabf273369eaf83a49016853425357f475c56/torchmultimodal/models/flava.py#L53
  • https://huggingface.co/aps/flava_full_pretrained_encoders_torchmm/tree/main

[dmarx]

SLIP demo install using dapm and loading weights using old strategy from DD: https://github.com/alembics/disco-diffusion/commit/c509aa1b9c00b2323a1fd95c5b0fc667bb12be4c

!wget https://dl.fbaipublicfiles.com/slip/slip_base_100ep.pt
!pip install napm

import napm
url = 'https://github.com/facebookresearch/SLIP'
napm.pseudoinstall_git_repo(url, add_install_dir_to_path=True)

import torch
import napm
import SLIP
from SLIP.models import SLIP_VITB16, SLIP, SLIP_VITL16

sd = torch.load('slip_base_100ep.pt', map_location=torch.device('cpu') )
real_sd = {}
for k, v in sd['state_dict'].items():
  new_key = '.'.join(k.split('.')[1:]) # strips "module" prefix. sure, why not.
  #print(k, new_key) 
  real_sd[new_key] = v
del sd

SLIPB16model = SLIP_VITB16(ssl_mlp_dim=4096, ssl_emb_dim=256)
SLIPB16model.load_state_dict(real_sd)

[dmarx]

CLOOB demo using dapm

!pip install git+https://github.com/openai/CLIP
import napm

url = "https://github.com/crowsonkb/cloob-training"
napm.pseudoinstall_git_repo(url, package_name='cloob')

import cloob
from cloob.cloob_training import model_pt, pretrained

config = pretrained.get_config('cloob_laion_400m_vit_b_16_16_epochs')
model = model_pt.get_pt_model(config)
checkpoint = pretrained.download_checkpoint(config)
model.load_state_dict(model_pt.get_pt_params(config, checkpoint), )
#model.eval().requires_grad_(False).to('cuda')

[dmarx]

CoCa https://arxiv.org/abs/2205.01917

[dmarx]

OTTER https://github.com/facebookresearch/OTTER

[dmarx]

ruDOLPH

• https://colab.research.google.com/drive/1gmTDA13u709OXiAeXWGm7sPixRhEJCga?usp=sharing
• https://github.com/ai-forever/ru-dolph

[dmarx]

https://socraticmodels.github.io/

[dmarx]

https://github.com/yxuansu/MAGIC

[apolinario]

https://github.com/rinnakk/japanese-clip (not the same as this)

[apolinario]

This is a very large and seemingly very good CLIP in Chinese that @Dango233 has shown me: https://wukong-dataset.github.io/wukong-dataset/benchmark.html

One problem though: It's pre-trained weights are on Mindspore (Huawei's PyTorch) so someone would need to convert that...

[Dango233]

https://github.com/mindspore-ai/models/tree/master/research/mm/wukong

[dmarx]

maybe just the fine tuned model?

https://github.com/j-min/clip-caption-reward

[apolinario]

A new (better, it seems) Multilingual CLIP https://github.com/FreddeFrallan/Multilingual-CLIP

[rom1504]

@apolinario indeed and now it's packaged properly on pypi as multilingual-clip

it's also available for easy testing at https://rom1504.github.io/clip-retrieval/?useMclip=true&query=%E9%BB%84%E8%89%B2%E3%81%84%E7%8C%AB&back=https%3A%2F%2Fknn5.laion.ai&index=laion5B

[dmarx]

@rom1504 @apolinario the m-clip release gave me a thought: maybe we could host mmc on pypi with essentially none of the other perceptors installed at all. Simple instructions for "finalizing" the mmc install could live in the README (as well as one-liners for specific perceptors PRN), and we could add a warning on import too. maybe we could ship an update script or a CLI command.

My thinking here is if we ship the core tooling as a bare library, then anyone could attach the mocking utilities upstream to quickly make new perceptors drop-in-able if they aren't already, which conversely would make them trivial to add to mmc (since they'd already be hooked into a conformant API one way or another).

Actually, it might be cleaner and simpler to isolate a simple mocking wrapper and package that for pypi?

I'm mostly just thinking out-loud now. Thoughts?

[apolinario]

I like the idea and spirit and I feel eventually if MMC gets way too many perceptors making some optional make a lot of sense. Now starting with all optional, I'm not sure - regardless I think your idea holds - just not sure if we ship empty or with some basics (OpenAI + OpenCLIP for e.g.) and let users further install from then on

[apolinario]

(New perceptor: https://github.com/microsoft/UniCL)

[apolinario]

https://github.com/goel-shashank/CyCLIP

[dmarx]

https://github.com/facebookresearch/omnivore

[dmarx]

https://github.com/microsoft/GLIP

[dmarx]

https://github.com/microsoft/RegionCLIP

[rom1504]

https://github.com/FacePerceiver/FaRL#use-farl-as-faceclip face clip

On Sat, Jun 25, 2022, 20:18 David Marx @.***> wrote:

https://github.com/microsoft/RegionCLIP

— Reply to this email directly, view it on GitHub https://github.com/dmarx/Multi-Modal-Comparators/issues/2#issuecomment-1166338042, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAR437SIUULWF4SL66ON72LVQ5EOVANCNFSM5S6LPHLA . You are receiving this because you were mentioned.Message ID: @.***>

[dmarx]

https://github.com/Lednik7/CLIP-ONNX/tree/main/clip_onnx

[dmarx]

https://github.com/OFA-Sys/OFA

[apolinario]

Turkish CLIP https://github.com/yusufani/TrCLIP

[dmarx]

https://github.com/salesforce/LAVIS

[dmarx]

EVA-CLIP - https://github.com/baaivision/EVA/blob/master/clip/README.md

basically already api compliant