keras-team / keras-hub

Pretrained model hub for Keras 3
Apache License 2.0
804 stars 243 forks source link

Add `CLIP` model #1955

Closed james77777778 closed 3 weeks ago

james77777778 commented 4 weeks ago

Related to #1752

Colab for demonstrating the prediction of the ported backbone: https://colab.research.google.com/drive/1MgrQ1jq8wcICfoSbxp075wfap2qYADGs?usp=sharing

Preset: openai/clip-vit-base-patch32 (should work for all CLIP models)

Inputs: Image Text
"a photo of a cat", "a photo of a dog"

Outputs (probability):

There are some questions about the upcoming task definition:

cc @divyashreepathihalli

james77777778 commented 3 weeks ago

Can you please add the presets and the conversion script? It could be in a follow up PR.

Sure. Will submit a PR soon. However, the questions remain: