search

dmarx / Multi-Modal-Comparators

Unified API to facilitate usage of pre-trained "perceptor" models, a la CLIP
39 stars 4 forks source link

Extend the Mock API capabilities #28

Closed apolinario closed 2 years ago

apolinario commented 2 years ago

This was useful to do for integrating MMC with Princess Generator. I know this makes the Mock API go beyond its scope, but:

  1. OpenAI's CLIP has this "tokenize" function. As we can't do it on the library level, as the tokenizer is different for every model - I think adding an option in the model itself might make sense
  2. We gotta think how much heavy lifting we want to do for the devs. From my perspective I think default operations such as tokenization, preprocessing, normalization should be abstracted to them

That said not sure if the way I did it here is the best to go about it - almost sure it's not

dmarx commented 2 years ago

here's a crazy idea: maybe we add a separate mechanism for wrapping the loader to mock the clip module (rather than the class). so like:

ldr = REGISTRY.find(id='my-model")
clip = MockClipModule(ldr)

tokenizer = clip.tokenize
model, preprocess_image = clip.load('my-model')
dmarx commented 2 years ago

check it out https://github.com/dmarx/Multi-Modal-Comparators/pull/31