Open ColinConwell opened 1 year ago
By way of example, here's a little script I worked out. If this looks incorrect, let me know!
import os, sys, torch
from PIL import Image
from torchvision import transforms
if not os.path.exists('DeCLIP'):
!git clone https://github.com/Sense-GVT/DeCLIP/
sys.path.append('DeCLIP')
sample_image = Image.open('dog.jpg')
from prototype.utils.misc import parse_config
config_path = 'DeCLIP/experiments/declip_experiments/declip88m/declip88m_r50_declip/config.yaml'
config = parse_config(config_path)
from prototype.model.declip import declip_res50
bpe_path = 'DeCLIP/prototype/text_info/bpe_simple_vocab_16e6.txt.gz'
config['model']['kwargs']['text_encode']['bpe_path'] = bpe_path
config['model']['kwargs']['clip']['text_mask_type'] = None
weights = torch.load('DeCLIP/weights/declip_88m/r50.pth.tar')['model']
weights = {k.replace('module.',''):v for k,v in weights.items()}
weights['logit_scale'] = weights['logit_scale'].unsqueeze(0)
model = declip_res50(**config['model']['kwargs'])
model.load_state_dict(weights, strict = False)
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
preprocess = transforms.Compose([transforms.Resize(256), transforms.ToTensor(), normalize])
inputs = preprocess(sample_image).unsqueeze(0)
model.visual(inputs)
Thank you for this exciting repository. Can you provide a simple example of how I might be able to load the models you provide in your model zoo?
Something along the lines of what is provided by the timm (pytorch-image-models) model repository:
Ideally, this would allow us to use the models in a jupyter notebook or other interactive context.
Thanks in advance!