Inference model-checkpoint generated from eval_linear_probe

[sakshamsingh1]

Hi, Thanks for the wonderful work and for sharing the code.

I have fine-tuned one of the provided pre-trained models(here) using the eval_linear_probe.py script. This saves a checkpoint with the params file.

I am getting Error(s) in loading state_dict for CLAP error while loading the fine-tuned model.

from open_clip import create_model

def get_model():
    param_path = <fine-tuned model saved param file>
    pretrained = <fine-tuned model saved checkpoint>

    precision = 'fp32'

    amodel = find_params_value(param_path, 'amodel')
    tmodel = find_params_value(param_path, 'tmodel')
    fusion_type = 'aff_2d'

    model, model_cfg = create_model(
        amodel,
        tmodel,
        pretrained,
        precision=precision,
        device=device,
        enable_fusion=enable_fusion,
        fusion_type=fusion_type
    )

    model.to(device)
    return model, model_cfg

How can I load this fine-tuned model?

Thanks

[RetroCirce]

Thank you for your post, now we are making our laion_clap as a pip library, with an ESC-50 evaluation tutorial (it should be released a week later).

Laion_clap is different from open_clip so you cannot use the open_clip library to use it. You can refer to our README.md. There is a tutorial on how to infer the embedding by our model.

Best, Ke

[RetroCirce]

Now we have updated our repo, release our pip library, and a tutorial code to evaluate it in the ESC-50 dataset. You can track our readme.md to use it.