search

huggingface / transformers.js

State-of-the-art Machine Learning for the web. Run 🤗 Transformers directly in your browser, with no need for a server!
https://huggingface.co/docs/transformers.js
Apache License 2.0
12.1k stars 766 forks source link

Add support for Salesforce BLIP in image-to-text #711

Open learyjk opened 7 months ago

learyjk commented 7 months ago

Model description

I’m new to this. Had fun experimenting with image-to-text today but found the gpt2 model was not great. Can we add support for the Salesforce BLIP model somehow?

I’ll look into exporting it to ONNX myself but I didn’t see it on the list for the second checkbox under prerequisites 🙃.

If I’m off base - any recommendations for how to improve the image-to-text pipeline?

Prerequisites

Additional information

No response

Your contribution

I can submit a PR but I might need a little guidance on the model training piece!

thank you!

jasonmayes commented 4 months ago

+1! I assumed it was available as listed in tasks but it seems to leave error in console when trying to use: Error: Unsupported model type: blip