xenova / transformers.js

State-of-the-art Machine Learning for the web. Run 🤗 Transformers directly in your browser, with no need for a server!
https://huggingface.co/docs/transformers.js
Apache License 2.0
10.65k stars 648 forks source link

Support nomic-ai/nomic-embed-vision-v1.5 #848

Open markthethomas opened 1 month ago

markthethomas commented 1 month ago

Model description

hey there! Was looking to use nomic-ai/nomic-embed-vision-v1.5 since I'm using the text version so I could support image / text queries using the same semantic space, but getting this error when I tried:



    at OnnxruntimeSessionHandler (/Users/....redacted..../node_modules/onnxruntime-node/lib/backend.ts:16:30)
    at <anonymous> (/Users/....redacted....node_modules/onnxruntime-node/lib/backend.ts:61:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)```

### Prerequisites

- [X] The model is supported in Transformers (i.e., listed [here](https://huggingface.co/docs/transformers/index#supported-models-and-frameworks))
- [X] The model can be exported to ONNX with Optimum (i.e., listed [here](https://huggingface.co/docs/optimum/main/en/exporters/onnx/overview))

### Additional information

_No response_

### Your contribution

I took a swing at re-converting to onnx using the provided script, but ran into some errors. It seems like there's already an `onnx` directory in the model files area, too, so that may be unnecessary
markthethomas commented 1 month ago

btw, this is the code I am running:

    const { pipeline } = await import("@xenova/transformers");
    console.time("embedImage");
    const extractor = await pipeline(
        "image-feature-extraction",
        "nomic-ai/nomic-embed-vision-v1.5",
        { quantized: true}
    );
    const embeddings = await extractor(path);
    console.dir(embeddings);
    console.timeEnd("embedImage");
    return embeddings.tolist();