Does Twinny support embeddings without Ollama?

[Erxl]

Despite setting llamacpp as the backend for embeddings, Twinny is still sending requests to Ollama instead of llamacpp.

[geekbozu]

I'm not sure whats going on with the above, however this seems to be a real issue still

I made a quick and dirty edit to post the url of the request to embeddings and its showing the ollama url even though my provider is set to use open-webui and such.

Any thoughts. Lots of progress happening :)

[rjmacarthy]

Hey, I just fixed it in v3.15.8 thanks for pointing it out.

[geekbozu]

Sadly still no dice for what I'm doing :( I do think it might be auto embedding tho? Not sure. The code seems to be calling this variant of fetchEmbeddings for some reason https://github.com/twinnydotdev/twinny/blob/main/src/extension/stream.ts#L96 Which seems to still be hard coded to the option path? Not sure what is going on here.

To spell out what I'm doing. I'm clicking the embed workspace documents button as show in the second screenshot above. Not necessarily using @workspace until I need it :)

Thanks for being on top of things :D

[rjmacarthy]

Hey @geekbozu Ollama updated they API since i released the version of embeddings from /v1/embeddings to /api/embed and also changed the payload and response, i was not aware of it. I just fixed in in the latest version. Thanks for pointing it out!

[geekbozu]

Sadly still having some issues here. Thanks for tackling this! Twinny has been awesome to use :)

https://github.com/user-attachments/assets/ec8ec318-6397-4eea-b6b5-e485f9ba220e I can't seem to select my embedding provider still? It just deselects....

And with the latest version. My settings as such. I'm still getting 404s in the log as it seems now be respecting the API path now but still not the server and port. I am using openwebui as a proxy here as my Server is hiding behind a WireGuard network ;) Additionally nothing is set in workspace settings for twinny, I explicitly made sure that was clear before testing this!

[rjmacarthy]

Hey, are you sure that /ollama/api/embeddings is right?

[geekbozu]

The openapi docs for openwebui claim it is that. (/ollama/docs) The issue I'm seeing has Todo with the host and port still being incorrect They are 0.0.0.0:11434 instead of the values I have in my provider or Settings file. Which are 192.168.1.2:3000

[rjmacarthy]

Have you updated latest version? It works for me to edit the provider.

[geekbozu]

I blame me for that one. A laptop reboot cleaned it up so gremlins clearly! BUT alas one more. The bearer token is not being set correctly now. :(

A local guess changed this line to use provider.apiKey which got the bearer token correct. The system is now returning a 422 error instead https://github.com/twinnydotdev/twinny/blob/7e0b3ddaf4eb7f6f90e7cb678c277328a49cf919/src/extension/embeddings.ts#L76

An example body its sending is just a file path

{
  "model": "all-minilm:latest",
  "input": "c:\\Users\\geekbozu\\Documents\\code\mycode.h",
  "stream": false,
  "options": {}
}

And the issue is that openwebui's ollama passthrough expects input prompt or body

{
  "detail": [
    {
      "type": "missing",
      "loc": [
        "body",
        "prompt"
      ],
      "msg": "Field required",
      "input": {
        "model": "all-minilm:latest",
        "input": "c:\\Users\\geekbozu\\Documents\\code\mycode.h",
        "stream": false,
        "options": {}
      }
    }
  ]
}

[rjmacarthy]

Hmm, you're right, I just released version 3.16.0 which should address your issue, I would also recommend that you visit the Ollama website and update to the latest version which expects input as the parameter for embeddings.

Hope that helps!

[geekbozu]

It does! Let me clarify something. I am using openwebui on top of Ollama. Ollama is NOT exposed on my network and only can be talked through via openwebui. And therefore is what Twinny is talking to. :) It seems that openwebui has the different endpoint syntax here on its ollama endpoint.

[rjmacarthy]

Yes, I am aware of how it works.

Thanks for pointing the issues out, much appreciated!

[geekbozu]

No thank you for being receptive! Twinny is awesome and I really enjoy using it. Keep up the great work with it and symmetry!

From my end this is now feature complete. The outstanding issues are NOT with Twinny. When pointing twinny at ollama directly via a provider it works as expected now 🎉

I'll work with OpenWebUI on this as its particular to my use case but OpenWebUI has NOT updated their API layer to support the new endpoint which currently is causing the incompatibility now. https://github.com/open-webui/open-webui/blob/4617f3a4e277a716ab75460934257813d8865663/backend/open_webui/apps/ollama/main.py#L542

Again awesome work :D

[unclemusclez]

I believe i am also experiencing this. I am trying to us e the Open WebUI embeddings but it automatically reverts to Local Ollama or nothing. for a few secs, then

[unclemusclez]

@geekbozu was this resolved? i've been trying to figure this one out