Closed haraldschilly closed 7 months ago
For comparison, with an ssh tunnel, it works
$ curl -s http://localhost:11434/api/generate -d '{ "model" : "gemma" , "stream": false, "prompt" : "what comes after horse, car, and airplane?" } ' | jq .response
"The answer is rocket.\n\nHorse, car, and airplane are all vehicles that are used on land, while rocket is a vehicle that is used in space."
What fails is doing an actual query using a POST request:
I just looked at the code and can see exactly why this doesn't work. The code assumes that if the request is a POST request and the cookie is not set, then user is making a post request from the sign in page, so it just redirects them to the page they really wanted to go to. Thus this:
$ curl -k https://35.XX.YYY.ZZZ/ollama-api/generate?auth_token=[SECRET] -d '{ "model" : "gemma" , "stream": false, "prompt" : "what comes after fruit, apple, banana?" } '
Found. Redirecting to
does not proxy on the POST, and instead does a redirect. It even says that above in "Redirecting to".
I'll plan to do two things:
After figuring out that the code was still an old proxy and I managed to update it, now it looks like this. (i.e. there is no req.body
)
$ curl -k https://$IP/ollama-api/generate?auth_token=$TOKEN -d '{ "model" : "gemma", "stream": false, "prompt": "what comes after writing, reading, learning?" }'
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>TypeError: Cannot read properties of undefined (reading 'cocalcReturnTo_0.7010720816630611')<br> at handle (/opt/proxy/nvm/versions/node/v20.11.1/lib/node_modules/@cocalc/compute-server-proxy/dist/lib/auth.js:55:54)<br> at Layer.handle [as handle_request]
[...]
Ok, I got it to work after two changes:
sudo vim /opt/proxy/nvm/versions/node/v20.11.1/lib/node_modules/@cocalc/compute-server-proxy/dist/lib/auth.js
and edited in line 55 and 72: req.body[...
→ req.body?.[COCALC_AUTH_RETURN_TO]
:tada: ... this now works on my local computer, where IP and TOKEN are just the two variables for the server
$ curl -s -k https://$IP/ollama-api/generate?auth_token=$TOKEN -d '{ "model" : "gemma", "stream": false, "prompt": "Explain iterators in python to me", "system": "be brief"}' | jq -r .response
Iterators are objects in Python that allow you to iterate over a sequence of items one item at a time. They are lazily evaluated, meaning that they don't store all the items in memory at once, but instead generate them on demand when you need them.
Here are some key concepts related to iterators:
**1. Iterables:**
- Iterables are objects that can be iterated over, such as lists, sets, dictionaries, and strings.
**2. Iterators:**
- Iterators are objects that can be used to iterate over an iterable.
[...]
There is "progress" regarding accessing the ollama api endpoint, but no success yet. At least, I don't know how to succeed.
Ordering changed, and I added
ollama-
because the webui also uses its own/api
.That endpoint should list all available models and their capabilities and indeed, I get the information. Great!
What fails is doing an actual query using a POST request:
and when I add
-L
to follow redirects, I end up with HTML asking for<p>Please enter the authentication token to proceed:</p>
I also tried setting that
AUTH_TOKEN=...
cookie directly viacurl
, but that just hangs and somehow crashes ollama (to get it working again, I'm restarting the ollama docker container)