chrismahoney
commented
1 day ago In your .env.local file, are you adding http://localhost:11434 as your Ollama base URL? If not, please duplicate .env.example to .env.local, add that entry, and try again.
Open marhensa opened 1 day ago
chrismahoney
commented
1 day ago In your .env.local file, are you adding http://localhost:11434 as your Ollama base URL? If not, please duplicate .env.example to .env.local, add that entry, and try again.
chrismahoney
commented
1 day ago I’m not sure at the moment (any may depend on your execution context) if you need to use 127.0.0.1 instead of localhost. Localhost has been working for me more or less from the start.
marhensa
commented
1 day ago I’m not sure at the moment (any may depend on your execution context) if you need to use 127.0.0.1 instead of localhost. Localhost has been working for me more or less from the start.
the problem is not localhost vs 127.0.0.1, I tried both and still got error.
even with 127.0.0.1 the Ollama model is listed in browser of this forked Bolt.new, as you can see in my screenshot. Model installed in Ollama is recognized by Bolt.
the problem is when conversation start, it immediately gives error.
chrismahoney
commented
1 day ago Oh, you're running from docker. You should have an .env.local file with the following specified for the Ollama base URL:
# You only need this environment variable set if you want to use oLLAMA models
# EXAMPLE http://localhost:11434
OLLAMA_API_BASE_URL=http://host.docker.internal:11434Your oTToDev instance is trying to locate Ollama at http://127.0.0.1:11434, which is not valid while you're running within a Docker container based on this output in the video you posted (thanks for the video!)
Please update .env.local, rebuild based on docker instructions and then re-run the container. Hopefully that takes care of it.
customize9292
commented
1 day ago I had the same problem. I just reinstalled it and everything worked.
chrismahoney
commented
1 day ago If the above host.docker.internal change fixes this for you, please respond so we can close this issue.
mmundy3832
commented
1 day ago I am also using docker, and I have the same issue as OP. Unfortunately, I was not able to get it running using the change to OLLAMA_API_BASE_URL. In fact, that doesn't make sense to me because, as OP mentioned, this command works
curl -D - http://127.0.0.1:11434/api/chat -d '{"model":"qwen2.5-coder-extra-ctx:7b","messages":[{"role":"user","content":"why is the sky blue?"}],"stream":false}
marhensa
commented
1 day ago Oh, you're running from docker. You should have an .env.local file with the following specified for the Ollama base URL:
Your oTToDev instance is trying to locate Ollama at http://127.0.0.1:11434, which is not valid while you're running within a Docker container based on this output in the video you posted (thanks for the video!)
Hi.. thank you to help me resolve this, but sadly it's still not working.
as I said in the first post, the oTToDev already see the model (so no matter it's 127.0.0.1, localhost, or host.docker.internal)
here the console of Ollama, it succeed to serve the /api/tags which basically listing the models (it's called from oTToDev when I open the browser).
it shows 200 which is sucessful, but still, whenever the conversation starts, it gives error.
so /api/tags is reachable but /api/chat somehow failed.
https://github.com/user-attachments/assets/53bea06c-3987-41e3-8a25-21571df3aec0
schilling3003
commented
1 day ago I am having a similar issue where the app sees the Ollama models, but the chat fails. What model is it trying to send the request to? For me, no matter what Ollama model I chose I always tries to send the request to Claude 3.5-Sonnet which makes it fail. Same problem when I try to use LMStudio. All local LLMs appear to be broken.
marhensa
commented
1 day ago I am having a similar issue where the app sees the Ollama models, but the chat fails. What model is it trying to send the request to? For me, no matter what Ollama model I chose I always tries to send the request to Claude 3.5-Sonnet which makes it fail. Same problem when I try to use LMStudio. All local LLMs appear to be broken.
All Ollama models can be listed in oTToDev, but they fail whenever a conversation starts, all of them.
radumalica
commented
20 hours ago same here, pulled the latest version and i have my OLLAMA_BASE_URL set properly, problem is when choosing Ollama, i can see my downloaded models, but for some reason when you hit enter to code something, the app si trying to get "claude-3-5-sonnet-latest" instead of the selected Ollama model:
at null.<anonymous> (file:///app/node_modules/.pnpm/@ai-sdk+provider-utils@1.0.20_zod@3.23.8/node_modules/@ai-sdk/provider-utils/src/response-handler.ts:72:16)
at async postToApi (file:///app/node_modules/.pnpm/@ai-sdk+provider-utils@1.0.20_zod@3.23.8/node_modules/@ai-sdk/provider-utils/src/post-to-api.ts:81:28)
at async OllamaChatLanguageModel.doStream (file:///app/node_modules/.pnpm/ollama-ai-provider@0.15.2_zod@3.23.8/node_modules/ollama-ai-provider/src/ollama-chat-language-model.ts:230:50)
at async fn (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/core/generate-text/stream-text.ts:345:23)
at null.<anonymous> (async file:///app/.wrangler/tmp/dev-Ih0o3c/functionsWorker-0.48321999920050174.js:30634:22)
at async _retryWithExponentialBackoff (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/util/retry-with-exponential-backoff.ts:37:12)
at async startStep (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/core/generate-text/stream-text.ts:310:13)
at async fn (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/core/generate-text/stream-text.ts:387:11)
at null.<anonymous> (async file:///app/.wrangler/tmp/dev-Ih0o3c/functionsWorker-0.48321999920050174.js:30634:22)
at async chatAction (file:///app/build/server/index.js:1038:20) {
cause: undefined,
url: 'http://ollama_url:11434/api/chat',
requestBodyValues: {
format: undefined,
model: 'claude-3-5-sonnet-latest',
options: { num_ctx: 32768, num_predict: 8000, temperature: 0 },
messages: [ [Object], [Object] ],
tools: undefined
},
statusCode: 404,
responseHeaders: {
'content-length': '78',
'content-type': 'application/json; charset=utf-8',
date: 'Wed, 13 Nov 2024 08:32:12 GMT'
},
responseBody: '{"error":"model \\"claude-3-5-sonnet-latest\\" not found, try pulling it first"}',
isRetryable: false,
data: undefined,
[Symbol(vercel.ai.error)]: true,
[Symbol(vercel.ai.error.AI_APICallError)]: true
}
[wrangler:inf] POST /api/chat 500 Internal Server Error (39ms)later edit with additional information:
In dev tools in browser (latest Chrome) the payload sent to the /api/chat:
{messages: [,…], apiKeys: {}}
apiKeys
:
{}
messages
:
[,…]
0
:
{role: "user", content: "[Model: qwen2.5-coder:7b]↵↵[Provider: Ollama]↵↵build a simple express API"}
content
:
"[Model: qwen2.5-coder:7b]\n\n[Provider: Ollama]\n\nbuild a simple express API"
role
:
"user"So the model and provider are properly sent to the app. This is picked up by Chat.client.tsx line 206 i think.
Also, from the error it seems that the provider is properly selected but not the model, and it defaults to DEFAULT_MODEL which is claude-3-5-sonnet-latest.
LATER EDIT 2:
Problem is in PR #188 which keeps the user choice and introduces regex match for provider and model. I added some console.log entries in stream-text.ts file and here is the output while choosing Ollama and code-qwen2.5 in the frontend:
more debug:
Model match result: null
Provider match result: null
Extracted Model in function: claude-3-5-sonnet-latest
Extracted Provider in function: Anthropic
Message content: [Model: qwen2.5-coder:7b]
[Provider: Ollama]
test
Model Match: null
Provider Match: null
Setting currentModel to: claude-3-5-sonnet-latest
[
'gpt-4o',
'anthropic/claude-3.5-sonnet',
'anthropic/claude-3-haiku',
'deepseek/deepseek-coder',
'google/gemini-flash-1.5',
'google/gemini-pro-1.5',
'x-ai/grok-beta',
'mistralai/mistral-nemo',
'qwen/qwen-110b-chat',
'cohere/command',
'gemini-1.5-flash-latest',
'gemini-1.5-pro-latest',
'llama-3.1-70b-versatile',
'llama-3.1-8b-instant',
'llama-3.2-11b-vision-preview',
'llama-3.2-3b-preview',
'llama-3.2-1b-preview',
'claude-3-5-sonnet-latest',
'claude-3-5-sonnet-20240620',
'claude-3-5-haiku-latest',
'claude-3-opus-latest',
'claude-3-sonnet-20240229',
'claude-3-haiku-20240307',
'gpt-4o-mini',
'gpt-4-turbo',
'gpt-4',
'gpt-3.5-turbo',
'grok-beta',
'deepseek-coder',
'deepseek-chat',
'open-mistral-7b',
'open-mixtral-8x7b',
'open-mixtral-8x22b',
'open-codestral-mamba',
'open-mistral-nemo',
'ministral-8b-latest',
'mistral-small-latest',
'codestral-latest',
'mistral-large-latest'
]REGEX is failing because the MODEL_LIST doesn't contain the OLLAMA models (i have 2 of them: qwen2.5-coder:7b and llama-3.1:latest) which are not in the list, hence the REGEX match failing and app defaulting to DEFAULT_MODEL and DEFAULT_PROVIDER.
radumalica
commented
17 hours ago workaround: if you are running in docker and you use an external URL for OLLAMA, just set in docker-compose.yaml 'RUNNING_IN_DOCKER=false' otherwise the app will silently use host.docker.internal URL in this function here:
const getOllamaBaseUrl = () => {
const defaultBaseUrl = import.meta.env.OLLAMA_API_BASE_URL || 'http://default_ollama_url:11434;
// Check if we're in the browser
if (typeof window !== 'undefined') {
// Frontend always uses localhost
return defaultBaseUrl;
}
// Backend: Check if we're running in Docker
const isDocker = process.env.RUNNING_IN_DOCKER === 'true';
return isDocker
? defaultBaseUrl.replace("localhost", "host.docker.internal")
: defaultBaseUrl;
};So if RUNNING_IN_DOCKER is true, the Base URL for Ollama in the BACKEND will be set to host.docker.internal and the model list pull will silently fail with no warning or errors.
In the FRONTEND however, it will be set to whatever you added in the .env.local file as OLLAMA_BASE_API_URL.
I don't know or found out where is that happening
This is a major inconsistency of the app. The BASE URL should be retrieved from .env.local to keep the application consistent, and only add instructions to the README file for users that need host.docker.internal as an additional dns entry in docker-compose file.
new4u
commented
15 hours ago RUNNING_IN_DOCKER=false
this is ture! i solved problem (running in docker and using openailike
mmundy3832
commented
8 hours ago RUNNING_IN_DOCKER=false
this is ture! i solved problem (running in docker and using openailike
Unfortunately, it did not solve my problem. I still have an exception, but mine appears just a little different:
Error: Network connection lost.
bolt-ai-1 | at async postToApi (file:///app/node_modules/.pnpm/@ai-sdk+provider-utils@1.0.20_zod@3.23.8/node_modules/@ai-sdk/provider-utils/src/post-to-api.ts:65:22)
bolt-ai-1 | at async OllamaChatLanguageModel.doStream (file:///app/node_modules/.pnpm/ollama-ai-provider@0.15.2_zod@3.23.8/node_modules/ollama-ai-provider/src/ollama-chat-language-model.ts:230:50)
bolt-ai-1 | at async fn (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/core/generate-text/stream-text.ts:345:23)
bolt-ai-1 | at null.<anonymous> (async file:///app/.wrangler/tmp/dev-bpgYTk/functionsWorker-0.16294152281104846.js:30634:22)
bolt-ai-1 | at async _retryWithExponentialBackoff (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/util/retry-with-exponential-backoff.ts:37:12)
bolt-ai-1 | at async startStep (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/core/generate-text/stream-text.ts:310:13)
bolt-ai-1 | at async fn (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/core/generate-text/stream-text.ts:387:11)
bolt-ai-1 | at null.<anonymous> (async file:///app/.wrangler/tmp/dev-bpgYTk/functionsWorker-0.16294152281104846.js:30634:22)
bolt-ai-1 | at async chatAction (file:///app/build/server/index.js:998:20)
bolt-ai-1 | at async Object.callRouteAction (file:///app/node_modules/.pnpm/@remix-run+server-runtime@2.10.2_typescript@5.5.2/node_modules/@remix-run/server-runtime/dist/data.js:37:16) {
bolt-ai-1 | retryable: true
bolt-ai-1 | }
[wrangler:inf] POST /api/chat 500 Internal Server Error (6ms) This error occurs with RUNNING_IN_DOCKER set to either true or false, and all combinations of OLLAMA_API_BASE_URL mentioned above
chrismahoney
commented
8 hours ago LATER EDIT 2:
Problem is in PR #188 which keeps the user choice and introduces regex match for provider and model. I added some console.log entries in stream-text.ts file and here is the output while choosing Ollama and code-qwen2.5 in the frontend
Thanks for the context, this sounds like an edge case based on remembering the last selected provider/model and that update of the fields registering as a model change. There were a couple of PRs in flight recently around this, making a note as I'm currently looking at env vars in general.
chrismahoney
commented
8 hours ago Note: To review this behavior with updates to providers in #251.
hardj300
commented
8 hours ago I am not sure if my issue is the same or slightly different. I have Ollama and OpenWebUI deployed on a single container and known to be working just fine. When I deploy your Bolt fork into the same docker instance I am unable to see any models in the second drop down. I am able to curl the ollama api url from the bolt container just fine and am pretty certain the ENV variable was passed succesfully. Feel free to help me check.
here is what is looks like when I run the compose up command.
Describe the bug
I have Ollama installed in Windows 11 24H2, default port 11434.
I install bolt.new-any-llm on WSL2 Debian 12 (mirrored network mode)
I can access and use Ollama just fine from WSL2 Debian bash terminal
I can even try to use it to ask "why is the sky blue?" question from WSL2 Debian bash using
curl, so the network / connection is not a problem.I set the
.env.localfor Ollama base URL withhttp://127.0.0.1:11434I run the container with that
.env.localfile the command isdocker compose --profile development --env-file .env.local upthe container runs fine, I opened up browser the Bolt can see the Ollama LLM lists:
but when the conversation starts, it gives error RIGHT AWAY.
in bash console there error accessing Ollama:
it cannot access
http://127.0.0.1:11434/api/chat? or what?is it POST vs GET thingy that Ollama have problem with?
Link to the Bolt URL that caused the error
http://localhost:5173
Steps to reproduce
127.0.0.1:11434) inside Linux WSL2..env.localdocker compose --profile development --env-file .env.local upExpected behavior
Error.
Screen Recording / Screenshot
https://github.com/user-attachments/assets/a3f6d9f8-6f43-4c56-a9c8-40d550f3e6bb
Platform
Additional context
No response