drupol
commented
2 weeks ago Is hybrid search enabled ?
Closed luochen1990 closed 1 week ago
drupol
commented
2 weeks ago Is hybrid search enabled ?
luochen1990
commented
2 weeks ago Is hybrid search enabled ?
No, hybrid search is disabled, actually I have not use RAG for now.
drupol
commented
2 weeks ago That's weird... can you copy the dump in here?
luochen1990
commented
2 weeks ago That's weird... can you copy the dump in here?
The coredump file is about 160MB, and it's not easy to upload here :(
drupol
commented
2 weeks ago Just the logs, like I did in the OpenBlas issue for example ?
hauskens
commented
2 weeks ago I do not know if this is useful, but i am able to trigger a similar issue with Web Search using SearXNG.
journalctl -u open-webui.service -r
Jul 09 19:36:29 caretaker systemd[1]: open-webui.service: Consumed 8.047s CPU time, received 554.0K IP traffic, sent 93.8K IP traffic.
Jul 09 19:36:29 caretaker systemd[1]: open-webui.service: Failed with result 'core-dump'.
Jul 09 19:36:29 caretaker systemd[1]: open-webui.service: Main process exited, code=dumped, status=8/FPE
Jul 09 19:36:29 caretaker open-webui[68691]: [47B blob data]
Jul 09 19:36:26 caretaker open-webui[68691]: INFO:apps.rag.main:deleting existing collection 75320a4f111e32e16e0c19dd8440f5acc084d8b40884357664ad68c8c2ee0ad
Jul 09 19:36:26 caretaker open-webui[68691]: lc. Archived from the original on 4 March 2016. Retrieved 28 January 2016.\n\n^ a b c Watts, Steve (24 October 2013). "How Europe inspired Pokemon X and Y\'s creature designs". Shacknews. GameFly. Archived from the original on 19 July 201>
Jul 09 19:36:26 caretaker open-webui[68691]: INFO:apps.rag.main:store_docs_in_vector_db [Document(page_content='List of Pokémon - Wikipedia\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nJump to content\n\n\n\n\n\n\n\nMain menu\n\n\n\n\n\nMain menu\nmove to s>
Jul 09 19:36:26 caretaker open-webui[68691]: lc. Archived from the original on 4 March 2016. Retrieved 28 January 2016.\n\n^ a b c Watts, Steve (24 October 2013). "How Europe inspired Pokemon X and Y\'s creature designs". Shacknews. GameFly. Archived from the original on 19 July 201>
Jul 09 19:36:26 caretaker open-webui[68691]: INFO:apps.rag.main:store_data_in_vector_db [Document(page_content='List of Pokémon - Wikipedia\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nJump to content\n\n\n\n\n\n\n\nMain menu\n\n\n\n\n\nMain menu\nmove to s>
Jul 09 19:36:25 caretaker open-webui[68691]: DEBUG:urllib3.connectionpool:https://en.wikipedia.org:443 "GET /wiki/List_of_Pok%C3%A9mon HTTP/1.1" 200 384347
Jul 09 19:36:25 caretaker open-webui[68691]: DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): en.wikipedia.org:443
Jul 09 19:36:25 caretaker open-webui[68691]: DEBUG:urllib3.connectionpool:http://127.0.0.1:8123 "GET /search?q=how+many+pokemon+are+there%3F&format=json&pageno=1&safesearch=1&language=en-US&time_range=&categories=&theme=simple&image_proxy=0 HTTP/1.1" 200 28509
Jul 09 19:36:23 caretaker open-webui[68691]: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1:8123
Jul 09 19:36:23 caretaker open-webui[68691]: DEBUG:apps.rag.search.searxng:searching http://127.0.0.1:8123/searchjournalctl -u systemd-coredump@2-68764-0.service > log
systemd-coredump.log
Documents > Hybrid search off Documents > Embedding Model Engine > Default Web Search > Search Result Count = 1
In my case it seems to be related to the Documents > Embedding Model Engine.
Changing it to Ollama and selecing a model i managed to get it to work.
drupol
commented
1 week ago Can you please run coredumpctl and do coredumpctl info on the relevant pid?, then paste the dump here?
Look at the commands I run in here https://github.com/OpenMathLib/OpenBLAS/issues/4766#issuecomment-2196506212 and try to do the same?
hauskens
commented
1 week ago Here you go, i ran coredumpctl debug with gdb available (just did the same as you in that comment).
Hope this helps, let me know if there is anything else i can do.
drupol
commented
1 week ago Those issue seems to come from OpenBlas:
#0 0x00007f6ac2ff60b4 gemm_driver.isra.0 (libopenblas.so.0 + 0x2060b4)
#1 0x00007f6ac2ff6513 sgemm_thread_tn (libopenblas.so.0 + 0x206513)
#2 0x00007f6ac2f111d5 sgemm_ (libopenblas.so.0 + 0x1211d5)I'm pretty sure this is going to be useful for @martin-frbg, the OpenBlas maintainer.
@hauskens @luochen1990 : Can you please write down a list of steps to follow to reproduce the issue locally? I believe the Martin doesn't use Nix, so it might be useful for him to have a clear list of steps to follow in case he wants to test by himself.
hauskens
commented
1 week ago I can try to reproduce this without NixOS the module and provide instructions with nix shell later today
hauskens
commented
1 week ago Hi @martin-frbg
Here are some instructions on how to reproduce the problem, hope this helps. Let me know if you are stuck at any point.
@drupol @luochen1990 FYI
Just a few notes first:
INFO: 127.0.0.1:60434 - "GET /static/favicon.png HTTP/1.1" 200 OK
INFO: 127.0.0.1:60450 - "GET /user.png HTTP/1.1" 200 OK
INFO: 127.0.0.1:60412 - "GET /api/v1/chats/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:60426 - "GET /api/v1/users/user/settings HTTP/1.1" 200 OK
INFO: 127.0.0.1:60420 - "GET /ollama/api/version HTTP/1.1" 200 OK
INFO: 127.0.0.1:60474 - "POST /api/v1/chats/new HTTP/1.1" 200 OK
INFO: 127.0.0.1:60474 - "GET /api/v1/chats/ HTTP/1.1" 200 OK
INFO:apps.ollama.main:url: https://my-ollama-server
INFO: 127.0.0.1:60474 - "POST /ollama/api/chat HTTP/1.1" 200 OK
INFO: 127.0.0.1:60474 - "POST /api/chat/completed HTTP/1.1" 200 OK
INFO: 127.0.0.1:60474 - "POST /api/v1/chats/5e7ee6c4-4296-4898-b53b-6c6e92331f0e HTTP/1.1" 200 OK
INFO: 127.0.0.1:60474 - "GET /api/v1/chats/ HTTP/1.1" 200 OK
generate_title
llama3:latest
INFO:apps.ollama.main:url: https://my-ollama-server
INFO: 127.0.0.1:60474 - "POST /api/task/title/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:60474 - "POST /api/v1/chats/5e7ee6c4-4296-4898-b53b-6c6e92331f0e HTTP/1.1" 200 OK
INFO: 127.0.0.1:60474 - "GET /api/v1/chats/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:60474 - "GET /api/v1/chats/ HTTP/1.1" 200 OK
['rXDHIC6LGz6pISYIAAAH']
generate_search_query
INFO: 127.0.0.1:60474 - "POST /api/task/query/completions HTTP/1.1" 400 Bad Request
INFO:root:trying to web search with ('searxng', 'test')
INFO:apps.rag.main:store_data_in_vector_db [Document(page_content='Internet Speed Test | Fast.com\n\n\n\n\n\n\n\nPrivacy\n\n\nEnglish (US) \n\nBahasa ..............(rest of search body)4284c2b73c8bce4b46a6adf23e16d9e2ec8da4bb#open-webui which is v0.3.7With that said, this is how to reproduce the problem:
SEARXNG_QUERY_URL and OLLAMA_BASE_URLS to your own
mkdir open-webui-temp
cd open-webui-tempexport ANONYMIZED_TELEMETRY="false"
export DO_NOT_TRACK="true"
export ENABLE_RAG_WEB_SEARCH="true"
export SCARF_NO_ANALYTICS="true"
export HF_HOME="."
export STATIC_DIR="."
export DATA_DIR="."
export SENTENCE_TRANSFORMERS_HOME="."
export SEARXNG_QUERY_URL="https://my-searchx-server-here/search?q=
nix run github:nixos/nixpkgs/4284c2b73c8bce4b46a6adf23e16d9e2ec8da4bb#open-webui --experimental-features 'nix-command flakes' -- serve --host 0.0.0.0 --port 8321
- As long as the above environment variables is set i do not need to do any modifications in UI (tested on a clean running directory, i'm able to do normal chats with it)
- Select a model (i tested with llama3:latest, mistral:7b and dolphin-mistral:latest)
- Enable Web Search, type something. This causes the crash
martin-frbg
commented
1 week ago @hauskens thanks - so this seems to need a fair bit of infrastructure to reproduce, including setting up a local searx instance... this will probably take me a while to get to
drupol
commented
1 week ago I can work on making a flake file that setup everything for you. I'll work on it tomorrow.
luochen1990
commented
1 week ago setting up a local searx instance
use Google PSE will also works, which will be as easy as click mouses on the web page
drupol
commented
1 week ago If you want, you can use that project to get everything (ollama, open-webui, searxng, tika) setup with one single command: nix run git+https://github.com/drupol/wip-services-flake-llm. I share that project with some of my colleagues already.
If you want to tweak some env. variables, just clone the project and edit them in the file flake.nix, then run again nix run . inside the repository directory.
shivaraj-bh
commented
1 week ago An alternative solution:
When I use RAG_EMBEDDING_ENGINE="ollama" and RAG_EMBEDDING_MODEL="mxbai-embed-large:latest" the web search seems to work (as it skips the default sentence transformer usage for embedding):
Note: You need to
ollama pull mxbai-embed-large:latestfor the embedding to work
martin-frbg
commented
1 week ago @drupol thanks, I've got it working now with a clone of your project. had to find out where to enable json in searchxng, but now that it does web queries from Open Webui I find that I cannot reproduce the problem.
To all affected: what is your cpu please? if anything more recent than Haswell generation (i.e. something with AVX512 capability in addition to AVX/AVX2), can you please try if setting OPENBLAS_CORETYPE=HASWELL in your shell environment helps ?
drupol
commented
1 week ago @drupol thanks, I've got it working now with a clone of your project. had to find out where to enable json in searchxng, but now that it does web queries from Open Webui I find that I cannot reproduce the problem.
If you tell me what you changed in, I can update the project including your changes.
martin-frbg
commented
1 week ago Ahh, sorry - had to add "- json" after the "- html" in the searx/settings.yml file in the /nix/store tree (there were two instances of this file, in the site-packages for python-3.11 and 3.12, I changed them both to be on the safe side. I'm sure there is a better way but I'm grossly unfamiliar with both nix and webui/llama)
drupol
commented
1 week ago drupol
commented
1 week ago Good news, I just gave a new try with Open-WebUI 0.3.8 and everything seems to work fine now, for the first time.
martin-frbg
commented
1 week ago Great (I see you applied both changes, so it is a bit unclear if this is from setting coretype for OpenBLAS, or from some improvement in Open-Webui 0.3.8 ? On the other hand, the upstream report was against OpenBLAS built and run on a Haswell host)
luochen1990
commented
1 week ago using open-webui 0.3.8 in nixpkgs, and now the search capability is available and will not cause coredump. but sometimes there is error stacktrace printed:
post request handler error
Traceback (most recent call last):
File "/nix/store/hza544rqa6xrn0mmkrga9jq4bbdrxia1-python3.12-python-engineio-4.9.1/lib/python3.12/site-packages/engineio/async_server.py", line 306, in handle_request
await socket.handle_post_request(environ)
File "/nix/store/hza544rqa6xrn0mmkrga9jq4bbdrxia1-python3.12-python-engineio-4.9.1/lib/python3.12/site-packages/engineio/async_socket.py", line 109, in handle_post_request
p = payload.Payload(encoded_payload=body)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/hza544rqa6xrn0mmkrga9jq4bbdrxia1-python3.12-python-engineio-4.9.1/lib/python3.12/site-packages/engineio/payload.py", line 13, in __init__
self.decode(encoded_payload)
File "/nix/store/hza544rqa6xrn0mmkrga9jq4bbdrxia1-python3.12-python-engineio-4.9.1/lib/python3.12/site-packages/engineio/payload.py", line 44, in decode
raise ValueError('Too many packets in payload')
ValueError: Too many packets in payload
ERROR:engineio.server:post request handler error
Traceback (most recent call last):
File "/nix/store/hza544rqa6xrn0mmkrga9jq4bbdrxia1-python3.12-python-engineio-4.9.1/lib/python3.12/site-packages/engineio/async_server.py", line 306, in handle_request
await socket.handle_post_request(environ)
File "/nix/store/hza544rqa6xrn0mmkrga9jq4bbdrxia1-python3.12-python-engineio-4.9.1/lib/python3.12/site-packages/engineio/async_socket.py", line 109, in handle_post_request
p = payload.Payload(encoded_payload=body)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/hza544rqa6xrn0mmkrga9jq4bbdrxia1-python3.12-python-engineio-4.9.1/lib/python3.12/site-packages/engineio/payload.py", line 13, in __init__
self.decode(encoded_payload)
File "/nix/store/hza544rqa6xrn0mmkrga9jq4bbdrxia1-python3.12-python-engineio-4.9.1/lib/python3.12/site-packages/engineio/payload.py", line 44, in decode
raise ValueError('Too many packets in payload')Great (I see you applied both changes, so it is a bit unclear if this is from setting coretype for OpenBLAS, or from some improvement in Open-Webui 0.3.8 ? On the other hand, the upstream report was against OpenBLAS built and run on a Haswell host)
I will do more tests in the upcoming days but as far as I can tell, I'm unable to reproduce the issue with Open-WebUI 0.3.8, with or without setting the OpenBLAS coretype.
luochen1990
commented
1 week ago as far as I can tell, I'm unable to reproduce the issue with Open-WebUI 0.3.8
Given that self.decode() appears in the error stack, I suggest you attempt to reproduce it using some unicode-related queries. I still don't know the exact cause of the error, but I did use Chinese charactors in my query
martin-frbg
commented
1 week ago can search with German umlauts, Egyptian hieroglyphs or simply smilies in the query string here without getting crashes
luochen1990
commented
1 week ago luochen1990
commented
1 week ago Maybe we should close this issue now, this issue seems fixed in Open-WebUI 0.3.8, and the new issue is not related, and already reported to upstream (not taking enough attention but I did what I can)
Describe the bug
I'm using Open WebUI via NixOS, I configured Google PSE and tries to search something, but It core dumped after some seconds, from the journalctl output it seems the search result is received, but something went wrong when processing the search result. (Not sure I provided the correct Google PSE Engine ID, but I can not find a more possible correct one).
I have already reported this issue to upstream, but this might be a nixos specific issue since there is another related issue which is also using nixos, and other users which is not using nixos have not report similar issue yet.
Steps To Reproduce
see the issue from upstream
Expected behavior
No error occurs
Screenshots
see the issue from upstream
Additional context
Some fact worth to be noticed:
Notify maintainers
@shivaraj-bh @drupol
Metadata
Please run
nix-shell -p nix-info --run "nix-info -m"and paste the result.Add a :+1: reaction to issues you find important.