[BUG]: (fetch failed on embed) HuggingFace/Xenova blocking download from user IP

[gabrie]

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

I'm using latest AnythingLLM Version 1.1.1 (1.1.1) on a M2 Mac Studio.

I tried to embed some PDF or TXT files with default LanceDB, fetch failed. Then I get Chroma from github installed and ran locally with below command, NOT using Docker. Which is suggested by Chroma github.

pip install chromadb
chroma run --path /Volumes/Thunder_1T/chromaDB/

Then I tested heartbeat, working. Then I tried to embed same docs again, still failed. Since I tried with LanceDB getting same error from AnythingLLM, I guess I should report the bug here instead of Chroma. If it's a bug on Chroma, please excuse me. Not a tech guy. Below finds running logs when error pops.

            (((((((((    (((((####
         ((((((((((((((((((((((#########
       ((((((((((((((((((((((((###########
     ((((((((((((((((((((((((((############
    (((((((((((((((((((((((((((#############
    (((((((((((((((((((((((((((#############
     (((((((((((((((((((((((((##############
     ((((((((((((((((((((((((##############
       (((((((((((((((((((((#############
         ((((((((((((((((##############
            (((((((((    #########

Running Chroma

Saving data to: /Volumes/Thunder_1T/chromaDB/ Connect to chroma at: http://localhost:8000 Getting started guide: https://docs.trychroma.com/getting-started

INFO: [27-02-2024 21:37:24] Set chroma_server_nofile to 65535 INFO: [27-02-2024 21:37:24] Anonymized telemetry enabled. See https://docs.trychroma.com/telemetry for more information. DEBUG: [27-02-2024 21:37:24] Starting component System DEBUG: [27-02-2024 21:37:24] Starting component OpenTelemetryClient DEBUG: [27-02-2024 21:37:24] Starting component SimpleAssignmentPolicy DEBUG: [27-02-2024 21:37:24] Starting component SqliteDB DEBUG: [27-02-2024 21:37:24] Starting component QuotaEnforcer DEBUG: [27-02-2024 21:37:24] Starting component Posthog DEBUG: [27-02-2024 21:37:24] Starting component LocalSegmentManager DEBUG: [27-02-2024 21:37:24] Starting component SegmentAPI INFO: [27-02-2024 21:37:24] Started server process [3055] INFO: [27-02-2024 21:37:24] Waiting for application startup. INFO: [27-02-2024 21:37:24] Application startup complete. INFO: [27-02-2024 21:37:24] Uvicorn running on http://localhost:8000 (Press CTRL+C to quit) INFO: [27-02-2024 21:37:48] ::1:54252 - "GET /api/v1/heartbeat HTTP/1.1" 200 ERROR: [27-02-2024 21:37:48] Collection test does not exist. Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/anyio/streams/memory.py", line 97, in receive return self.receive_nowait() ^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/anyio/streams/memory.py", line 92, in receive_nowait raise WouldBlock anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/middleware/base.py", line 159, in call_next message = await recv_stream.receive() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/anyio/streams/memory.py", line 112, in receive raise EndOfStream anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/chromadb/server/fastapi/init.py", line 82, in catch_exceptions_middleware return await call_next(request) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/middleware/base.py", line 165, in call_next raise app_exc File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/middleware/base.py", line 151, in coro await self.app(scope, receive_or_disconnect, send_no_error) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/middleware/base.py", line 189, in call with collapse_excgroups(): File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py", line 155, in exit self.gen.throw(value) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/_utils.py", line 93, in collapse_excgroups raise exc File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/middleware/base.py", line 191, in call response = await self.dispatch_func(request, call_next) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/chromadb/server/fastapi/init.py", line 96, in check_http_version_middleware return await call_next(request) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/middleware/base.py", line 165, in call_next raise app_exc File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/middleware/base.py", line 151, in coro await self.app(scope, receive_or_disconnect, send_no_error) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/routing.py", line 758, in call await self.middleware_stack(scope, receive, send) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/routing.py", line 778, in app await route.handle(scope, receive, send) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/routing.py", line 299, in handle await self.app(scope, receive, send) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/routing.py", line 79, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/routing.py", line 74, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/fastapi/routing.py", line 193, in run_endpoint_function return await run_in_threadpool(dependant.call, values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool return await anyio.to_thread.run_sync(func, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread return await future ^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/chromadb/telemetry/opentelemetry/init.py", line 127, in wrapper return f(*args, *kwargs) ^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/chromadb/auth/fastapi.py", line 242, in wrapped return f(args, kwargs) ^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/chromadb/server/fastapi/init.py", line 440, in get_collection return self._api.get_collection( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/chromadb/telemetry/opentelemetry/init.py", line 127, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/chromadb/api/segment.py", line 261, in get_collection raise ValueError(f"Collection {name} does not exist.") ValueError: Collection test does not exist. INFO: [27-02-2024 21:37:48] ::1:54254 - "GET /api/v1/collections/test?tenant=default_tenant&database=default_database HTTP/1.1" 500

Are there known steps to reproduce?

Launch Chroma locally Launch AnythingLLM Set Vector DB to Chroma Try to embed any docs in a workspace in AnythingLLM

[timothycarambat]

Oddly enough, this is just because we are querying the vectors for a collection which does not exist. This should not exit the Chroma process though. Will look into it. Could be a bug with Chroma

[gabrie]

Oddly enough, this is just because we are querying the vectors for a collection which does not exist. This should not exit the Chroma process though. Will look into it. Could be a bug with Chroma

Thanks a lot, it also appeared when I use LanceDB.

[timothycarambat]

Did you get the fetch error when using LanceDB? That is local so there is no fetching at all. Did you get a different message?

[gabrie]

I got same fetch failed error popup when using LanceDB. Which happened at "save and embed" step

](url)

[timothycarambat]

When you swapped your Vectordatabase you also clicked "save changes"? Can you screenshot that page as well? There are no external fetch requests when embedding documents with LanceDB to the DB. The only possible failure would be downloading the embedding model.

Can you additionally check ~/Library/Application Support/anythingllm-desktop/storage/models There should be a folder called Xenova with a subfolder called all-MiniLM-L6-v2.

If it does not exist, that is the fetch failure and could be permissions or firewall related.

[gabrie]

When you swapped your Vectordatabase you also clicked "save changes"? Can you screenshot that page as well? There are no external fetch requests when embedding documents with LanceDB to the DB. The only possible failure would be downloading the embedding model. Yes, saving VectorDB preferences was successful. Attached Screenshot

Can you additionally check ~/Library/Application Support/anythingllm-desktop/storage/models There should be a folder called Xenova with a subfolder called all-MiniLM-L6-v2.

If it does not exist, that is the fetch failure and could be permissions or firewall related.

Sorry, I found ~/Library/Application Support/anythingllm-desktop/storage/models folder, inside there is NO Xenova with a subfolder called all-MiniLM-L6-v2. I created them manually, tried again, still fetch failed. What should I do next?

[timothycarambat]

hm, TLDR; you need to get this repo closed into the models/Xenova/all-MiniLM-L6-v2 folder https://huggingface.co/Xenova/all-MiniLM-L6-v2/tree/main

In the past we have encountered this when HF is IP-blocking the user's IP either due to sanctions, abuse, VPN blocking or otherwise.

Looks like the primary way to avoid this is to pre-bundle the embedder with the app, which is not ideal. But that is the core issue. Alternatively, you can swap to another embedder like a local Ollama embedder, OpenAI, or otherwise.

That is the core issue though. Thank you for your help tracing this.

[gabrie]

hm, TLDR; you need to get this repo closed into the models/Xenova/all-MiniLM-L6-v2 folder https://huggingface.co/Xenova/all-MiniLM-L6-v2/tree/main

In the past we have encountered this when HF is IP-blocking the user's IP either due to sanctions, abuse, VPN blocking or otherwise.

Looks like the primary way to avoid this is to pre-bundle the embedder with the app, which is not ideal. But that is the core issue. Alternatively, you can swap to another embedder like a local Ollama embedder, OpenAI, or otherwise.

That is the core issue though. Thank you for your help tracing this.

Fixed, thanks! Altough embedding all-MiniLM-L6-v2 model into the app bundle is not that ideal. But to be honest, comparing to other solutions such like privateGPT which needs users to know install from github/homebrew etc, I do feel AnythingLLM is much more smoother for regular users. This is great for on-boarding, so maybe pre-bundle with the embedder is not a bad way. And all-MiniLM-L6-v2 is not so big at all.

[gabrie]

Thanks a lot for swift reaction and help!

[timothycarambat]

@gabrie If anything, we will at least host that model on our own CDN so that this critical piece is not missing. Really want to do everything we can to prevent bloating the app or adding models someone may not ever even use.

Closing as fixed. This issue has popped up some other places so I'm glad we got a fix. If you would like to comment how you downloaded and copy/pasted the folder from HF that would be appreciated as they do not have a "Download repo" button on their UI.

[gabrie]

Steps to fix 1/ Go HF 2/ Download everything manually

3/ Copy all downloaded file into ~/Library/Application Support/anythingllm-desktop/storage/models After you finished, the folder should looks like below

4/ Go AnythingLLM, try embed a PDF, it should work.

[iandol]

@timothycarambat -- thanks for the help! I am in China, and while my browser works as it uses a system proxy, probably ATLLM app ignores the system proxy (I know LM Studio also doesn't have proxy support and I download models manually there, while GPT4All does use the system proxy and connects to HF without issue).

I removed the ATLLM app, then reinstalled it (now V1.2.0) and manually downloaded the model - In ~/Library/Application Support I hope no typos:

Then restarted the app, but I get the same error. Starting from the console I see this error:

Adding new vectorized document into namespace brain
[INFO] Using Native Embedder
Chunks created from document: 147
[INFO] Using Native Embedder
[INFO] The native embedding model has never been run and will be downloaded right now. Subsequent runs will be faster. (~23MB)

Failed to load the native embedding model: TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11457:11)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at process.processImmediate (node:internal/timers:447:9)
    at async getModelFile (file:///Applications/AnythingLLM.app/Contents/Resources/backend/node_modules/@xenova/transformers/src/utils/hub.js:471:24)
    at async getModelJSON (file:///Applications/AnythingLLM.app/Contents/Resources/backend/node_modules/@xenova/transformers/src/utils/hub.js:575:18)
    at async Promise.all (index 0)
    at async loadTokenizer (file:///Applications/AnythingLLM.app/Contents/Resources/backend/node_modules/@xenova/transformers/src/tokenizers.js:61:18)
    at async AutoTokenizer.from_pretrained (file:///Applications/AnythingLLM.app/Contents/Resources/backend/node_modules/@xenova/transformers/src/tokenizers.js:4348:50)
    at async Promise.all (index 0) {
  cause: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (node:internal/deps/undici/undici:8422:28)
      at node:internal/deps/undici/undici:8380:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:8411:13)
      at process.processImmediate (node:internal/timers:476:21) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}
addDocumentToNamespace fetch failed
Failed to vectorize An integrative, multiscale view on neural theories of consciousness
[TELEMETRY SENT] {
  event: 'documents_embedded_in_workspace',
  distinctId: 'ec9b4015-bd21-4c0c-98dc-6c52323fa87a',
  properties: {
    LLMSelection: 'lmstudio',
    VectorDbSelection: 'lancedb',
    runtime: 'desktop'
  }
}
[Event Logged] - workspace_documents_added

For some reason it didn't use the manually downloaded model and still tries to download.

[taozhiyuai]

Steps to fix 1/ Go HF 2/ Download everything manually

3/ Copy all downloaded file into ~/Library/Application Support/anythingllm-desktop/storage/models After you finished, the folder should looks like below

4/ Go AnythingLLM, try embed a PDF, it should work.

same as @iandol, download and save there. still not working

[timothycarambat]

[iandol]

The ... in my previous ss was just from my file manager, the name itself is the same as on HF.

[timothycarambat]

🤔 , are the permissions on the file able to be accessed by the docker-container/user profile? they may be a different privilege level and that is why they are not found

[iandol]

Permissions were 644. macOS can sometimes have annoying security restrictions, so I added the desktop app to "Full disk access" and removed all download metadata. Restart the app and this is now working for me. Thanks @timothycarambat for your patience... Let's hope proxy support in a future release will make this more robust.

[CheatingOP]

@gabrie Didn't find ~/Library/Application Support/anythingllm-desktop/storage/models

[timothycarambat]

@CheatingOP there is a storage folder in that image. Open that, create models folder if it does not exist, then follow the thread above

[watsonzh]

I got same fetch failed error popup when using LanceDB. Which happened at "save and embed" step

](url) me to

[emulk]

Hi, i'm having the same error, but i'm running anythingllm as windows desktop application, and i'm using ollama with llama3.1

I don't get how to solve it, i have even trying on running ollama with a finetuned gguf model

{"level":"error","message":"Failed to vectorize fileDescription.txt","service":"backend"} {"level":"info","message":"\u001b[32m[TELEMETRY SENT]\u001b[0m {\"event\":\"documents_embedded_in_workspace\",\"distinctId\":\"88654d0c-d7af-453e-8ce9-27f77e7f4706\",\"properties\":{\"LLMSelection\":\"ollama\",\"Embedder\":\"native\",\"VectorDbSelection\":\"lancedb\",\"TTSSelection\":\"native\",\"runtime\":\"desktop\"}}","service":"backend"} {"level":"info","message":"\u001b[32m[Event Logged]\u001b[0m - workspace_documents_added","service":"backend"}

[timothycarambat]

@emulk

The second solution on this document will help resolve your issues since you noted you are on Windows: https://docs.anythingllm.com/fetch-failed-on-upload#windows-visual-c-redistributable