search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
44.96k stars 2.18k forks source link

[BUG] failed to download ViT-B-32 #4666

Closed t3dium closed 10 months ago

t3dium commented 10 months ago

The bug

The VIT-B-32 model fails to download in immich, leading to encoding clips failing. Logs are shown below.

Immich's machine learning container:

Actions

    raw_response = await run_endpoint_function(

                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 163, in run_endpoint_function

    return await dependant.call(**values)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/src/app/main.py", line 75, in predict

    model = await load(await app.state.model_cache.get(model_name, model_type, **kwargs))

            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/src/app/main.py", line 101, in load

    await loop.run_in_executor(app.state.thread_pool, _load)

  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run

    result = self.fn(*self.args, **self.kwargs)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/src/app/main.py", line 94, in _load

    model.load()

  File "/usr/src/app/models/base.py", line 63, in load

    self.download()

  File "/usr/src/app/models/base.py", line 58, in download

    self._download()

  File "/usr/src/app/models/clip.py", line 43, in _download

    self._download_model(*models[0])

  File "/usr/src/app/models/clip.py", line 101, in _download_model

    download_model(

  File "/opt/venv/lib/python3.11/site-packages/clip_server/model/pretrained_models.py", line 239, in download_model

    raise RuntimeError(

RuntimeError: Failed to download https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276696365/onnx/ViT-B-32/textual.onnx within retry limit 3

[10/27/23 00:43:56] INFO     Downloading clip model 'ViT-B-32::openai'.This may 

                             take a while.                                      

Failed to download 

https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d536572

76696365/onnx/ViT-B-32/textual.onnx with <HTTPError 403: 'Forbidden'> at the 0th

attempt

Failed to download 

https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d536572

76696365/onnx/ViT-B-32/textual.onnx with <HTTPError 403: 'Forbidden'> at the 1th

attempt

Failed to download 

https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d536572

76696365/onnx/ViT-B-32/textual.onnx with <HTTPError 403: 'Forbidden'> at the 2th

attempt

  textual.onnx 0.0% • 0/100 bytes • ? • -:--:--

Exception in ASGI application

Traceback (most recent call last):

  File "/opt/venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi

    result = await app(  # type: ignore[func-returns-value]

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__

    return await self.app(scope, receive, send)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/venv/lib/python3.11/site-packages/fastapi/applications.py", line 276, in __call__

    await super().__call__(scope, receive, send)

  File "/opt/venv/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__

    await self.middleware_stack(scope, receive, send)

  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__

    raise exc

  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__

    await self.app(scope, receive, _send)

  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__

    raise exc

  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__

    await self.app(scope, receive, sender)

  File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__

    raise e

  File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__

    await self.app(scope, receive, send)

  File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__

    await route.handle(scope, receive, send)

  File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle

    await self.app(scope, receive, send)

  File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 66, in app

    response = await func(request)

               ^^^^^^^^^^^^^^^^^^^

  File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 237, in app

    raw_response = await run_endpoint_function(

                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 163, in run_endpoint_function

    return await dependant.call(**values)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/src/app/main.py", line 75, in predict

    model = await load(await app.state.model_cache.get(model_name, model_type, **kwargs))

            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/src/app/main.py", line 101, in load

    await loop.run_in_executor(app.state.thread_pool, _load)

  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run

    result = self.fn(*self.args, **self.kwargs)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/src/app/main.py", line 94, in _load

    model.load()

  File "/usr/src/app/models/base.py", line 63, in load

    self.download()

  File "/usr/src/app/models/base.py", line 58, in download

    self._download()

  File "/usr/src/app/models/clip.py", line 43, in _download

    self._download_model(*models[0])

  File "/usr/src/app/models/clip.py", line 101, in _download_model

    download_model(

  File "/opt/venv/lib/python3.11/site-packages/clip_server/model/pretrained_models.py", line 239, in download_model

    raise RuntimeError(

RuntimeError: Failed to download https://clip-as-service.s3.us-east-2.amazonaws.com/models-436c69702d61732d53657276696365/onnx/ViT-B-32/textual.onnx within retry limit 3

Also I think it would be best if there's a limit on how many times the download link is queried per a given time period, rather than infinitely trying every few seconds. Otherwise it's going to look malicious on amazon's end if this is occurring on every new immich user's device.

The OS that Immich Server is running on

Dietpi (debian based)

Version of Immich Server

v1.82.1

Version of Immich Mobile App

n/a

Platform with the issue

Your docker-compose.yml content

I followed the default config https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml

Your .env content

https://github.com/immich-app/immich/releases/latest/download/example.env

Reproduction steps

1. Started the immich stack
2. Uploaded photos
3. Tried searching for images

Additional information

No response

tdopierre commented 10 months ago

This is known issue, please see https://github.com/immich-app/immich/issues/4117#issuecomment-1772790612