docker.errors.ImageNotFound on Unraid

[Pepeanuts]

Describe the bug Glances wont start working correctly

Expected behavior On logs i've these errors `

Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status response.raise_for_status() File "/usr/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/3a53aa3cbdb58ae1d11fdc3f845e70ef1c2d1d142b2c72c237fb3ddaf2d743fa/json

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/usr/lib/python3.10/site-packages/glances/main.py", line 19, in glances.main() File "/usr/lib/python3.10/site-packages/glances/init.py", line 179, in main start(config=core.get_config(), args=core.get_args()) File "/usr/lib/python3.10/site-packages/glances/init.py", line 108, in start mode = GlancesMode(config=config, args=args) File "/usr/lib/python3.10/site-packages/glances/webserver.py", line 31, in init self.stats.update() File "/usr/lib/python3.10/site-packages/glances/stats.py", line 216, in update self._plugins[p].update() File "/usr/lib/python3.10/site-packages/glances/plugins/glances_plugin.py", line 1114, in wrapper ret = fct(self, *args, kw) File "/usr/lib/python3.10/site-packages/glances/plugins/glances_plugin.py", line 1131, in wrapper ret = fct(*args, *kw) File "/usr/lib/python3.10/site-packages/glances/plugins/glances_docker.py", line 255, in update container_stats['Image'] = container.image.tags File "/usr/lib/python3.10/site-packages/docker/models/containers.py", line 40, in image return self.client.images.get(image_id.split(':')[1]) File "/usr/lib/python3.10/site-packages/docker/models/images.py", line 335, in get return self.prepare_model(self.client.api.inspect_image(name)) File "/usr/lib/python3.10/site-packages/docker/utils/decorators.py", line 19, in wrapped return f(self, resource_id, args, kwargs) File "/usr/lib/python3.10/site-packages/docker/api/image.py", line 251, in inspect_image return self._result( File "/usr/lib/python3.10/site-packages/docker/api/client.py", line 274, in _result self._raise_for_status(response) File "/usr/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status raise create_api_error_from_http_exception(e) from e File "/usr/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception raise cls(e, response=response, explanation=explanation) from e docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/3a53aa3cbdb58ae1d11fdc3f845e70ef1c2d1d142b2c72c237fb3ddaf2d743fa/json: Not Found ("no such image: 3a53aa3cbdb58ae1d11fdc3f845e70ef1c2d1d142b2c72c237fb3ddaf2d743fa: No such image: sha256:3a53aa3cbdb58ae1d11fdc3f845e70ef1c2d1d142b2c72c237fb3ddaf2d743fa")

`

Additional context Docker on Unraid. Delete and fresh install of container dont change anything

[Pepeanuts]

after new install : ok i dont know what happened

[RazCrimson]

after new install : ok i dont know what happened

The issue seems to be due to python docker client trying to look for image that is not found, I am not sure how that ended up happening tho.

Maybe you deleted some images while the containers were listed in glances?

[TadMSTR]

I had the same thing. However redoing the container did not resolve and now I cannot reproduce that error in the logs. Glances start but nothing loads and the log just show a black screen.

Edit: turns out it was resolved from doing a fresh install of the container. different reason for it not loading in the browser. at some point I changed the network on the container and didn't change it back.

[RazCrimson]

@TadMSTR

So is the issue still present or is it resolved now?

[TadMSTR]

@TadMSTR

So is the issue still present or is it resolved now?

Yea I am good now. I initially kept getting the same errors in the log so redid the container a few times. At some point I must have also changed the network setting. I had stopped getting those errors and the container was running. Once I finally caught the network setting and changed back to host it was all good.

[RazCrimson]

Cool, then could you please close the issue as there are no problems?

[Jackeldk]

seems i get the same issue as this, but reinstalling does not help.

reverting to 3.2.7 lets me run it but upgrading to 3.3.0 stops it with this error

system ( unraid )

[RazCrimson]

@Jackeldk Can you just try to restart glances once?

The issue seems to be due to python docker client trying to look for image that is not found, I am not sure how that ended up happening tho.

Maybe you deleted some images while the containers were listed in glances?

As mentioned above, it could be caused by deleting some docker images, while glances in active, but it should be fixed with a simple restart.

Could try that?

[Jackeldk]

@Jackeldk Can you just try to restart glances once?

The issue seems to be due to python docker client trying to look for image that is not found, I am not sure how that ended up happening tho. Maybe you deleted some images while the containers were listed in glances?

As mentioned above, it could be caused by deleting some docker images, while glances in active, but it should be fixed with a simple restart.

Could try that?

restarted 10 times or so, have reinstalled 15 times and still i cant go over 3.2.7 then i get this error.

[RazCrimson]

Weird, there is no change with the Docker API interaction when we compare 3.2.7 and and the current develop The only change is some additional sorting options, which should not cause any problems.

@Jackeldk What version of docker are you using?

Also, one difference is that docker python library version used in glances:3.2.7 is 5.0.3, where as in glances:3.3.1 it is 6.0.1. So one option would be to open a shell and reinstall docker with pip install docker==5.0.3 and try launching glances again. The command to exec inside would be:

docker exec -it <container name or id> /bin/sh
# Now we are inside the container
pip install docker==5.0.3
glances

Hopefully this makes it work :)

[Jackeldk]

im on unraid so dont think i can update like that

its says docker version is 20.10.21

[stephansmolek]

I have the same issue. I am also on Unraid. Docker Version is 20.10.21.

Seems to work with an older version (I tried alpine-3.2.6.3) I have definitely not deleted images after installing, but before.

pip install docker==5.0.3

in the container also did not fix the issue.

[RazCrimson]

@Jackeldk @stephansmolek

Please try the fix mentioned here: https://forums.unraid.net/topic/134016-fix-to-a-small-issue-with-updating-oci-docker-images/

Source: https://github.com/nicolargo/glances/discussions/2258#discussioncomment-4834506

[Jackeldk]

@Jackeldk @stephansmolek

Please try the fix mentioned here: https://forums.unraid.net/topic/134016-fix-to-a-small-issue-with-updating-oci-docker-images/

Source: #2258 (comment)

mine already looks like this.

[stephansmolek]

@Jackeldk @stephansmolek

Please try the fix mentioned here: https://forums.unraid.net/topic/134016-fix-to-a-small-issue-with-updating-oci-docker-images/

Source: #2258 (comment)

Sadly this does not resolve the issue.

[RazCrimson]

@stephansmolek @Jackeldk

Can you guys check if the following line executes without issues on a Unraid Terminal?

docker run -it -v /var/run/docker.sock:/var/run/docker.sock:ro python /bin/bash
python -m pip install docker==5.0.3
python3 -c 'import docker; [print(c.image.tags) for c in docker.from_env().containers.list()]'

The above lines start a docker container with the docker socket bound to the container, then downloads docker's python lib and then tries to list the image tags of the containers

[stephansmolek]

@stephansmolek @Jackeldk

Can you guys check if the following line executes without issues on a Unraid Terminal?

docker run -it -v /var/run/docker.sock:/var/run/docker.sock:ro python /bin/bash
python -m pip install docker==5.0.3
python3 -c 'import docker; [print(c.image.tags) for c in docker.from_env().containers.list()]'

The above lines start a docker container with the docker socket bound to the container, then downloads docker's python lib and then tries to list the image tags of the containers

Sure, here is the complete output:

# docker run -it -v /var/run/docker.sock:/var/run/docker.sock:ro python /bin/bash
Unable to find image 'python:latest' locally
latest: Pulling from library/python
32fb02163b6b: Already exists 
167c7feebee8: Pull complete 
d6dfff1f6f3d: Pull complete 
e9cdcd4942eb: Pull complete 
ca3bce705f6c: Pull complete 
5e1c6c4f8bbf: Pull complete 
2da42ff3382c: Pull complete 
86f9457966ab: Pull complete 
896264e2a03c: Pull complete 
Digest: sha256:0d753a7365274cef746b34dde9b4aaa27644f64e1567ed8f40ccd191ac4bd530
Status: Downloaded newer image for python:latest
root@0a479a9866c5:/# python -m pip install docker==5.0.3
Collecting docker==5.0.3
  Downloading docker-5.0.3-py2.py3-none-any.whl (146 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 146.2/146.2 kB 1.9 MB/s eta 0:00:00
Collecting websocket-client>=0.32.0
  Downloading websocket_client-1.5.1-py3-none-any.whl (55 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.9/55.9 kB 2.1 MB/s eta 0:00:00
Collecting requests!=2.18.0,>=2.14.2
  Downloading requests-2.28.2-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 2.6 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (197 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 197.3/197.3 kB 3.3 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Downloading idna-3.4-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 1.9 MB/s eta 0:00:00
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.9/140.9 kB 4.1 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 3.3 MB/s eta 0:00:00
Installing collected packages: websocket-client, urllib3, idna, charset-normalizer, certifi, requests, docker
Successfully installed certifi-2022.12.7 charset-normalizer-3.1.0 docker-5.0.3 idna-3.4 requests-2.28.2 urllib3-1.26.15 websocket-client-1.5.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.3.1 -> 23.0.1
[notice] To update, run: pip install --upgrade pip
root@0a479a9866c5:/# python3 -c 'import docker; [print(c.image.tags) for c in docker.from_env().containers.list()]'
['python:latest']
['binhex/arch-krusader:latest']
['binhex/arch-qbittorrentvpn:latest']
['nicolargo/glances:alpine-3.2.6.3']
['x1unix/go-playground:latest']
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/0bd580990e4d77e0948f9115082baa27dc90bd46da486e57353e6a2429289fc5/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <listcomp>
  File "/usr/local/lib/python3.11/site-packages/docker/models/containers.py", line 40, in image
    return self.client.images.get(image_id.split(':')[1])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/models/images.py", line 314, in get
    return self.prepare_model(self.client.api.inspect_image(name))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/image.py", line 251, in inspect_image
    return self._result(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/0bd580990e4d77e0948f9115082baa27dc90bd46da486e57353e6a2429289fc5/json: Not Found ("no such image: 0bd580990e4d77e0948f9115082baa27dc90bd46da486e57353e6a2429289fc5: No such image: sha256:0bd580990e4d77e0948f9115082baa27dc90bd46da486e57353e6a2429289fc5")

[quincy5891]

Anyone find a fix. Mine I think does not work due to docker socket proxy since when i removed it, glances started working again.

[RazCrimson]

@stephansmolek @Jackeldk @quincy5891 The docker plugin is failing as the one of the calls to the docker socket is failing with a 404. Specifically the call to retrieve the container image name is failing for some container.

I believe the problem is either with the unraid's packaging or the particular version of docker used by unraid.

I could make a simple patch that would just ignore the error and continue, but then the image name for that container would not be present. The issue would probably fixed when newer versions for the docker engine is released on unraid.

Is that acceptable for you all?

[quincy5891]

That works for me.

[stephansmolek]

Sounds good to me.

[RazCrimson]

@stephansmolek @quincy5891

Please try out the changes on the latest develop branch.

If you are using the docker images, then use the images tagged with dev. They can be found here: https://hub.docker.com/r/nicolargo/glances/tags

[quincy5891]

Working for me, Thank you!

[DrSpaldo]

FYI, I just noticed my glances was not working either after updating Unraid to 6.11.5. I changed to the :dev tag and it is working again. Thanks

Edit/Update: it actually didn't work. The webpage worked great but the API was not interacting with Home Assistant. So I downgraded the version to nicolargo/glances:3.3.0.2 and it started working again. I will just leave it as an older version for now