search

golemfactory / clay

Golem is creating a global market for computing power.
https://golem.network
GNU General Public License v3.0
2.91k stars 286 forks source link

"None of the Docker images are available", even though it was pulled before #5100

Closed etam closed 4 years ago

etam commented 4 years ago

Description

Golem Version: 6875c1e20c2f5b76b8e1b66774777b978fe70b39

OS: Linux

Branch: develop

Mainnet/Testnet: testnet

Description of the issue:

When golem starts:

2020-02-08 05:27:31 WARNING  golem.docker.manager                Docker: pulling image 'golemfactory/blender:1.13' 
2020-02-08 05:27:31 DEBUG    golem.docker.commands.docker        Docker command: ['docker', '-D', 'pull', 'golemfactory/blender:1.13'] 
2020-02-08 05:27:33 DEBUG    golem.docker.commands.docker        Docker command output: b'1.13: Pulling from golemfactory/blender\n7413c47ba209: Already exists\n0fe7e7cbb2e8: Already exists\n1d425c982345: Already exists\n344da5c95cec: Already exists\n2b8343116144: Already exists\n5a5dbc4f3ed6: Already exists\nfc30871f3bb3: Already exists\nac547fa334c7: Already exists\nac547fa334c7: Already exists\n50c8ba9f2d55: Already exists\n403533530ae8: Already exists\n0699bdd23371: Already exists\n761dfa91e3d5: Already exists\n3b7ef43927e2: Already exists\n15fde92ed668: Already exists\n386fb76ef31e: Already exists\nDigest: sha256:29f4071141b7ed353b9163b75a746dc302855c06d22848bedb6f17d8a756d465\nStatus: Image is up to date for golemfactory/blender:1.13\n'

but later task computation fails with

2020-02-08 05:30:06 DEBUG    golem.docker.image                  DockerImage NotFound 
Traceback (most recent call last):
  File "/home/buildbot-worker/worker/test_node_integration/build/.venv/lib/python3.6/site-packages/docker/api/client.py", line 229, in _raise_for_status
    response.raise_for_status()
  File "/home/buildbot-worker/worker/test_node_integration/build/.venv/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/golemfactory/blender:1.13/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/buildbot-worker/worker/test_node_integration/build/golem/docker/image.py", line 51, in is_available
    info = client.inspect_image(self.name)
  File "/home/buildbot-worker/worker/test_node_integration/build/.venv/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/buildbot-worker/worker/test_node_integration/build/.venv/lib/python3.6/site-packages/docker/api/image.py", line 245, in inspect_image
    self._get(self._url("/images/{0}/json", image)), True
  File "/home/buildbot-worker/worker/test_node_integration/build/.venv/lib/python3.6/site-packages/docker/api/client.py", line 235, in _result
    self._raise_for_status(response)
  File "/home/buildbot-worker/worker/test_node_integration/build/.venv/lib/python3.6/site-packages/docker/api/client.py", line 231, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/buildbot-worker/worker/test_node_integration/build/.venv/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("no such image: golemfactory/blender:1.13: No such image: golemfactory/blender:1.13")
2020-02-08 05:30:06 WARNING  golem.task.taskthread               Task computing error None of the Docker images are available

Steps To Reproduce

Run integration tests

Logs and any additional context

https://buildbot.golem.network/buildbot/#/builders/15/builds/970 This happened in those tests:

etam commented 4 years ago

OK, this is just misconfiguration of our test server (a cleanup script was to eager to remove docker images).