search

All-Hands-AI / OpenHands

πŸ™Œ OpenHands: Code Less, Make More
https://all-hands.dev
MIT License
37.61k stars 4.25k forks source link

[Bug]: Docker Build Failure During Runtime Initialization #5194

Closed young010101 closed 1 week ago

young010101 commented 1 week ago

Is there an existing issue for the same bug?

Describe the bug and reproduction steps

While attempting to run the application using DEBUG=1 make run, the runtime initialization fails during the Docker image build process. The following error is encountered:

Command '['docker', 'buildx', 'build', '--progress=plain', '--build-arg=OPENHANDS_RUNTIME_VERSION=0.14.0', '--build-arg=OPENHANDS_RUNTIME_BUILD_TIME=2024-11-22T10:07:42.239879', '--tag=ghcr.io/all-hands-ai/runtime:oh_v0.14.0_b8e5sgdbmjhb7iho_vcyllcgnnrejwgi3', '--load', '/tmp/tmpzkppehim']' returned non-zero exit status 1.

Subsequent log outputs indicate:

Full logs with debug mode enabled are attached below.

Steps to Reproduce:

Ensure Docker and its dependencies are installed and configured correctly. Run make run to start the application. Observe the errors during runtime initialization. Expected Behavior: The runtime should initialize successfully, and the application should run without issues.

Actual Behavior: The runtime initialization fails due to a Docker build error, preventing the application from running.

❯ head config.toml
[core]
workspace_base="./workspace"
# sandbox_base_container_image="docker.all-hands.dev/all-hands-ai/runtime"
# sandbox_base_container_image="custom-image-tmp"
sandbox_runtime_container_image="docker.all-hands.dev/all-hands-ai/runtime:0.14-nikolaik"

[llm]
model="xai/grok-beta"

❯ docker images
REPOSITORY                                    TAG                   IMAGE ID       CREATED         SIZE
docker.all-hands.dev/all-hands-ai/runtime     0.14-nikolaik         ad7b2d1122ec   2 days ago      5.55GB
ghcr.io/all-hands-ai/runtime                  0.14-nikolaik         ad7b2d1122ec   2 days ago      5.55GB
docker.all-hands.dev/all-hands-ai/openhands   0.14                  7a6f63ce1184   2 days ago      2.77GB
debian                                        latest                63ba939a68fd   11 days ago     117MB
nikolaik/python-nodejs                        python3.12-nodejs22   17e15ede0982   3 weeks ago     1.38GB
php                                           latest                303365c01725   4 weeks ago     535MB
ghcr.io/invariantlabs-ai/server               openhands             7933a0c6aa37   3 months ago    2.34GB
hello-world                                   latest                d2c94e258dcb   18 months ago   13.3kB
❯ DEBUG=1 make run

OpenHands Installation

Development workflow

OpenHands Version

0.14.2, commit: 68d1e76ccda11eb8e900a532795d1f35afd8cabe

Operating System

Linux

Logs, Errors, Screenshots, and Additional Context

❯ git last
commit 68d1e76ccda11eb8e900a532795d1f35afd8cabe (HEAD -> main, upstream/main)
Author: niliy01 <WannaTen@users.noreply.github.com>
Date:   Fri Nov 22 08:55:26 2024 +0800

    fix: remove repeated completion assignment in llm.py (#5167)

cyang in 🌐 onnes in OpenHands on ξ‚  main [?⇑] is πŸ“¦ v0.14.2 via 🐍 v3.12.7 via πŸ…’ openhands

cyang in 🌐 onnes in OpenHands on ξ‚  main [?⇑] is πŸ“¦ v0.14.2 via 🐍 v3.12.7 via πŸ…’ openhands
❯ DEBUG=1 make run
Running the app...
Starting backend server...
Waiting for the backend to start...
Backend started successfully.
Starting frontend...

> openhands-frontend@0.14.2 dev
> npm run make-i18n && cross-env VITE_MOCK_API=false remix vite:dev --port 3001 --host 127.0.0.1

> openhands-frontend@0.14.2 make-i18n
> node scripts/make-i18n-translations.cjs

DEBUG:openhands:DEBUG mode enabled.
10:07:12 - openhands:DEBUG: logger.py:265 - Logging initialized
10:07:12 - openhands:DEBUG: logger.py:277 - Logging to file in: /data/users/cyang/repos/OpenHands/logs
  ➜  Local:   http://127.0.0.1:3001/
  ➜  press h + enter to show help
2024-11-22 10:07 - openhands_aci:DEBUG - Logger initialized
10:07:15 - openhands:DEBUG: utils.py:135 - Attempt to load default LLM config from config toml
10:07:15 - openhands:DEBUG: utils.py:142 - Attempt to load group qwen from config toml as llm config
10:07:15 - openhands:DEBUG: utils.py:142 - Attempt to load group qwencoder from config toml as llm config
10:07:15 - openhands:DEBUG: utils.py:142 - Attempt to load group mistral from config toml as llm config
10:07:15 - openhands:DEBUG: utils.py:142 - Attempt to load group gpt-3 from config toml as llm config
10:07:15 - openhands:DEBUG: utils.py:142 - Attempt to load group llama from config toml as llm config
10:07:15 - openhands:DEBUG: utils.py:142 - Attempt to load group llama405 from config toml as llm config
10:07:15 - openhands:DEBUG: listen.py:151 - File upload config: max_size=0MB, restrict_types=False, allowed_extensions=['.*']
INFO:     Started server process [180028]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
ERROR:    [Errno 98] error while attempting to bind on address ('127.0.0.1', 3000): [errno 98] address already in use
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
10:07:32 - openhands:INFO: session.py:73 - WebSocket disconnected, sid: e6904920-0c58-4bc6-9b9a-60547e18cecf
INFO:     connection closed
INFO:     127.0.0.1:55812 - "GET /app HTTP/1.1" 200 OK
INFO:     127.0.0.1:55812 - "GET /locales/en/translation.json HTTP/1.1" 200 OK
INFO:     127.0.0.1:55812 - "GET /config.json HTTP/1.1" 200 OK
INFO:     ('127.0.0.1', 55828) - "WebSocket /ws" [accepted]
10:07:33 - openhands:INFO: listen.py:341 - New session: e6904920-0c58-4bc6-9b9a-60547e18cecf
INFO:     connection open
INFO:     127.0.0.1:55818 - "GET /api/options/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:55820 - "GET /api/options/agents HTTP/1.1" 200 OK
INFO:     127.0.0.1:55812 - "GET /api/options/security-analyzers HTTP/1.1" 200 OK
10:07:33 - openhands:INFO: codeact_agent.py:87 - Function calling not enabled for model xai/grok-beta. Mocking function calling via prompting.
10:07:33 - openhands:INFO: runtime_build.py:176 - Building image: ghcr.io/all-hands-ai/runtime:oh_v0.14.0_b8e5sgdbmjhb7iho_vcyllcgnnrejwgi3
================ DOCKER BUILD STARTED ================
10:07:50 - openhands:ERROR: docker.py:122 - Image build failed:
Command '['docker', 'buildx', 'build', '--progress=plain', '--build-arg=OPENHANDS_RUNTIME_VERSION=0.14.0', '--build-arg=OPENHANDS_RUNTIME_BUILD_TIME=2024-11-22T10:07:42.239879', '--tag=ghcr.io/all-hands-ai/runtime:oh_v0.14.0_b8e5sgdbmjhb7iho_vcyllcgnnrejwgi3', '--load', '/tmp/tmpzkppehim']' returned non-zero exit status 1.
10:07:50 - openhands:ERROR: docker.py:123 - Command output:

10:07:50 - openhands:ERROR: agent_session.py:198 - Runtime initialization failed: Command '['docker', 'buildx', 'build', '--progress=plain', '--build-arg=OPENHANDS_RUNTIME_VERSION=0.14.0', '--build-arg=OPENHANDS_RUNTIME_BUILD_TIME=2024-11-22T10:07:42.239879', '--tag=ghcr.io/all-hands-ai/runtime:oh_v0.14.0_b8e5sgdbmjhb7iho_vcyllcgnnrejwgi3', '--load', '/tmp/tmpzkppehim']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 275, in _raise_for_status
    response.raise_for_status()
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.45/containers/openhands-runtime-e6904920-0c58-4bc6-9b9a-60547e18cecf/json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/users/cyang/repos/OpenHands/openhands/runtime/impl/eventstream/eventstream_runtime.py", line 207, in connect
  File "/data/users/cyang/repos/OpenHands/openhands/utils/async_utils.py", line 18, in call_sync_from_async
    result = await coro
             ^^^^^^^^^^
  File "/data/users/cyang/anaconda3/envs/openhands/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cyang/repos/OpenHands/openhands/utils/async_utils.py", line 17, in <lambda>
    coro = loop.run_in_executor(None, lambda: fn(*args, **kwargs))
                                              ^^^^^^^^^^^^^^^^^^^
  File "/data/users/cyang/repos/OpenHands/openhands/runtime/impl/eventstream/eventstream_runtime.py", line 393, in _attach_to_container
    raise e

  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/models/containers.py", line 954, in get
    resp = self.client.api.inspect_container(container_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/api/container.py", line 793, in inspect_container
    return self._result(
           ^^^^^^^^^^^^^
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 281, in _result
    self._raise_for_status(response)
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 277, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.45/containers/openhands-runtime-e6904920-0c58-4bc6-9b9a-60547e18cecf/json: Not Found ("No such container: openhands-runtime-e6904920-0c58-4bc6-9b9a-60547e18cecf")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/users/cyang/repos/OpenHands/openhands/server/session/agent_session.py", line 196, in _create_runtime
    await self.runtime.connect()
  File "/data/users/cyang/repos/OpenHands/openhands/runtime/impl/eventstream/eventstream_runtime.py", line 221, in connect
    raise ValueError(

  File "/data/users/cyang/repos/OpenHands/openhands/runtime/utils/runtime_build.py", line 134, in build_runtime_image
    result = build_runtime_image_in_folder(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cyang/repos/OpenHands/openhands/runtime/utils/runtime_build.py", line 225, in build_runtime_image_in_folder
    _build_sandbox_image(
  File "/data/users/cyang/repos/OpenHands/openhands/runtime/utils/runtime_build.py", line 352, in _build_sandbox_image
    image_name = runtime_builder.build(
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cyang/repos/OpenHands/openhands/runtime/builder/docker.py", line 114, in build
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['docker', 'buildx', 'build', '--progress=plain', '--build-arg=OPENHANDS_RUNTIME_VERSION=0.14.0', '--build-arg=OPENHANDS_RUNTIME_BUILD_TIME=2024-11-22T10:07:42.239879', '--tag=ghcr.io/all-hands-ai/runtime:oh_v0.14.0_b8e5sgdbmjhb7iho_vcyllcgnnrejwgi3', '--load', '/tmp/tmpzkppehim']' returned non-zero exit status 1.
ERROR:asyncio:Future exception was never retrieved
future: <Future finished exception=CalledProcessError(1, ['docker', 'buildx', 'build', '--progress=plain', '--build-arg=OPENHANDS_RUNTIME_VERSION=0.14.0', '--build-arg=OPENHANDS_RUNTIME_BUILD_TIME=2024-11-22T10:07:42.239879', '--tag=ghcr.io/all-hands-ai/runtime:oh_v0.14.0_b8e5sgdbmjhb7iho_vcyllcgnnrejwgi3', '--load', '/tmp/tmpzkppehim'])>
Traceback (most recent call last):
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 275, in _raise_for_status
    response.raise_for_status()
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.45/containers/openhands-runtime-e6904920-0c58-4bc6-9b9a-60547e18cecf/json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/users/cyang/repos/OpenHands/openhands/runtime/impl/eventstream/eventstream_runtime.py", line 207, in connect
  File "/data/users/cyang/repos/OpenHands/openhands/utils/async_utils.py", line 18, in call_sync_from_async
    result = await coro
             ^^^^^^^^^^
  File "/data/users/cyang/anaconda3/envs/openhands/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cyang/repos/OpenHands/openhands/utils/async_utils.py", line 17, in <lambda>
    coro = loop.run_in_executor(None, lambda: fn(*args, **kwargs))
                                              ^^^^^^^^^^^^^^^^^^^
  File "/data/users/cyang/repos/OpenHands/openhands/runtime/impl/eventstream/eventstream_runtime.py", line 393, in _attach_to_container
    raise e

  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/models/containers.py", line 954, in get
    resp = self.client.api.inspect_container(container_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/api/container.py", line 793, in inspect_container
    return self._result(
           ^^^^^^^^^^^^^
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 281, in _result
    self._raise_for_status(response)
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/api/client.py", line 277, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cyang/.cache/pypoetry/virtualenvs/openhands-ai-LOhk372j-py3.12/lib/python3.12/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.45/containers/openhands-runtime-e6904920-0c58-4bc6-9b9a-60547e18cecf/json: Not Found ("No such container: openhands-runtime-e6904920-0c58-4bc6-9b9a-60547e18cecf")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/users/cyang/anaconda3/envs/openhands/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cyang/repos/OpenHands/openhands/server/session/agent_session.py", line 92, in _start_thread
    asyncio.run(self._start(*args), debug=True)
  File "/data/users/cyang/anaconda3/envs/openhands/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/data/users/cyang/anaconda3/envs/openhands/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cyang/anaconda3/envs/openhands/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/data/users/cyang/repos/OpenHands/openhands/server/session/agent_session.py", line 108, in _start
    await self._create_runtime(
  File "/data/users/cyang/repos/OpenHands/openhands/server/session/agent_session.py", line 196, in _create_runtime
    await self.runtime.connect()
  File "/data/users/cyang/repos/OpenHands/openhands/runtime/impl/eventstream/eventstream_runtime.py", line 221, in connect
    raise ValueError(

  File "/data/users/cyang/repos/OpenHands/openhands/runtime/utils/runtime_build.py", line 134, in build_runtime_image
    result = build_runtime_image_in_folder(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cyang/repos/OpenHands/openhands/runtime/utils/runtime_build.py", line 225, in build_runtime_image_in_folder
    _build_sandbox_image(
  File "/data/users/cyang/repos/OpenHands/openhands/runtime/utils/runtime_build.py", line 352, in _build_sandbox_image
    image_name = runtime_builder.build(
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/cyang/repos/OpenHands/openhands/runtime/builder/docker.py", line 114, in build
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['docker', 'buildx', 'build', '--progress=plain', '--build-arg=OPENHANDS_RUNTIME_VERSION=0.14.0', '--build-arg=OPENHANDS_RUNTIME_BUILD_TIME=2024-11-22T10:07:42.239879', '--tag=ghcr.io/all-hands-ai/runtime:oh_v0.14.0_b8e5sgdbmjhb7iho_vcyllcgnnrejwgi3', '--load', '/tmp/tmpzkppehim']' returned non-zero exit status 1.
young010101 commented 1 week ago

after kill <PID of port 3000> and make run again, it works.