search

microsoft / TaskWeaver

A code-first agent framework for seamlessly planning and executing data analytics tasks.
https://microsoft.github.io/TaskWeaver/
MIT License
5.38k stars 689 forks source link

Getting Docker Daemon Issues when running an image, but works locally #285

Closed antoineross closed 7 months ago

antoineross commented 8 months ago

Describe the bug Running Taskweaver on a custom Docker Image seems to run into errors. I get the following error:

Failed to connect to Docker daemon: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')).

It runs normally when running locally and with docker on, but building an image and running it seems to run into the error above.

To Reproduce Steps to reproduce the behavior: Run the web application on a custom Docker Image. This is my Dockerfile, is there anything I need to add perhaps to make it work? 

FROM python:3.11-slim as python-base

# Install necessary packages including GDAL and g++ for Fiona
RUN apt-get update && \
    apt-get install -y git gcc g++ python3-dev gdal-bin libgdal-dev

# Allow statements and log messages to immediately appear in the Cloud Run logs
ENV PYTHONUNBUFFERED 1

# Create a non-root user with home directory
RUN useradd -m -u 1000 user

# Set user and environment variables
USER user
ENV HOME=/home/user \
    PATH=/home/user/.local/bin:$PATH \
    GDAL_CONFIG=/usr/bin/gdal-config

# Set the working directory in the container
WORKDIR $HOME/app

# Copy the requirements.txt file to the container
COPY requirements.txt $HOME/app/

# Install Python dependencies from requirements.txt
# Use --global-option to specify GDAL version if necessary
RUN pip install --user -r $HOME/app/requirements.txt

# Copy the application files, including app.py
COPY --chown=user:user . $HOME/app/

# Ensure user has write permission to the app directory
USER root
RUN chown -R user:user $HOME/app
USER user

# The application must listen on the port defined by the PORT environment variable.
# This is a requirement of Google Cloud Run.
ENV PORT 8080

# Specify the command to run your application
CMD ["chainlit", "run", "app.py", "--port", "8080"]

I have my app.py on the home directory so just need to clone repo, so to reproduce just move the app.py there, add the dockerfile, and then run docker build, and docker run with env variables on Project/taskweaver_config.json

antoineross commented 8 months ago

I'm adding the full terminal error here:

^Z2024-04-06 07:16:28 - Translated markdown file for en-US not found. Defaulting to chainlit.md.
2024-04-06 07:16:29 - Session 20240406-071629-d2f5a93e is initialized
2024-04-06 07:16:29 - Plugin embeddings loaded
2024-04-06 07:16:29 - CodeGenerator initialized successfully
2024-04-06 07:16:29 - Failed to connect to Docker daemon: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')). 
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/home/user/.local/lib/python3.11/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/usr/local/lib/python3.11/http/client.py", line 1298, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1058, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.11/http/client.py", line 996, in send
    self.connect()
  File "/home/user/.local/lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/home/user/.local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/home/user/.local/lib/python3.11/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/usr/local/lib/python3.11/http/client.py", line 1298, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1058, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.11/http/client.py", line 996, in send
    self.connect()
  File "/home/user/.local/lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.11/site-packages/docker/api/client.py", line 213, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/docker/utils/decorators.py", line 44, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/docker/api/client.py", line 236, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/app/taskweaver/ces/environment.py", line 143, in __init__
    self.docker_client = docker.from_env()
                         ^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/docker/client.py", line 94, in from_env
    return cls(
           ^^^^
  File "/home/user/.local/lib/python3.11/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/docker/api/client.py", line 220, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.11/site-packages/chainlit/utils.py", line 39, in wrapper
    return await user_function(**params_values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/app/app.py", line 399, in start
    app_session_dict[user_session_id] = app.get_session()
                                        ^^^^^^^^^^^^^^^^^
  File "/home/user/app/taskweaver/app/app.py", line 65, in get_session
    return self.session_manager.get_session(session_id, prev_round_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/app/taskweaver/app/session_manager.py", line 29, in get_session
    return self._get_session_from_store(session_id, True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/app/taskweaver/app/session_manager.py", line 87, in _get_session_from_store
    new_session = self.injector.create_object(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 998, in create_object
    self.call_with_injection(init, self_=instance, kwargs=additional_kwargs)
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 1040, in call_with_injection
    return callable(*full_args, **dependencies)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/app/taskweaver/session/session.py", line 102, in __init__
    role_instance = self.session_injector.create_object(role_entry.module, {"role_entry": role_entry})
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 998, in create_object
    self.call_with_injection(init, self_=instance, kwargs=additional_kwargs)
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 1031, in call_with_injection
    dependencies = self.args_to_inject(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 91, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 1079, in args_to_inject
    instance: Any = self.get(interface)
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 91, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 975, in get
    result = provider_instance.get(self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 264, in get
    return injector.create_object(self._cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 998, in create_object
    self.call_with_injection(init, self_=instance, kwargs=additional_kwargs)
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 1031, in call_with_injection
    dependencies = self.args_to_inject(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 91, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 1079, in args_to_inject
    instance: Any = self.get(interface)
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 91, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 975, in get
    result = provider_instance.get(self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 302, in get
    return injector.call_with_injection(self._callable)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/injector/__init__.py", line 1040, in call_with_injection
    return callable(*full_args, **dependencies)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/app/taskweaver/module/execution_service.py", line 39, in provide_executor_manager
    self.manager = code_execution_service_factory(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/app/taskweaver/ces/__init__.py", line 12, in code_execution_service_factory
    return SubProcessManager(
           ^^^^^^^^^^^^^^^^^^
  File "/home/user/app/taskweaver/ces/manager/sub_proc.py", line 74, in __init__
    self.env = Environment(
               ^^^^^^^^^^^^
  File "/home/user/app/taskweaver/ces/environment.py", line 145, in __init__
    raise docker.errors.DockerException(f"Failed to connect to Docker daemon: {e}. ")
docker.errors.DockerException: Failed to connect to Docker daemon: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')).
liqul commented 8 months ago

We have provided an all-in-one docker container to run TaskWeaver here. If you want to build your own, please take a look at this Dockerfile. From the error message, it suggests that the executor is running in docker mode. But if you put TaskWeaver inside a container, you need to switch it to local mode. Please read this for more on the execution mode.

antoineross commented 7 months ago

Hi thank you for pointing me to the Dockerfile. I was actually thinking of doing a Docker in Docker situation but i'm not sure how good of an idea that is. Anyway, I'll close this issue now. Thanks again.