How to correctly capture DockerException?

[rwuthric]

I am not sure to understand how to correctly capture the errors of some python-on-whales calls.

A specific example:

docker = DockerClient(compose_files=['docker-compose.yml'])
docker.compose.up(detach=True)

The docker-compose.yml contains an error (in this case an external network is missing). The output I get is this:

Error response from daemon: network factory-net not found
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/wuthrich/codes/DemoFactory/OpenFactory/venv/openfact/lib/python3.10/site-packages/python_on_whales/components/compose/cli_wrapper.py", line 1017, in up
    run(full_cmd, capture_stdout=quiet, capture_stderr=quiet)
  File "/home/wuthrich/codes/DemoFactory/OpenFactory/venv/openfact/lib/python3.10/site-packages/python_on_whales/utils.py", line 194, in run
    raise DockerException(
python_on_whales.exceptions.DockerException: The docker command executed was `/bin/docker --file docker-compose up --detach`.
It returned with code 1
The content of stdout can be found above the stacktrace (it wasn't captured).
The content of stderr can be found above the stacktrace (it wasn't captured).

The error I am interested in catching is Error response from daemon: network factory-net not found. How would I do this?

A try except clause doesn't seem to work here.

try:
    docker.compose.up(detach=True)
except DockerException as err:
    print(err)

which results in

The docker command executed was `/bin/docker --file docker-compose up --detach`.
It returned with code 1
The content of stdout can be found above the stacktrace (it wasn't captured).
The content of stderr can be found above the stacktrace (it wasn't captured).

The error message I am interested in can't be find in the raised DockerException as neither stdout nor stderr were captured (they both return None). I guess the message Error response from daemon: network factory-net not found should be either in stdout or stderr.

Do I have to configure somehow my DockerClient in order the stdout and stderr get captured? Or how do I access the error response from the Docker daemon?

[gabrieldemarmiesse]

Well, this is tricky, and definitly an area where Python-on-whales could be improved. To enable TTY by default, we must avoid capturing any output. This is because the docker client can detect if stdout and stderr are TTY and changes the output accordingly. You can disable TTY, but it's still not captured from the python side at the moment.

You could try contextlib.redirect_stdout and contextlib.redirect_stderr with tty=False, but i'm not even sure that would work.

The right solution would be to make the subprocess believe it's using a TTY while we grab the output, with something like https://docs.python.org/fr/3/library/pty.html for example, but that's a lot of work, and may not be a good fit for this simple codebase.

You can try to experiment with subprocess directly, without python-on-whales. If you manage to do what you want feel free to post it here, that might help us find a better long term solution for this package.

[rwuthric]

I see. We will experiment with subprocess directly.

As a first quick and dirty way to do it (which works for our use case, but doesn't address the issues you mentioned above):

import subprocess

p = subprocess.Popen("docker compose -f docker-compose.yml up -d",
                     shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
ret = p.communicate()
print('stderr:\n', ret[1].decode('utf8'))
print('stdout:\n', ret[0].decode("utf8"))
print('return code', p.returncode)

It captures indeed error messages as expected. However, as docker compose sends everything to stderr, when the docker-compose.yml has no error, all messages will still be in stderr. But using the return code of Docker that could be handled.

[gabrieldemarmiesse]

Indeed I wish they would return different error codes depending on the error type, then we wouldn't have to parse logs to see what kind of error it was. I raised an issue a while ago but it didn't get much traction.