AirbusDefenceAndSpace / pesto

PESTO provides a set of tools to ease the process of packaging a Python algorithm as a processing web service into a docker image. The deployment of a web service becomes now as easy as filling few configuration files.
https://airbusdefenceandspace.github.io/pesto
Apache License 2.0
24 stars 8 forks source link

pesto --help needs to be root user #2

Closed rkanavath closed 3 years ago

rkanavath commented 3 years ago

Following the installation guide: https://airbusdefenceandspace.github.io/pesto/installation.html git clone https://github.com/AirbusDefenceAndSpace/pesto.git cd pesto && make install

Installation works but running pesto --help fails.

pesto --help
rkm@ubuntu18:~$ pesto --help
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/rkm/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/rkm/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rkm/.local/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/rkm/.local/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/rkm/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/rkm/.local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rkm/.local/bin/pesto", line 5, in <module>
    from pesto.cli.app import main
  File "/home/rkm/.local/lib/python3.8/site-packages/pesto/cli/app.py", line 9, in <module>
    from pesto.cli import build, init, list as list_builds, test
  File "/home/rkm/.local/lib/python3.8/site-packages/pesto/cli/test.py", line 5, in <module>
    from pesto.common.testing.test_runner import TestRunner
  File "/home/rkm/.local/lib/python3.8/site-packages/pesto/common/testing/test_runner.py", line 8, in <module>
    from pesto.common.testing.service_manager import ServiceManager
  File "/home/rkm/.local/lib/python3.8/site-packages/pesto/common/testing/service_manager.py", line 11, in <module>
    class ServiceManager:
  File "/home/rkm/.local/lib/python3.8/site-packages/pesto/common/testing/service_manager.py", line 12, in ServiceManager
    CLIENT = docker.from_env()
  File "/home/rkm/.local/lib/python3.8/site-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/home/rkm/.local/lib/python3.8/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/home/rkm/.local/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/home/rkm/.local/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

Running pesto as root works

root@ubuntu18:/home/rkm# ./.local/bin/pesto --help [2021-01-18 08:11:08,513] 2931-INFO app::display_banner():l21:



| _ | __/ _| / \ | \ _ () / | | |_ | |) | | _ \ | || | | | () | |) | '/ _ \ / / \/ / | | ' \ / ` | | | / ` |/ | / | '| | | | | /| |__ ) || || || | | /| | | () | (| /_ _ \ | | | | (| | | | (_| | (| || () | | | || | |_| |___|_/ || _/ () || || _/ \_||__//|| ||_, | || _,_|_|_\/|| _, | |/ |/ ----- ProcESsing facTOry : 1.0.0 ------------------------------------------------------------------------------------- usage: pesto [-h] {init,build,test} ...

optional arguments: -h, --help show this help message and exit

subcommands: {init,build,test} Valid subcommands root@ubuntu18:/home/rkm#

tapiab commented 3 years ago

The main issue here seems to be that your user is not in group docker, which cause this issue. But in some way, there is no reason to call this code just for printing help.

We will fix this in the next release

rkanavath commented 3 years ago

I am not running it as docker. Infact docker is not even installed on this machine. running groups command reveals this

$groups
rkm adm cdrom sudo dip plugdev lxd

If the code specifically checks for a group 'docker' that is a minor fix in documentation. My root user isn't in docker group either but it works if run as root from asudo -i shell. So I don't see how docker comes into the equation here. I am missing something here obviously.

there is no reason to call this code just for printing help.

Follow docs as is and report is what I do at first. Next I will try to run tests and report.

tapiab commented 3 years ago

The code does not check for group docker, but Docker is a dependency of PESTO (https://airbusdefenceandspace.github.io/pesto/installation.html#prerequisite). We can see in the message that it's the docker.from_env() which is causing the error. Using sudo seems to bypass something in the docker python package, but you'll be stuck later if docker is not installed on your machine. Can you try again with docker package installed?

rkanavath commented 3 years ago

correction: docker, docker-python package is already installed during make install. See reference to python files. I checked docker command line and docker-python package both exists. However I didn't do any docker related configuration. It maybe handled by make install and Ubuntu is also good and guessing what I must "have" instead of simply asking. So long story short, docker is installed.

We can see in the message that it's the docker.from_env() which is causing the error.

Each error comes from a different point in code. I agree last one is from docker.from_env()but there are many others.

File "/home/rkm/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect

Here is the reference to the file on local disk. I try to find the path to unix_socket variable in code and it is /var/run/docker.sock. This file is accessed in docker/transport/unixconn.py

root@ubuntu18:~# ll /var/run/docker.sock
srw-rw---- 1 root docker 0 Jan 19 14:31 /var/run/docker.sock=

The file can be accessed by root and users docker group which explains why it works withsudo -i

Documentation must say user must be root or in a docker group.

tapiab commented 3 years ago

Indeed, all the following errors comes from docker lib and not Pesto.

I suggest you use the docker package from docker instead the vanilla ubuntu one, it will spare you some pain. The install section in docker doc will guide you for docker group configuration.

I think we can state this is not an issue, but we'll take action to clarify this in next release.

rkanavath commented 3 years ago

I think we can state this is not an issue, but we'll take action to clarify this in next release.

That sounds fair. Closing