gsc build fails for any distro workload on a Ubuntu 24.04 host system

anjalirai-intel commented 1 month ago

Description of the problem

gsc build fails for any distro workload on a Ubuntu 24.04 host system

As per gsc requirement pip3 install docker jinja2 tomli tomli-w pyyaml when we try to install these libraries, it fails with below errors.

pip3 install docker jinja2 tomli tomli-w pyyaml
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Ubuntu adopted PEP 668 from 23.04 , which dictates that pip can no longer install packages managed by the distro's general-purpose package manager, hence we use apt-get. Above listed requirement are installed via apt, versions are mentioned below

docker: 5.0.3 jinja2 : 3.1.2 tomli : 2.0.1 tomli-w : 1.0.0 pyyaml : 6.0.1

Steps to reproduce

https://github.com/gramineproject/gsc/tree/master/test#building-sample-images

Expected results

./gsc build -nc --insecure-args bash-test test/ubuntu20.04-bash.manifest
Building unsigned graminized Docker image `gsc-bash-test-unsigned` from original application image `bash-test`...
Warning: Duplicate key `loader.env.PATH`. Concatenating values from `<merged test/ubuntu20.04-bash.manifest and ubuntu/entrypoint.manifest.template>` and `<bash-test image env>`.
Step 1/29 : FROM ubuntu:20.04 AS gramine

 ---> 2abc4dfd8318
Step 2/29 : RUN env DEBIAN_FRONTEND=noninteractive apt-get update     && env DEBIAN_FRONTEND=noninteractive apt-get install -y         autoconf         bison         build-essential         coreutils         curl         gawk         git         libprotobuf-c-dev         linux-headers-generic         nasm         ninja-build         pkg-config         protobuf-c-compiler         protobuf-compiler         python3         python3-cryptography         python3-protobuf         wget            python3-pip            && /usr/bin/python3 -B -m pip install 'tomli>=1.1.0' 'tomli-w>=0.4.0' 'meson>=0.56,!=1.2.*'

 ---> Running in 6b1a9fa7af23
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3608 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3669 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1205 kB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [29.8 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [4143 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1502 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [32.5 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3758 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [55.2 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [28.6 kB]
Fetched 31.5 MB in 6s (5035 kB/s)
Reading package lists...

Reading package lists...

Building dependency tree...

Reading state information...
coreutils is already the newest version (8.30-3ubuntu2).
The following additional packages will be installed:
  automake autotools-dev binutils binutils-common binutils-x86-64-linux-gnu
  ca-certificates cpp cpp-9 dirmngr dpkg-dev fakeroot file g++ g++-9 gcc gcc-9
  .....
  .....
  .....
  ....
  ...
  Step 29/29 : ENTRYPOINT ["/bin/bash", "/gramine/app_files/apploader.sh"]

 ---> Running in 93e57c7507b4
 ---> d734a23fb3fc
Successfully built d734a23fb3fc
Successfully tagged gsc-bash-test-unsigned:latest
Successfully built an unsigned graminized Docker image `gsc-bash-test-unsigned` from original application image `bash-test`.

Actual results

Error:

./gsc build -nc --insecure-args bash-test test/ubuntu20.04-bash.manifest
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 497, in _make_request
    conn.request(
TypeError: HTTPConnection.request() got an unexpected keyword argument 'chunked'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/intel/jenkins/workspace/local_ci_graphene_gsc/gsc_dev_repo/./gsc", line 12, in <module>
    sys.exit(main(sys.argv))
             ^^^^^^^^^^^^^^
  File "/home/intel/jenkins/workspace/local_ci_graphene_gsc/gsc_dev_repo/./gsc.py", line 680, in main
    return args.command(args)
           ^^^^^^^^^^^^^^^^^^
  File "/home/intel/jenkins/workspace/local_ci_graphene_gsc/gsc_dev_repo/./gsc.py", line 284, in gsc_build
    docker_socket = docker.from_env()
                    ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/client.py", line 96, in from_env
    return cls(
           ^^^^
  File "/usr/lib/python3/dist-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: HTTPConnection.request() got an unexpected keyword argument 'chunked'

marchuk-vlad commented 1 month ago

Same problem. Install requests version 2.31.0 until it will be fixed pip3 install requests==2.31.0 docker jinja2 tomli tomli-w pyyaml

anjalirai-intel commented 1 month ago

We can no longer install packages via pip

mkow commented 1 month ago

Does it work if you install these packages from apt?

anjalirai-intel commented 1 month ago

Does it work if you install these packages from apt?

All the packages are installed via apt

jkr0103 commented 1 month ago

Root cause of the issue:

GSC uses Docker SDK for python for building the docker images which internally makes use of requests and urllib3 for connections with docker service.

Ubuntu 24.04 installs python3-urllib3=2.0.7 and python3-requests=2.31.0 versions which are not compatible as described here. This compatibility issue is fixed in Docker SDK for python version 6.1.0 with PR 3116 but Ununtu 24.04 installs Docker SDK for python version 5.0.3 which doesn't have the fix in it, hence we see the issue.

jkr0103 commented 1 month ago

There are three approaches to fix this issue, @dimakuv @mkow, Please suggest which one suits better:

Force docker sdk for python installation with pip using command: pip3 install docker --break-system-packages which installs the python3-docker version >=6.1.0.

Build/install manually from source code:

git clone --branch 7.0.0 https://github.com/docker/docker-py.git
cd docker-py
python3 setup.py build
sudo python3 setup.py install

Test:

$ python3
Python 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker;
>>> client = docker.from_env()

Note: setup.py is not available after version 7.0.0 which break above commands (build/install). I created an issue here for the same.

Wait for the python3-docker package upgrade to >=6.1.0 until Gramine announses support for ubuntu 24.04. This would automatically fix the issue. I have created a bug for ubuntu here which is in confirmed state now.

dimakuv commented 1 month ago

Gramine and GSC currently don't claim official support for Ubuntu 24.04. So I think we can safely wait for option 3 -- this bug should be fixed in Ubuntu itself, and looks like there is no easy workaround for GSC at the moment.

So my suggestion is to sit and wait for ~2 months. If nothing happens in this time frame, then we should revisit this issue.

gramineproject / gsc