Docker image does not build

[mcatanzaro]

Yo. With EOL of Ubuntu 19.04, the Docker image no longer builds:

$ podman build --tag azafea .
STEP 1: FROM ubuntu:disco
Getting image source signatures
Copying blob 0a4ccbb24215 done
Copying blob 4dc9c2fff018 done
Copying blob 5ff1eaecba77 done
Copying blob c0f243bc6706 done
Copying config c88ac1f841 done
Writing manifest to image destination
Storing signatures
STEP 2: ENV LANG C.UTF-8
03253aa050eac02e2a013a9d976efeca759419a3687e9415bd0763e8510fad59
STEP 3: WORKDIR /opt/azafea/src
32bcd4c990adefe5f48b33dc0e34fdc8ac2c0827e3752f101131906771e1a320
STEP 4: COPY Pipfile.lock .
4fa7b5c1b7438a8a0e627e2ed5f2a141d85cda003eb1c2022ffa0a11ef2afc0f
STEP 5: ARG build_type
63956f5e5f602095f3fbb575998a2ad1a17836569126ab1688a704fd3d6ceaad
STEP 6: RUN apt --quiet --assume-yes update &&     apt --quiet --assume-yes --no-install-recommends install         gcc         gir1.2-glib-2.0         gobject-introspection         libcairo2-dev         libffi-dev         libgirepository-1.0-1         libgirepository1.0-dev         libglib2.0-dev         libpq5         libpq-dev         python3         python3-dev         python3-pip         python3-setuptools         python3-wheel         &&     pip3 install pipenv &&     pipenv install --ignore-pipfile &&     if [ "${build_type}" = "dev" ]; then         pipenv install --ignore-pipfile --dev     ; else         apt --quiet --assume-yes autoremove --purge             gcc             libcairo2-dev             libffi-dev             libgirepository1.0-dev             libglib2.0-dev             libpq-dev             python3-dev             &&         rm -rf /var/cache/{apt,debconf}                /var/lib/apt/lists/*                /var/log/{apt,dpkg.log}                ~/.cache     ; fi
Ign:1 http://security.ubuntu.com/ubuntu disco-security InRelease
Err:2 http://security.ubuntu.com/ubuntu disco-security Release
  404  Not Found [IP: 2001:67c:1562::15 80]
Ign:3 http://archive.ubuntu.com/ubuntu disco InRelease
Ign:4 http://archive.ubuntu.com/ubuntu disco-updates InRelease
Ign:5 http://archive.ubuntu.com/ubuntu disco-backports InRelease
Err:6 http://archive.ubuntu.com/ubuntu disco Release
  404  Not Found [IP: 2001:67c:1360:8001::24 80]
Err:7 http://archive.ubuntu.com/ubuntu disco-updates Release
  404  Not Found [IP: 2001:67c:1360:8001::24 80]
Err:8 http://archive.ubuntu.com/ubuntu disco-backports Release
  404  Not Found [IP: 2001:67c:1360:8001::24 80]
Reading package lists...
E: The repository 'http://security.ubuntu.com/ubuntu disco-security Release' does not have a Release file.
E: The repository 'http://archive.ubuntu.com/ubuntu disco Release' does not have a Release file.
E: The repository 'http://archive.ubuntu.com/ubuntu disco-updates Release' does not have a Release file.
E: The repository 'http://archive.ubuntu.com/ubuntu disco-backports Release' does not have a Release file.
Error: error building at STEP "RUN apt --quiet --assume-yes update &&     apt --quiet --assume-yes --no-install-recommends install         gcc         gir1.2-glib-2.0         gobject-introspection         libcairo2-dev         libffi-dev         libgirepository-1.0-1         libgirepository1.0-dev         libglib2.0-dev         libpq5         libpq-dev         python3         python3-dev         python3-pip         python3-setuptools         python3-wheel         &&     pip3 install pipenv &&     pipenv install --ignore-pipfile &&     if [ "${build_type}" = "dev" ]; then         pipenv install --ignore-pipfile --dev     ; else         apt --quiet --assume-yes autoremove --purge             gcc             libcairo2-dev             libffi-dev             libgirepository1.0-dev             libglib2.0-dev             libpq-dev             python3-dev             &&         rm -rf /var/cache/{apt,debconf}                /var/lib/apt/lists/*                /var/log/{apt,dpkg.log}                ~/.cache     ; fi": error while running runtime: exit status 100

It seems Ubuntu deletes its update repos once a release hits EOL? That's pretty aggressive. Anyway, I tried upgrading to 19.10:

diff --git a/Dockerfile b/Dockerfile
index 425b321..1df323b 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,4 @@
-FROM ubuntu:disco
+FROM ubuntu:eoan

 ENV LANG C.UTF-8

But that fails because it hits a file descriptor limit:

Collecting zipp>=0.5 (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv->pipenv)
  Downloading https://files.pythonhosted.org/packages/b2/34/bfcb43cc0ba81f527bc4f40ef41ba2ff4080e047acb0586b56b3d017ace4/zipp-3.1.0-py3-none-any.whl
Building wheels for collected packages: distlib
  Running setup.py bdist_wheel for distlib ... done
  Stored in directory: /root/.cache/pip/wheels/6e/e8/db/c73dae4867666e89ba3cfbc4b5c092446f0e584eda6f409cbb
Successfully built distlib
Installing collected packages: certifi, virtualenv-clone, appdirs, filelock, distlib, zipp, importlib-metadata, virtualenv, pipenv
Successfully installed appdirs-1.4.3 certifi-2020.4.5.1 distlib-0.3.0 filelock-3.0.12 importlib-metadata-1.6.0 pipenv-2018.11.26 virtualenv-20.0.18 virtualenv-clone-0.5.4 zipp-3.1.0
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/_internal/commands/install.py", line 430, in run
    wheel_cache.cleanup()
  File "/usr/lib/python3/dist-packages/pip/_internal/utils/temp_dir.py", line 58, in __exit__
    self.cleanup()
  File "/usr/lib/python3/dist-packages/pip/_internal/utils/temp_dir.py", line 81, in cleanup
    rmtree(self.path)
  File "/usr/share/python-wheels/retrying-1.3.3-py2.py3-none-any.whl/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/share/python-wheels/retrying-1.3.3-py2.py3-none-any.whl/retrying.py", line 212, in call
    raise attempt.get()
  File "/usr/share/python-wheels/retrying-1.3.3-py2.py3-none-any.whl/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/share/python-wheels/six-1.12.0-py2.py3-none-any.whl/six.py", line 693, in reraise
    raise value
  File "/usr/share/python-wheels/retrying-1.3.3-py2.py3-none-any.whl/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/lib/python3/dist-packages/pip/_internal/utils/misc.py", line 111, in rmtree
    onerror=rmtree_errorhandler)
  File "/usr/lib/python3.7/shutil.py", line 494, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.7/shutil.py", line 432, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.7/shutil.py", line 432, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.7/shutil.py", line 432, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  [Previous line repeated 2 more times]
  File "/usr/lib/python3.7/shutil.py", line 436, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/usr/lib/python3.7/shutil.py", line 434, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 24] Too many open files: '__pycache__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/base_command.py", line 143, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/_internal/commands/install.py", line 430, in run
    wheel_cache.cleanup()
  File "/usr/lib/python3/dist-packages/pip/_internal/req/req_tracker.py", line 32, in __exit__
    self.cleanup()
  File "/usr/lib/python3/dist-packages/pip/_internal/req/req_tracker.py", line 67, in cleanup
    self._temp_dir.cleanup()
  File "/usr/lib/python3/dist-packages/pip/_internal/utils/temp_dir.py", line 81, in cleanup
    rmtree(self.path)
  File "/usr/share/python-wheels/retrying-1.3.3-py2.py3-none-any.whl/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/share/python-wheels/retrying-1.3.3-py2.py3-none-any.whl/retrying.py", line 212, in call
    raise attempt.get()
  File "/usr/share/python-wheels/retrying-1.3.3-py2.py3-none-any.whl/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/share/python-wheels/six-1.12.0-py2.py3-none-any.whl/six.py", line 693, in reraise
    raise value
  File "/usr/share/python-wheels/retrying-1.3.3-py2.py3-none-any.whl/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/lib/python3/dist-packages/pip/_internal/utils/misc.py", line 111, in rmtree
    onerror=rmtree_errorhandler)
  File "/usr/lib/python3.7/shutil.py", line 485, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/usr/lib/python3/dist-packages/pip/_internal/utils/misc.py", line 119, in rmtree_errorhandler
    if os.stat(path).st_mode & stat.S_IREAD:
OSError: [Errno 24] Too many open files: '/tmp/pip-req-tracker-1urt61iw'
Error: error building at STEP "RUN apt --quiet --assume-yes update &&     apt --quiet --assume-yes --no-install-recommends install         gcc         gir1.2-glib-2.0         gobject-introspection         libcairo2-dev         libffi-dev         libgirepository-1.0-1         libgirepository1.0-dev         libglib2.0-dev         libpq5         libpq-dev         python3         python3-dev         python3-pip         python3-setuptools         python3-wheel         &&     pip3 install pipenv &&     pipenv install --ignore-pipfile &&     if [ "${build_type}" = "dev" ]; then         pipenv install --ignore-pipfile --dev     ; else         apt --quiet --assume-yes autoremove --purge             gcc             libcairo2-dev             libffi-dev             libgirepository1.0-dev             libglib2.0-dev             libpq-dev             python3-dev             &&         rm -rf /var/cache/{apt,debconf}                /var/lib/apt/lists/*                /var/log/{apt,dpkg.log}                ~/.cache     ; fi": error while running runtime: exit status 2

I tried increasing the soft fd limit:

diff --git a/Dockerfile b/Dockerfile
index 425b321..2b21912 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -7,6 +7,7 @@ WORKDIR /opt/azafea/src
 COPY Pipfile.lock .

 ARG build_type
+RUN ulimit -n 4096
 RUN apt --quiet --assume-yes update && \
     apt --quiet --assume-yes --no-install-recommends install \
         gcc 

That failed because both the hard and soft limits inside the container build are 1024. (Not sure why; my host machine has a higher hard limit, so podman must be lowering the limit itself.)

I tried increasing the hard limit as well:

diff --git a/Dockerfile b/Dockerfile
index 425b321..f47e2cf 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,4 @@
-FROM ubuntu:disco
+FROM ubuntu:eoan

 ENV LANG C.UTF-8

@@ -7,6 +7,9 @@ WORKDIR /opt/azafea/src
 COPY Pipfile.lock .

 ARG build_type
+
+RUN ulimit -H -n 4096
+RUN ulimit -S -n 4096
 RUN apt --quiet --assume-yes update && \
     apt --quiet --assume-yes --no-install-recommends install \
         gcc \

That failed:

STEP 6: RUN ulimit -H -n 4096
/bin/sh: 1: ulimit: error setting limit (Operation not permitted)
Error: error building at STEP "RUN ulimit -H -n 4096": error while running runtime: exit status 2

Not quite sure why I have permission to use apt but not ulimit. I don't know much about containers.

I tried upgrading to 20.04, since that seems like a good environment to target for the next couple years:

diff --git a/Dockerfile b/Dockerfile
index 425b321..1df323b 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,4 @@
-FROM ubuntu:disco
+FROM ubuntu:focal

 ENV LANG C.UTF-8

That failed due to dpkg repeatedly crashing. The focal image seems to be in real bad shape.

I gave up on Ubuntu and switched to Debian:

diff --git a/Dockerfile b/Dockerfile
index 425b321..1df323b 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,4 @@
-FROM ubuntu:disco
+FROM debian:buster

 ENV LANG C.UTF-8

That got a lot farther than the other attempts, but eventually failed after installing pipenv:

To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (2: No such file or directory)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?
Error: error building at STEP "RUN apt --quiet --assume-yes update &&     apt --quiet --assume-yes --no-install-recommends install         gcc         gir1.2-glib-2.0         gobject-introspection         libcairo2-dev         libffi-dev         libgirepository-1.0-1         libgirepository1.0-dev         libglib2.0-dev         libpq5         libpq-dev         python3         python3-dev         python3-pip         python3-setuptools         python3-wheel         &&     pip3 install pipenv &&     pipenv install --ignore-pipfile &&     if [ "${build_type}" = "dev" ]; then         pipenv install --ignore-pipfile --dev     ; else         apt --quiet --assume-yes autoremove --purge             gcc             libcairo2-dev             libffi-dev             libgirepository1.0-dev             libglib2.0-dev             libpq-dev             python3-dev             &&         rm -rf /var/cache/{apt,debconf}                /var/lib/apt/lists/*                /var/log/{apt,dpkg.log}                ~/.cache     ; fi": error while running runtime: exit status 100

I tried converting the dockerfile to use CentOS, but hit the 1024 fd limit there too. I even used ulimit -H -n 4096 to raise the hard fd limit on my host to make sure that's not used to set the ulimit inside the container, but that had no effect. At this point, I think I'll give up on containers and try installing the old-fashioned way....

[dbnicholson]

Interesting. Looks to be a podman doing the limiting.

$ docker run -it --rm debian:buster sh -c 'ulimit -Hn; ulimit -Sn'
1048576
1048576
$ podman run -it --rm debian:buster sh -c 'ulimit -Hn; ulimit -Sn'
1024
1024

Pass --ulimit=4096:4096 or something.

Anyways, the original issue stands. We use buster for most everything else. I don't know why ubuntu disco was chosen here.

[mcatanzaro]

Pass --ulimit=4096:4096 or something.

Been a while. ;) Thanks for the hint!

[mcatanzaro]

Been a while. ;) Thanks for the hint!

Note: 4096 (my hard limit) isn't enough for a Debian Buster container. It ends in:

OSError: [Errno 24] Too many open files: '/root/.local/share/virtualenvs/src-Gb_cNZZO/bin/pip'

I will do try this the old-fashioned way. ;)

[adarnimrod]

Fixed with #106.

[dbnicholson]

Crazy that pip opens so many files without closing them during the transaction. Unfortunately, this seems like something you have to fix when launching the container runtime as discussed above or a bug in pip that I doubt we're going to track down here.

[adarnimrod]

@mcatanzaro I tested it locally with both Docker and podman and both were able to successfully build the image. I'm closing this issue, but feel free to reopen it in case you have some new information or steps to reproduce.

[mcatanzaro]

The new Dockerfile works, thanks!

That said, it is a bit confusing to use, per my comment at https://github.com/endlessm/azafea-metrics-proxy/issues/12#issuecomment-620002365.