Randomly fails to compile on Alpine Linux (usually on dot_h.o)

[IlyaSemenov]

When I compile uwsgi 2.0.13.1 with python3 in Alpine Linux (in a Docker container), it fails randomly. The worst thing it's not consistent.

Consider the following Dockerfile:

FROM alpine:3.4
RUN apk add --no-cache build-base linux-headers python3-dev
RUN pip3 install --no-cache-dir --upgrade pip setuptools
RUN pip3 install --no-cache-dir --disable-pip-version-check --verbose uwsgi

Sometimes docker build . will fail like this:

...
[thread 0][gcc] core/ssl.o
[thread 1][gcc] core/legion.o
[thread 0][gcc] core/dot_h.o
Running setup.py install for uwsgi: finished with status 'error'

sometimes like this:

...
[thread 1][gcc] core/legion.o
[thread 0][gcc] core/dot_h.o
gcc: error trying to exec 'cc1': execvp: No such file or directory
Running setup.py install for uwsgi: finished with status 'error'

and sometimes it will compile just fine:

...
[thread 0][gcc] core/legion.o
[thread 1][gcc] core/dot_h.o
[thread 1][gcc] core/config_py.o
*** uWSGI compiling embedded plugins ***
[thread 1][gcc] plugins/python/python_plugin.o
[thread 0][gcc] plugins/python/pyutils.o
...

You can check full output logs for failed and successful runs in this Gist - https://gist.github.com/IlyaSemenov/20ba0a84d548aad2dc73832551abf99f

It looks like there is a race condition of some kind.

I repeated this on 4 different Docker hosts with different base OS'es (OS X and Ubuntu 16.04).

[JayH5]

+1 we have this same problem. We build wheels for uWSGI in Docker containers on Debian with Python 2.7/3.5 and Alpine Linux with Python 2.7/3.5. It builds fine every time on 3/4 of those platforms. But on Alpine Linux + Python 3.5 it fails to build probably around half the time, always with the same error:

gcc: error trying to exec 'cc1': execvp: No such file or directory

I find that it's usually at core/dot_h.o, but not always. Here's a gist of it failing at lib/linux_ns.o: https://gist.github.com/JayH5/90242fa154451754218c909fe39dc551

Wish I could be of more help. It's hard to know what's going on here 😢

[JayH5]

I've seen this happen once on Alpine Linux + Python 2.7 now, although it seems rarer than with Python 3.5. Seems likely that it is gcc-specific rather than Python-specific.

EDIT: Seems very common with Python 2.7 as well now. Not sure if something changed in the underlying Alpine image...

[IlyaSemenov]

@JayH5 typically there's no need to compile uwsgi on alpine with Python 2.7 as there is a precompiled package uwsgi-python. Just a heads up if you missed it.

[IlyaSemenov]

Just wanted to update that the problem still persists in 2.0.14:

  Downloading from URL https://pypi.python.org/packages/11/ef/3d64655693e12f163f659bc3a45837893d7ebbb242e608feb9adc788d2ca/uwsgi-2.0.14.tar.gz#md5=d4b2c7db274dc20be54a977c9b040b7e (from https://pypi.python.org/simple/uwsgi/)
  Running setup.py (path:/tmp/pip-build-ot8l_tsj/uwsgi/setup.py) egg_info for package uwsgi
...
    [thread 1][gcc] proto/scgi.o
    [thread 1][gcc] proto/puwsgi.o
    [thread 1][gcc] lib/linux_ns.o
    [thread 1][gcc] core/yaml.o
    [thread 1][gcc] core/dot_h.o
    Running setup.py install for uwsgi: finished with status 'error'
Cleaning up...
  Removing source in /tmp/pip-build-ot8l_tsj/uwsgi
Command "/usr/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ot8l_tsj/uwsgi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-9xmdi7uv-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-ot8l_tsj/uwsgi/
Exception information:
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3.5/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/usr/lib/python3.5/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/lib/python3.5/site-packages/pip/req/req_install.py", line 878, in install
    spinner=spinner,
  File "/usr/lib/python3.5/site-packages/pip/utils/__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "/usr/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ot8l_tsj/uwsgi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-9xmdi7uv-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-ot8l_tsj/uwsgi/
The command '/bin/sh -c pip3 install --no-cache-dir --disable-pip-version-check --verbose uwsgi' returned a non-zero code: 1

[IlyaSemenov]

Also, if anyone finds that useful, the two workarounds I'm using are:

1) Loop until it compiles:

RUN apk add --no-cache \
        build-base \
        linux-headers `# for uwsgi` \
        nginx \
        python3 \
        python3-dev \
        sudo \
        supervisor && \
    pip3 install --no-cache-dir --upgrade pip setuptools && \
    `# uwsgi compile fails randomly, see https://github.com/unbit/uwsgi/issues/1318` && \
    (while true; do pip3 install --no-cache-dir --disable-pip-version-check --verbose uwsgi && break; done) && \
    ...

2) Use Alpine v3.5's pre-built package:

FROM alpine:3.4
# Use alpine v3.5 for uwsgi-python3 (get rid of this once there is an official Docker tag)
RUN sed -i -e 's/v3\.4/v3.5/' /etc/apk/repositories

RUN apk add --no-cache \
        ...
        uwsgi \
        uwsgi-python3 \
        ...

(you can also try to add a named repo for Alpine 3.5 and add uwsgi-python3@3.5but for me it would then fail to install build-base which I needed for other packages).

[jhoos]

I encountered this error fairly consistently with uwsgi 2.0.17.1 in an Alpine 3.8 container I was using today, exactly as the original poster described. I'm running Docker 18.06.1-ce-mac73 (26764) on MacOS, with 2 CPUs allocated to Docker. I also saw the error consistently in CircleCI, also running with 2 CPUs.

Curiously, I did not see the error when running Docker on my Mac with 4 CPUs instead of 2.

In my particular case, doing export CPUCOUNT=1 prior to installing uwsgi (which limits uwsgi to only using one CPU thread to build its C code) seems to have worked around it; without that, my builds failed pretty consistently, and with it I've run 10-12 builds in a row without failure.

[SuperSandro2000]

This happens for 100% when i build my docker image with buildkit. Normal docker build work fine...

[SuperSandro2000]

I think I found the issue. I changed WORKDIR before install and got x86_64-linux-gnu-gcc: fatal error: cannot execute ‘cc1’: execvp: No such file or directory not changing it or after fixes it. Strange.

[btorellALTA]

I am running into this exact same issue using a base Ubuntu 20.04 image for my Docker container.

Setting CPUCOUNT=1 "fixes" the issue, but that doesn't really feel like a fix to me. Setting the command to repeatedly run until it succeeds also "works" but feels like an even worse solution.

Oddly enough, I'm able to get it to work cleanly on Ubuntu 18.04. Could this be some sort of compiler version issue?

[satoshi]

I ran into the same problem when I switched from Ubuntu MATE 20.04 LTS to Xubuntu 20.04 LTS (same OS, different DE). The CPUCOUNT=1 trick worked; this workaround also did the trick (the path may vary, depending on where you have cc1):

export PATH=$PATH:/usr/lib/gcc/x86_64-linux-gnu/9

[sbraz]

We're also running into failures with Portage's sandbox: https://bugs.gentoo.org/724394

I have tried to use processes instead of threads and that fixes the build for me. Can you guys try that patch with 2.0.18 and tell me if that works? In that case, I'll make a PR:

diff --git a/uwsgiconfig.py b/uwsgiconfig.py
index 22c9dd346..3dd6cb388 100644
--- a/uwsgiconfig.py
+++ b/uwsgiconfig.py
@@ -12,14 +12,9 @@

 import sys
 import subprocess
-from threading import Thread,Lock
+from multiprocessing import Process, Lock, Queue
 from optparse import OptionParser

-try:
-    from queue import Queue
-except:
-    from Queue import Queue
-
 from distutils import sysconfig

 try:
@@ -271,7 +266,7 @@ def build_uwsgi(uc, print_only=False, gcll=None):
         print_lock = Lock()
         compile_queue = Queue(maxsize=CPUCOUNT)
         for i in range(0,CPUCOUNT):
-            t = Thread(target=thread_compiler,args=(i,))
+            t = Process(target=thread_compiler,args=(i,))
             t.daemon = True
             t.start()
             thread_compilers.append(t)

[whyscream]

This problem is still happening, tested with uwsgi 2.0.20, debian 11 and python 3.10. A simple Dockerfile that exposes the problem:

FROM python:3.10-slim-bullseye

# Install debian build tools
RUN apt-get update \
  && apt-get install --yes --no-install-recommends build-essential \
  && rm -rf /var/lib/apt/lists/*

# Configure virtualenv
ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# Install python build tools
RUN pip install --upgrade pip setuptools wheel

# Install uwsgi
RUN pip install uwsgi

The issue can still be resolved by setting CPUCOUNT=1 before running pip install.

[categulario]

Same issue here using podman and image ubuntu:20.04. CPUCOUNT=1 solved it

[pajot]

Same problem on Ubuntu 20.04 with build-essential-12.8, gcc 9.3.0 and uWSGI 2.0.20 installed in a python venv. CPUCOUNT=1 works around it.

[vakorol]

I can confirm that CPUCOUNT=1 fixes the issue when using the Debian-based Docker image for Python:

FROM python:3.12-bullseye as build                                                      
ENV PYTHONUNBUFFERED=1 \                                                                
    PYROOT=/pyroot \                                                                    
    PROJECT_DIR=/code                                                                   

ENV PYTHONUSERBASE=$PYROOT \                                                            
    PATH=$PYROOT/bin:$PATH \                                                            
    PYTHONPATH=$PYTHONPATH:$PROJECT_DIR/src \                                           
    MYPYPATH=$PYTHONPATH:$PROJECT_DIR/src                                               

RUN apt-get update \                                                                    
    && apt-get install -y --no-install-recommends \                                     
    ▎   libxml2 \                                                                       
    && rm -rf /var/lib/apt/lists/*                                                      

RUN PIP_USER=1 pip install --upgrade pip pip-tools                                      
WORKDIR $PROJECT_DIR                                                                    
COPY requirements.txt dev-requirements.txt $PROJECT_DIR/                                

RUN PIP_USER=1 CPUCOUNT=1 pip-sync --pip-args --no-deps requirements.txt dev-requirements.txt