[BUG] Docker build is failing, compilation in Docker and out of Docker is failing too

guilhemmarchand commented 4 years ago

Describe the bug

With the goal of using Eventgen in a PyPI engine to leverage more capabilities, I am facing failures to build a Docker container, and even building EvenGen out of Docker.

In addition, while the repo provides a Dockerfile, there are 0 instructions at all reagrding Docker, and not even a mention of the Docker option in the docs, what about publishing a release in Docker Hub ? This would be really awesome and time saving ;-)

To Reproduce Steps to reproduce the behavior:

Use the master branch provided Dockerfile and starting building.

Expected behavior

Build should be successful and Docker image ready.

Actual behavior

Once the the package is built successfully with:

python setup.py sdist

Creating tar archive
removing 'splunk_eventgen-7.1.0' (and everything under it)

Running Docker build (instructions would be nice)

docker build . -f dockerfiles/Dockerfile -t splunk-eventgen:701

Results:

Collecting git+https://github.com/esnme/ultrajson.git
  Cloning https://github.com/esnme/ultrajson.git to /tmp/pip-req-build-gdcrJ7
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
ujson requires Python '>=3.5' but the running Python is 2.7.18
The command '/bin/sh -c apk --no-cache upgrade &&   apk add --no-cache --update     python3     python3-dev     python2-dev     py2-pip     gcc     libc-dev    libffi-dev  openssl-dev     libxml2-dev     libxslt-dev     bash    sudo openssh    tar     acl     g++     git     curl &&     pip3 install --upgrade pip &&   rm -rf /tmp/* &&    rm -rf /var/cache/apk/* &&  ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa &&     mkdir -p /var/run/sshd &&   mkdir -p /root/.ssh &&  chmod 0700 /root/.ssh &&    passwd -u root &&   pip2 install filelock twisted requests queuelib ujson psutil crochet msgpack-python unidecode attrdict service_identity &&  pip2 install git+https://github.com/esnme/ultrajson.git' returned a non-zero code: 1

I have tried different things, like using a more recent version of the redis alpine: (note: py2-pip does not exist anymore, py3-pip is required and pip2 changed to pip3)

FROM redis:6.0.5-alpine

RUN apk --no-cache upgrade && \
    apk add --no-cache --update \
    python3 \
    python3-dev \
    python2-dev \
    py3-pip \
    gcc \
    libc-dev \
    libffi-dev \
    openssl-dev \
    libxml2-dev \
    libxslt-dev \
    bash \
    sudo \
    openssh \
    tar \
    acl \
    g++ \
    git \
    curl && \
    pip3 install --upgrade pip && \
    rm -rf /tmp/* && \
    rm -rf /var/cache/apk/* && \
    ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa && \
    mkdir -p /var/run/sshd && \
    mkdir -p /root/.ssh && \
    chmod 0700 /root/.ssh && \
    passwd -u root && \
    # install dependencies of conduct2 used by perf
    pip3 install filelock twisted requests queuelib ujson psutil crochet msgpack-python unidecode attrdict service_identity && \
    pip3 install git+https://github.com/esnme/ultrajson.git

COPY dockerfiles/sshd_config /etc/ssh/sshd_config
COPY dockerfiles/entrypoint.sh /sbin/entrypoint.sh
COPY dist/* /root/splunk_eventgen.tgz
RUN pip3 install /root/splunk_eventgen.tgz && \
    rm /root/splunk_eventgen.tgz

EXPOSE 2222 6379 9500
RUN chmod a+x /sbin/entrypoint.sh
WORKDIR /usr/lib/python3.7/site-packages/splunk_eventgen
ENTRYPOINT ["/sbin/entrypoint.sh"]

But complilation fails:

    src/lxml/etree.c:231453:35: note: in expansion of macro '__Pyx_PyCode_New'
    231453 |   __pyx_codeobj__638 = (PyObject*)__Pyx_PyCode_New(1, 0, 2, 0, CO_OPTIMIZED|CO_NEWLOCALS, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple__637, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_src_lxml_debug_pxi, __pyx_n_s_dict_size, 25, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__638)) __PYX_ERR(25, 25, __pyx_L1_error)
           |                                   ^~~~~~~~~~~~~~~~
    In file included from /usr/include/python3.8/compile.h:5,
                     from /usr/include/python3.8/Python.h:138,
                     from src/lxml/etree.c:97:
    /usr/include/python3.8/code.h:122:28: note: declared here
      122 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
          |                            ^~~~~~~~~~
    src/lxml/etree.c:411:11: error: too many arguments to function 'PyCode_New'
      411 |           PyCode_New(a, 0, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
          |           ^~~~~~~~~~
    src/lxml/etree.c:231465:35: note: in expansion of macro '__Pyx_PyCode_New'
    231465 |   __pyx_codeobj__640 = (PyObject*)__Pyx_PyCode_New(3, 0, 5, 0, CO_OPTIMIZED|CO_NEWLOCALS, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple__639, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_src_lxml_debug_pxi, __pyx_n_s_dump, 36, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__640)) __PYX_ERR(25, 36, __pyx_L1_error)
           |                                   ^~~~~~~~~~~~~~~~
    In file included from /usr/include/python3.8/compile.h:5,
                     from /usr/include/python3.8/Python.h:138,
                     from src/lxml/etree.c:97:
    /usr/include/python3.8/code.h:122:28: note: declared here
      122 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
          |                            ^~~~~~~~~~
    src/lxml/etree.c:411:11: error: too many arguments to function 'PyCode_New'
      411 |           PyCode_New(a, 0, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
          |           ^~~~~~~~~~
    src/lxml/etree.c:231477:35: note: in expansion of macro '__Pyx_PyCode_New'
    231477 |   __pyx_codeobj__642 = (PyObject*)__Pyx_PyCode_New(3, 0, 4, 0, CO_OPTIMIZED|CO_NEWLOCALS, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple__641, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_src_lxml_debug_pxi, __pyx_n_s_show, 66, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__642)) __PYX_ERR(25, 66, __pyx_L1_error)
           |                                   ^~~~~~~~~~~~~~~~
    In file included from /usr/include/python3.8/compile.h:5,
                     from /usr/include/python3.8/Python.h:138,
                     from src/lxml/etree.c:97:
    /usr/include/python3.8/code.h:122:28: note: declared here
      122 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
          |                            ^~~~~~~~~~
    src/lxml/etree.c: In function '__Pyx_CreateCodeObjectForTraceback':
    src/lxml/etree.c:411:11: error: too many arguments to function 'PyCode_New'
      411 |           PyCode_New(a, 0, k, l, s, f, code, c, n, v, fv, cell, fn, name, fline, lnos)
          |           ^~~~~~~~~~
    src/lxml/etree.c:247326:15: note: in expansion of macro '__Pyx_PyCode_New'
    247326 |     py_code = __Pyx_PyCode_New(
           |               ^~~~~~~~~~~~~~~~
    In file included from /usr/include/python3.8/compile.h:5,
                     from /usr/include/python3.8/Python.h:138,
                     from src/lxml/etree.c:97:
    /usr/include/python3.8/code.h:122:28: note: declared here
      122 | PyAPI_FUNC(PyCodeObject *) PyCode_New(
          |                            ^~~~~~~~~~
    Compile failed: command 'gcc' failed with exit status 1
    creating tmp
    cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInit73hx7a2z.c -o tmp/xmlXPathInit73hx7a2z.o
    cc tmp/xmlXPathInit73hx7a2z.o -L/lib -lxml2 -o a.out
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-go_4mgyl/lxml/setup.py'"'"'; __file__='"'"'/tmp/pip-install-go_4mgyl/lxml/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-m5966m6t/install-record.txt --single-version-externally-managed --compile --install-headers /usr/include/python3.8/lxml Check the logs for full command output.
The command '/bin/sh -c pip3 install /root/splunk_eventgen.tgz &&   rm /root/splunk_eventgen.tgz' returned a non-zero code: 1

I assume the current version of the setup.py would need some update to work with latest version of Python3 and pip ? But the develop branch does not contain anything at all.

Thank you in advance

guilhemmarchand commented 4 years ago

Update on this issue:

I could finally successfully generate a Docker image doing the following:

From master branch, I created the eventgen tgz:

python setup.py sdist

Then I switched to the develop branch, and ran the docker build using:

docker build --pull --rm -f "dockerfiles/Dockerfile" -t eventgen:710 .

Notes: Having these intruction would be helpful for others.

Finally I could achieve my goal and running successfully the EventGen as a PyPI within Docker, which is super awesome.

I pushed to: https://hub.docker.com/repository/docker/guilhemmarchand/splunk-eventgen

With some quick instructions using docker-compose.

guilhemmarchand commented 4 years ago

Note I tried to get the last alpine version working but the compilation will fail:

    Compile failed: command 'gcc' failed with exit status 1
    creating tmp
    cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInithcfcdu93.c -o tmp/xmlXPathInithcfcdu93.o
    cc tmp/xmlXPathInithcfcdu93.o -L/lib -lxml2 -o a.out
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-4zmufbxl/lxml/setup.py'"'"'; __file__='"'"'/tmp/pip-install-4zmufbxl/lxml/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-lat_iuku/install-record.txt --single-version-externally-managed --compile --install-headers /usr/include/python3.8/lxml Check the logs for full command output.

Tried the following Dockerfile:

FROM redis:6.0.5-alpine

RUN apk --no-cache upgrade && \
    apk add --no-cache --update \
    python3 \
    python3-dev \
    python2-dev \
        py3-pip \
    gcc \
    libc-dev \
    libffi-dev \
    openssl-dev \
    libxml2-dev \
    libxslt-dev \
    bash \
    sudo \
    openssh \
    tar \
    acl \
    g++ \
    git \
    curl && \
    pip3 install --upgrade pip && \
    rm -rf /tmp/* && \
    rm -rf /var/cache/apk/* && \
    ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa && \
    mkdir -p /var/run/sshd && \
    mkdir -p /root/.ssh && \
    chmod 0700 /root/.ssh && \
    passwd -u root && \
    # install dependencies of conductor2 used by perf
    pip3 install filelock twisted requests queuelib psutil crochet msgpack-python unidecode attrdict service_identity && \
    pip3 install git+https://github.com/esnme/ultrajson.git@v1.35

COPY dockerfiles/sshd_config /etc/ssh/sshd_config
COPY dockerfiles/entrypoint.sh /sbin/entrypoint.sh
COPY dist/splunk_eventgen*.tar.gz /root/splunk_eventgen.tgz
RUN pip3 install /root/splunk_eventgen.tgz && \
    rm /root/splunk_eventgen.tgz
COPY pyproject.toml /usr/lib/python3.7/site-packages/splunk_eventgen/pyproject.toml
COPY poetry.lock /usr/lib/python3.7/site-packages/splunk_eventgen/poetry.lock

EXPOSE 2222 6379 9500
RUN chmod a+x /sbin/entrypoint.sh
WORKDIR /usr/lib/python3.7/site-packages/splunk_eventgen
ENTRYPOINT ["/sbin/entrypoint.sh"]

Replacing py2-pip by pi3-pip and pip2 by pip3 (as it is not available anymore) but won't work.

jmeixensperger commented 4 years ago

As you mentioned, the fix is in the develop branch and will be available in the next release, so closing this issue out. We are planning to get a new release out by the end of day today. If you still see the issue, feel free to re-open. We also don't have any plans to upgrade the alpine image, as we have dependencies on the python2 ujson package.

guilhemmarchand commented 4 years ago

@jmeixensperger thank you very much, perfectly fine 👍

splunk / eventgen

[BUG] Docker build is failing, compilation in Docker and out of Docker is failing too #403