Migrate from Python 3.8 to PyPy3 for performance

[ChrisTimperley]

This is most likely caused by unversioned dependencies (i.e., the latest version of cython is being used, which breaks compatibility):

Step 20/47 : RUN pypy3 -m easy_install cython
 ---> Running in 04f90db9da1e
Traceback (most recent call last):
  File "/usr/lib/pypy3.9/runpy.py", line 200, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/pypy3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3/dist-packages/easy_install.py", line 4, in <module>
    from setuptools.command.easy_install import main
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 14, in <module>
    from setuptools.dist import Distribution, Feature
  File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 24, in <module>
    from setuptools.depends import Require
  File "/usr/lib/python3/dist-packages/setuptools/depends.py", line 7, in <module>
    from .py33compat import Bytecode
  File "/usr/lib/python3/dist-packages/setuptools/py33compat.py", line 54, in <module>
    unescape = getattr(html, 'unescape', html_parser.HTMLParser().unescape)
AttributeError: 'HTMLParser' object has no attribute 'unescape'
The command '/bin/sh -c pypy3 -m easy_install cython' returned a non-zero code: 1
Makefile:41: recipe for target 'crepair' failed
make: *** [crepair] Error 1
make: Leaving directory '/usr0/home/chris/tools/CrashRepair/docker'

I strongly suggest either using pip freeze to generate a complete set of versioned dependencies from an existing image (and subsequently using pip install -r requirements.txt) or otherwise manually specifying versions for each package in Dockerfile.crepair.

[ChrisTimperley]

Here's a workaround Dockerfile for now:

FROM crepair:builder
COPY --from=crepair:z3 /opt/z3/ /opt/z3/
COPY --from=crepair:llvm-6 /opt/llvm-6/ /opt/llvm-6/
COPY --from=crepair:llvm-11 /opt/llvm/ /opt/llvm11/
COPY --from=crepair:klee /klee/ /klee
COPY --from=crepair:klee /klee-uclibc /klee-uclibc

RUN add-apt-repository -y ppa:pypy/ppa
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y  --no-install-recommends --force-yes \
    bear \
    clang-tidy \
    clang-10 \
    file \
    gfortran \
    pypy3 \
    pypy3-dev \
    python3.8 \
    python3.8-dev \
    python3-pip \
    python3-setuptools

RUN python3.8 -m pip install --upgrade pip
RUN python3.8 -m pip --disable-pip-version-check --no-cache-dir install setuptools
RUN python3.8 -m pip --disable-pip-version-check --no-cache-dir install pylint
RUN python3.8 -m pip --disable-pip-version-check --no-cache-dir install cython
RUN python3.8 -m pip --disable-pip-version-check --no-cache-dir install pysmt==0.9.0
RUN pysmt-install --z3 --confirm-agreement
RUN python3.8 -m pip --disable-pip-version-check --no-cache-dir install funcy
RUN python3.8 -m pip --disable-pip-version-check --no-cache-dir install six
RUN python3.8 -m pip --disable-pip-version-check --no-cache-dir install numpy==1.19.1
RUN python3.8 -m pip --disable-pip-version-check --no-cache-dir install wllvm; return 0;
RUN python3.8 -m pip --disable-pip-version-check --no-cache-dir install sympy

# WARNING: CT thinks that this may use the same Python package directory as the standard pip
RUN pypy3 -m pip install \
      cython==0.29.32 \
      distlib==0.3.6 \
      setuptools==65.6.3 \
      pip==22.3.1 \
      funcy==1.17 \
      six==1.16.0
RUN pypy3 -m pip install \
      wllvm==1.3.1 \
      sympy==1.11.1
RUN python3 -m pip install wllvm==1.3.1
ENV PATH "/opt/llvm-6/bin:/klee/build/bin:${PATH}"
ENV LLVM_COMPILER=clang

WORKDIR /CrashRepair

COPY lib ./lib
RUN cd /CrashRepair/lib \
 && KLEE_INCLUDE_PATH=/klee/source/include make

COPY app ./app
COPY bin ./bin
COPY compiler ./compiler
COPY experiments ./experiments
COPY tests ./tests
COPY Repair.py ./Repair.py

ENV PATH="/opt/crashrepair/bin:/CrashRepair/compiler:/klee/build/bin:${PATH}"
ENV CC=crepair-cc
ENV CXX=crepair-cxx
ENV LD_LIBRARY_PATH "/opt/crashrepair/lib:/CrashRepair/lib:/klee/build/lib:${LD_LIBRARY_PATH}"

RUN ln -s /CrashRepair/bin/crepair /usr/bin/crepair

RUN crepair --help && exit 0

COPY --from=crepair:sourcerepair /opt/crashrepair /opt/crashrepair
RUN rm /opt/crashrepair/lib/libz3.so
COPY --from=crepair:orchestrator /opt/crashrepair /opt/crashrepair

I notice that there's three separate versions of Python in this image. Which one is actually used by the analysis?

[ChrisTimperley]

I've added a workaround to the Dockerfile for this issue, but we should remove the redundant package installations. That is, we should figure out whether Python 3 (system), Python 3.8, or PyPy is being used by the analysis. I assume that only one of these is running (and is being used by the analysis module), but if that isn't the case, let me know. Once we figure that out, we should remove the package installations for the alternative Python versions.

[ChrisTimperley]

@rshariffdeen's reply at our technical meeting: we currently use Python 3.8 but plan to PyPy for better performance (as long as there are no fatal incompatibilities). There shouldn't be any Python 3 (system) package installations.