oracle / graalpython

A Python 3 implementation built on GraalVM
Other
1.2k stars 104 forks source link

Building pendulum takes forever #268

Closed dszakallas closed 2 years ago

dszakallas commented 2 years ago

I've been trying to use pendulum in a Python polyglot native image, however building pendulum takes forever.

docker buildx build . --target pendulum -t pendulum --progress plain 

I've stopped the build after 1 hour spent building pendulum, with a single CPU core at max usage.

Here's the Dockerfile to reproduce the issue:

# syntax=docker/dockerfile:1.4
#########################################################
FROM scratch as paris
COPY <<"EOF" Test.java

import org.graalvm.polyglot.*;
import org.graalvm.polyglot.proxy.*;
import org.graalvm.polyglot.Context.Builder;

public class Test {
  public static String PYTHON = "python";

  public static String PROGRAM = """
    import pendulum

    now_in_paris = pendulum.now('Europe/Paris')
    print(now_in_paris)
    """;

  public static void main(String[] args) {
    Builder ctxBuilder = Context
      .newBuilder(PYTHON)
      .allowAllAccess(true)
      .option("python.ForceImportSite", "true");

    String pythonPath = System.getenv("PYTHONPATH");
    if (pythonPath != null) {
      ctxBuilder.option("python.PythonPath", pythonPath);
    }

    try (Context context = ctxBuilder.build();) {
      context.eval(PYTHON, PROGRAM);
    }
  }
}
EOF

#########################################################
FROM python:3.8-slim as base

ARG APP_PATH=/app

WORKDIR $APP_PATH

ENV LANGUAGE=en_US.UTF-8
ENV LANG=en_US.UTF-8
ENV LC_ALL=en_US.UTF-8
ENV LC_CTYPE=en_US.UTF-8
ENV LC_MESSAGES=en_US.UTF-8

RUN set -ex \
    && apt-get update -yqq \
    && mkdir -pv /usr/share/man/man1 \
    && mkdir -pv /usr/share/man/man7 \
    && apt-get install -yqq --no-install-recommends \
        locales \
    && sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g' /etc/locale.gen \
    && locale-gen \
    && update-locale \
    && apt-get install -yqq --no-install-recommends \
      build-essential \
      curl \
      libssl-dev \
      libffi-dev \
      libz-dev \
    && pip install virtualenv \
    && curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y \
    && apt-get autoremove -yqq --purge \
    && apt-get clean \
    && rm -rf \
        /var/lib/apt/lists/* \
        /tmp/* \
        /var/tmp/* \
        /usr/share/man \
        /usr/share/doc \
        /usr/share/doc-base

ENV PATH=$HOME/.cargo/bin:$PATH

#########################################################
FROM base as pendulum

RUN cd /opt/ && curl -fsSL https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-22.1.0/graalvm-ce-java17-linux-amd64-22.1.0.tar.gz | tar -xz

ENV PATH=/opt/graalvm-ce-java17-22.1.0/bin:$PATH \
    JAVA_HOME=/opt/graalvm-ce-java17-22.1.0 \
    GRAALVM_HOME=/opt/graalvm-ce-java17-22.1.0

RUN gu install python native-image \
    && graalpython -m venv $APP_PATH 

RUN . $APP_PATH/bin/activate \
    && . $HOME/.cargo/env \
    && pip install --no-cache "pendulum==2.1.2"

COPY --from=paris Test.java Test.java

RUN javac Test.java \
    && native-image --language:python Test test \
    -Dorg.graalvm.launcher.relative.python.home=$GRAALVM_HOME/languages/python \
    -Dorg.graalvm.launcher.relative.llvm.home=$GRAALVM_HOME/languages/llvm \
    && chmod +x test && mv test $APP_PATH/bin

ENV PYTHONPATH=$APP_PATH/lib/python3.8/site-packages

ENV PATH=$APP_PATH/bin:$PATH

ENTRYPOINT ["test"]
msimacek commented 2 years ago

I fixed the problem that was causing the infinite loop. Unfortunately, that's not enough for pendulum to install, because some of its build time dependencies uses ast.literal_eval, which is not yet implemented on GraalPython. It's currently being worked on.