To build the docker image I need to clone CPython from private GitLab

[LiuMicheal]

Excellent work! I am learning it. Here is a little question: I found that many docker images used in $PROJECT_PATH/exp are based on dmerge-base image. The Dockerfile for building dmerge-base image is in $PROJECT_PATH/samples/knative/dockerfiles. dmerge-base image is based on cpython-build image, and building cpython-build image requires cloning CPython from private GitLab.

FROM ubuntu:16.04 as Build

ARG ACCESS_TOKEN=token

WORKDIR /usr/src RUN apt-get update && \ apt-get install -y git build-essential wget curl tar libffi-dev zlib1g-dev libbz2-dev \ libssl-dev libncurses-dev libsqlite3-dev libreadline-dev tk-dev gcc make cron locales \ cmake autoconf automake libtool

RUN git clone https://docker:${ACCESS_TOKEN}@ipads.se.sjtu.edu.cn:1312/distributed-rdma-serverless/distributed-merge/cpython.git my_python_repo && \ cd my_python_repo && \ git checkout 3.7 && \ ./configure --enable-optimizations && \ make -j build_all && \ make -j install && \ make clean && \ cd .. && \ rm -rf my_python_repo && \ ln -s /usr/local/bin/python3.7 /usr/local/bin/python && \ ln -s /usr/local/bin/pip3 /usr/local/bin/pip && \ pip install --upgrade pip

wget https://bootstrap.pypa.io/get-pip.py

RUN python -m pip install Flask==2.0.3 requests cloudevents redis numpy==1.21.6 yfinance lightgbm==2.0.7

Is there anything special about this CPython? Can I use https://github.com/python/cpython instead? Thank you very much.

[xiehongrui]

We made some modifications on cpython, you can get our cpython-build image by:

docker pull hongruixie/cpython-build

[LiuMicheal]

Thank you very much for your answer. I will try it.

[LiuMicheal]

Maybe I need to use cpython-build image to build cpython-base image, and here is /root/dmerge/samples/knative/dockerfiles/Dockerfile-cpython-base:

FROM val01:5000/cpython-build as build

**WORKDIR /app**

# Install dependencies
RUN rm -f requirements.txt && \
    echo "cython" >> requirements.txt && \
    echo "numpy==1.19.3" >> requirements.txt && \
    echo "Flask==2.0.3" >> requirements.txt && \
    echo "requests" >> requirements.txt && \
    echo "yfinance" >> requirements.txt && \
    echo "cloudevents==1.9.0" >> requirements.txt && \
    echo "redis" >> requirements.txt && \
    echo "lightgbm==2.0.7" >> requirements.txt && \
    echo "urllib3==1.25.3" >> requirements.txt && \
    echo "minio" >> requirements.txt && \
    echo "fastapi" >> requirements.txt && \
    echo "gunicorn" >> requirements.txt && \
    echo "uvicorn" >> requirements.txt && \
    pip install -r requirements.txt

# LD_PRELOAD related
COPY . .
RUN ls && \
    cd dynlib-wrappers/jemalloc-wrapper && \
    mkdir -p build && cd build && \
    cmake .. && \
    make -j && \
    cp libmalloc_wrapper.so /usr/local/share/libmalloc_wrapper.so && \
    cd ../../ && rm -rf jemalloc-wrapper

However, when I use

docker pull hongruixie/cpython-build

and

ls / bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var

It seems that there is no /app. Maybe I can not build cpython-base image. Could you please provide some suggestions on how I can further search for a solution? I will do my best to explore further. Thanks.

[xiehongrui]

If the WORKDIR doesn’t exist, it will be created. So it's ok if no /app in cpython-build image. You can follow the makefile in samples/knative/dockerfile to build cpython-base (you may need to check the image name in makefile and dockerfile). You can also get it by:

docker pull hongruixie/cpython-base

I have pushed some necessary images.

[LiuMicheal]

Since I have run Microbenchmark and Application evaluations, I found some images are missing: dmerge-micro dmerge-finra dmerge-ml-pipeline dmerge-digital-minist dmerge-wordcount dmerge-test dmerge-thpt Maybe they are not in DockerHub hongruixie now. Could you please provide these images? Thanks a lot!

[LiuMicheal]

I solved these problems. I found that there were Dockerfiles in the corresponding directories in $PROJECT_PATH/exp. I used docker build to generate all the above images based on hongruixie/dmerge-base.

It is worth mentioning that after I built the image, an error occurred during pod startup, there lacks async_timeout for python runtime. I add a line to requirements.txt: async_timeout

After that, these images I generated by myself all run correctly.

[xiehongrui]

Glad to see it run correctly and thank you for mentioning that. I pushed these images in case other users find similar issue