search

renpy / renpy-build

Build system for the Ren'Py visual novel engine. (The engine itself, not games.)
76 stars 51 forks source link

Dosent work on a clean install #85

Closed Meithal closed 1 year ago

Meithal commented 1 year ago

I tried to follow the instructions for making a clean install but there were many things that didn't work out the box, which make me thing the install instructions only work in a tampered environment. I tried to make a dockerfile from a clean image like this

FROM ubuntu:22.04

COPY . .
RUN apt-get -qq update && ./prepare.sh

But it complained about sudo, so I ended up making a complete dockerfile but still some tasks require sudo so I would have to tamper with the other files in it, can you look into having this work from a clean install,

Basically try this Dockefile

FROM ubuntu:22.04

# see https://ubuntu.com/blog/we-reduced-our-docker-images-by-60-with-no-install-recommends
RUN echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf.d/00-docker
RUN echo 'APT::Install-Recommends "0";' >> /etc/apt/apt.conf.d/00-docker

SHELL ["/bin/bash", "-xe", "-c"]
ENV RENPY_QUIET=--verbose

ARG DEBIAN_FRONTEND=noninteractive
ARG ROOT=/app
ARG BASE=$ROOT
ARG REFS=$ROOT

WORKDIR $ROOT

COPY . .

RUN apt-get -qq update && \
 apt-get -qq install \
    git build-essential ccache unzip autoconf autoconf-archive automake libtool-bin \
    lsb-release wget software-properties-common gnupg \
    python2-dev python3-dev python3-venv \
    curl \
    python3-jinja2 \
    debootstrap qemu-user-static \
    libgmp-dev libmpfr-dev libmpc-dev \
    libssl-dev libbz2-dev liblzma-dev \
    cmake clang libxml2-dev llvm \
    quilt \
    libavcodec-dev libavformat-dev \
    libswresample-dev libswscale-dev libfreetype6-dev libfribidi-dev libsdl2-dev \
    libsdl2-image-dev libsdl2-gfx-dev libsdl2-mixer-dev libsdl2-ttf-dev libjpeg-dev && \
    apt-get autoremove -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

RUN mkdir -p $ROOT/tmp && \
    wget -O llvm.sh https://apt.llvm.org/llvm.sh && \
    chmod +x llvm.sh && \
    ./llvm.sh 15 && \
    tar xzf "$BASE/prebuilt/clang_rt.tar.gz" -C /usr/lib/clang/15/lib/

RUN pushd tars && \
    wget -q https://github.com/alexey-lysiuk/macos-sdk/releases/download/12.3/MacOSX12.3.tar.xz && \
    wget -q https://github.com/growtopiajaw/iPhoneOS-SDK/releases/download/v1.0/iPhoneOS14.0.sdk.zip && \
    wget -q https://github.com/jnicholls/cross-rust/raw/main/iPhoneSimulator14.3.sdk.tar.xz && \
    wget -q https://dl.google.com/android/repository/android-ndk-r25b-linux.zip && \
    wget -q https://cubism.live2d.com/sdk-native/bin/CubismSdkForNative-4-r.6.2.zip && \
    tar -xJf MacOSX12.3.tar.xz && tar -cjf MacOSX12.3.tar.bz2 MacOSX12.3.sdk && \
    unzip iPhoneOS14.0.sdk.zip && tar -czf iPhoneOS14.0.sdk.tar.gz iPhoneOS14.0.sdk && \
    tar -xJf iPhoneSimulator14.3.sdk.tar.xz && tar -czf iPhoneSimulator14.0.sdk.tar.gz iPhoneSimulator14.3.sdk && \
    popd

RUN VENV="$ROOT/tmp/virtualenv.py3" && \
    export RENPY_DEPS_INSTALL=/usr::/usr/lib/x86_64-linux-gnu/ && \
    . $BASE/nightly/git.sh && \
    . $BASE/nightly/python.sh || true

The || true is because python.sh calls renpy/run.sh which calls renpy/modules/setup.py which fails with Could not find required library SDL2. which comes from include("SDL.h", directory="SDL2"). It's not apparent to me at what point the SDL2 folder is created. That and seeing how many things are cached make me think the install instructions only work in a tampered environnement. In my opinion there should be a CI thing running this dockerfile to check that the build process works whenever this repo or the dependencies of this repo (renpy, pygame_sdl2...) change.

Run it with docker build . --progress=plain -t renpy

Once it is built you can start it with docker run -it --name=rc renpy Inside the shell type . tmp/virtualenv.py3/bin/activate then ./build.py. It should fail because one task requires sudo, On my local copy I removed the sudo things and it went through. You can use docker cp to get the build artifacts.

renpytom commented 1 year ago

We don't support building in docker, so I wouldn't expect things to work there. It should work in a VM, with the instructions givern - if there's an issue with the supplied scripts, please report them so they can be fixed.

brainos233 commented 1 year ago

I think env RENPY_DEPS_INSTALL affects the result of run.sh

$ ./run.sh launcher quit; echo "exit code: $?"
/home/brainos/Builds/renpy-build-test/tmp/virtualenv.py3/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
warning: no files found matching 'gen/*.c'
warning: no files found matching 'gen/*.h'
Could not find required library SDL2.
exit code: 255
$ RENPY_DEPS_INSTALL=/usr::/usr/lib/x86_64-linux-gnu/ ./run.sh launcher quit; echo "exit code: $?"
/home/brainos/Builds/renpy-build-test/tmp/virtualenv.py3/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
warning: no files found matching 'gen/*.c'
warning: no files found matching 'gen/*.h'
/home/brainos/Builds/renpy-build-test/tmp/virtualenv.py3/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
exit code: 0
$

But I suggest replacing all sudo manually.

sed -i "32a sed -i 's#sudo ##g' prepare.sh reset.sh tasks/sysroot.py" nightly/git.sh && sed -i 's#sudo ##g' prepare.sh reset.sh tasks/sysroot.py && ./prepare.sh

And you also need steamworks_sdk_156.zip. Otherwise the build will fail.

Meithal commented 1 year ago

Yes, I managed to have it working to the end by adding

ENV RENPY_DEPS_INSTALL=/usr::/usr/lib/aarch64-linux-gnu

at the top and removing the line

export RENPY_DEPS_INSTALL=/usr::/usr/lib/x86_64-linux-gnu/

at the bottom. For some reason the libSDL2 is installed in the folder /usr/lib/aarch64-linux-gnu on docker instead of /usr/lib/x86_64-linux-gnu/

A portable way to fix this would be to use dirname $(find /usr/lib* -name "libSDL2*" | head - n 1)

Now my dockerfile is

FROM ubuntu:22.04

SHELL ["/bin/bash", "-xe", "-c"]
ENV RENPY_QUIET=--verbose
ENV RENPY_DEPS_INSTALL=/usr::/usr/lib/aarch64-linux-gnu

ARG DEBIAN_FRONTEND=noninteractive
ARG ROOT=/app
ARG BASE=$ROOT
ARG REFS=$ROOT

WORKDIR $ROOT

RUN apt-get -qq update && \
 apt-get -qq install -y --no-install-recommends \
    sudo git build-essential ccache unzip autoconf autoconf-archive automake libtool-bin \
    lsb-release wget software-properties-common gnupg \
    python2-dev python3-dev python3-venv \
    curl \
    python3-jinja2 \
    debootstrap qemu-user-static \
    libgmp-dev libmpfr-dev libmpc-dev \
    libssl-dev libbz2-dev liblzma-dev \
    cmake clang libxml2-dev llvm \
    quilt \
    libavcodec-dev libavformat-dev \
    libswresample-dev libswscale-dev libfreetype6-dev libfribidi-dev libsdl2-dev \
    libsdl2-image-dev libsdl2-gfx-dev libsdl2-mixer-dev libsdl2-ttf-dev libjpeg-dev && \
    && apt-get -y clean \
    && apt-get -y autoclean \
    && apt-get -y autoremove \
    && rm -rf /var/lib/apt/lists/* \
    && rm -rf /var/cache/debconf/*-old \
    && rm -rf /usr/share/doc/* \
    && rm -rf /usr/share/man/?? \
    && rm -rf /usr/share/man/??_* \

RUN mkdir tars && pushd tars && \
    wget -q https://github.com/alexey-lysiuk/macos-sdk/releases/download/12.3/MacOSX12.3.tar.xz && \
    wget -q https://github.com/growtopiajaw/iPhoneOS-SDK/releases/download/v1.0/iPhoneOS14.0.sdk.zip && \
    wget -q https://github.com/jnicholls/cross-rust/raw/main/iPhoneSimulator14.3.sdk.tar.xz && \
    wget -q https://dl.google.com/android/repository/android-ndk-r25c-linux.zip && \
    wget -q https://cubism.live2d.com/sdk-native/bin/CubismSdkForNative-4-r.6.2.zip && \
    tar -xJf MacOSX12.3.tar.xz && tar -cjf MacOSX12.3.sdk.tar.bz2 MacOSX12.3.sdk && \
    unzip iPhoneOS14.0.sdk.zip && tar -czf iPhoneOS14.0.sdk.tar.gz iPhoneOS14.0.sdk && \
    tar -xJf iPhoneSimulator14.3.sdk.tar.xz && tar -czf iPhoneSimulator14.0.sdk.tar.gz iPhoneSimulator14.3.sdk && \
    popd

# we put this after the previous RUNs so that their cache
# dont get invalidated.
COPY . .

# This requires prebuilt/ folder to exist, so we cant cache it.
RUN mkdir -p $ROOT/tmp && \
    wget -O llvm.sh https://apt.llvm.org/llvm.sh && \
    chmod +x llvm.sh && \
    ./llvm.sh 15 && \
    tar xzf "$BASE/prebuilt/clang_rt.tar.gz" -C /usr/lib/clang/15/lib/

RUN VENV="$ROOT/tmp/virtualenv.py3" && \
    . $BASE/nightly/git.sh && \
    . $BASE/nightly/python.sh

It works all the way through

But now when I try to build I get this error 

Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/54217a0950bb1dafe8808cc6207d378e323f9d74/wasm-binaries-arm64.tbz2': HTTP Error 404: Not Found

Maybe replace version 3.14 by latest one? Does the nightly script fail to catch this 404 because it has cached the download?

renpytom commented 1 year ago

Docker isn't a supported build environment. Please discuss this someplace else - I get notified each time someone posts here, and I don't need to be bothered by unsupported builds.

heeeeeeeeeeh commented 1 year ago

@Meithal you can fix the 404 error by replacing 3.1.24 with latest in line 102 of the file renpy-build/tasks/toolchain.py. The reason there is a 404 error is explained in this issue