search

DOMjudge / domjudge-packaging

DOMjudge packaging for (Linux) distributions and live image
31 stars 37 forks source link

Support Kotlin out of the box OR guide to add it manually #156

Closed agcom closed 6 months ago

agcom commented 9 months ago

I was wondering, as Kotlin is included in the worlds final programming environment (see https://docs.icpc.global/worldfinals-programming-environment/), shouldn't Kotlin be supported out of the box?

The Judgehost lacks kotlinc, kotlin-stdlib.jar, and other stuff required by the compiler; I have gone through the following steps to manually provide those inside the chroot environment, and make Kotlin work:

Now, that's just a hack applied to a single Judgehost container, and cumbersome to reproduce (although it is possible to make an image out of it via sudo docker commit command); I would like some guidance to ease the process, and ultimately apply these in the Judgehost Docker image build steps.

vmcj commented 9 months ago

I was wondering, as Kotlin is included in the worlds final programming environment (see https://docs.icpc.global/worldfinals-programming-environment/), shouldn't Kotlin be supported out of the box?

DOMjudge is besides a system used in the ICPC World finals also something on its own. So we don't always have the same content in the default installation which is used in the finals. Because kotlin is not in the default repo's (for ubuntu 22.04 there is a version though) we prefer to not have it in the chroot to keep this simple. We can help to reproduce a setup like the finals though.

The Judgehost lacks kotlinc, kotlin-stdlib.jar, and other stuff required by the compiler; I have gone through the following steps to manually provide those inside the chroot environment, and make Kotlin work:

* Run official DOMjudge and Judgehost Docker containers beside MariaDB, with standard setup: `sudo docker compose up`.

* Run an interactive shell inside the Judgehost container: `sudo docker compose exec judgehost bash`.

* Go inside the chroot environment, and switch to bash for convenience; `dj_run_chroot`; `bash`.

* Install SDKMAN; `apt install curl zip unzip`; `curl -s "https://get.sdkman.io" | bash`; `source "/root/.sdkman/bin/sdkman-init.sh"`.

* Install Kotlin by SDKMAN; `sdk install kotlin 1.7.21`.

* Copy Kotlin installation to `/usr/lib/kotlin/`; `cp -r /root/.sdkman/candidates/kotlin/1.7.21/ /usr/lib/kotlin/`.

* Go into DOMjudge admin web interface and enable Kotlin language; also, modify the Kotlin compile script (`run.sh`) and set the `KOTLIN_DIR` variable to `"/usr/lib/kotlin/bin"`, and save.

* Done.

Now, that's just a hack applied to a single Judgehost container, and cumbersome to reproduce (although it is possible to make an image out of it via sudo docker commit command); I would like some guidance to ease the process, and ultimately apply these in the Judgehost Docker image build steps.

We don't use the docker images that often, but we do use the kotlinc deb used at the finals. And install that in the chroot with the -i flag, that makes this a lot easier.

* How to avoid the requirement to modify the Kotlin compile script (`run.sh`)? In other words, how to add `/usr/lib/kotlin/bin` to the `PATH` variable inside the chroot environment?

There is no clean way of doing this, but you could either create a symlink in a location which is in the PATH or change the profile to include this in the PATH both are non standard changes. We prefer to stick to the FHS guide in this case to keep it easy to change if you want a custom config. Personally I would change the run.sh but all would be valid choices.

* Is incorporating the above steps into the `chroot-and-tar.sh` file enough to build a Judgehost Docker image with Kotlin stuff (I will try it soon and inform here, if not late)?

As far as I remember yes, you should be able to fork the repo and use the script with the changes you propose and get a container in the way how you want it. The version of kotlin might be a bit different from the finals though.

  • What are the security concerns?

Any package you add extra is a concern, we have to select a set and picked the most popular languages at that point. I'll discuss internally if we should make a chroot based on the config of an ICPC contest.

nickygerritsen commented 9 months ago

Note that besides Docker we also have native installs, so IF we somehow make Kotlin supported by default we should have it in both, ideally by modifying the script to create a chroot.

As @vmcj already said at ICPC contests we normally use a .deb for kotlin that is packaged by the ICPC people. We can't publicly share this .deb though since it's not supposed to be used outside of ICPC contests.

The PR you created will help by the way to use Kotlin, so thanks for that!

agcom commented 9 months ago

Kotlin compiler builds can be found in their GitHub releases' assets (such as JetBrains/Kotlin/v1.7.21); building a deb out of it is an easy task.

Or instead, after the first step, craft a deb out of it, and pass it to dj_make_chroot.

I couldn't find any script to create chroot for native installs; is there any? Where?

nickygerritsen commented 9 months ago

dj_make_chroot is that script

agcom commented 9 months ago

I added Kotlin compiler install steps to the chroot-and-tar.sh script in my fork (see commit https://github.com/agcom/domjudge-packaging/commit/4ef6678cf745729761abeb947cb2ef103a6c9860), and it works.

Unrelated to Kotlin compiler, I hit the following two errors in the build/test process:

I use community Docker engine version 24.0.6, and passed version 8.2.1 to the /docker/build.sh script.

Should I create separate issues for them? or is it just my machine?

nickygerritsen commented 9 months ago

I added Kotlin compiler install steps to the chroot-and-tar.sh script in my fork (see commit agcom@4ef6678), and it works.

Unrelated to Kotlin compiler, I hit the following two errors in the build/test process:

  • dj_make_chroot script fails (called inside the chroot-and-tar.sh script) with a permission denied error; see commit agcom@be9dfb3 for more description and an applied solution; the failure won't cause the image build to stop though.

Yeah sure but the solution is fine, I think.

  • The newly built domjudge/domserver image/container fails to startup with the following error: 
    /scripts/start.d/10-timezone.sh: 11: cannot create /etc/php/7.?//fpm/conf.d/99-timezone.ini: Directory nonexistent
domjudge-packaging-fork-domserver-1  | [!!] Start script 10-timezone.sh failed

I think we need to use 8.? now since we have a newer PHP.

I use community Docker engine version 24.0.6, and passed version 8.2.1 to the /docker/build.sh script.

Should I create separate issues for them?

Sure!

agcom commented 7 months ago

I'll discuss internally if we should make a chroot based on the config of an ICPC contest.

Any updates @vmcj?

If Kotlin is not going to be added to DOMjudge by default, I am willing to write a document on how to add Kotlin to various installations (all kinda share the same steps).

vmcj commented 7 months ago

I'll discuss internally if we should make a chroot based on the config of an ICPC contest.

Any updates @vmcj?

If Kotlin is not going to be added to DOMjudge by default, I am willing to write a document on how to add Kotlin to various installations (all kinda share the same steps).

Yes, I have a local branch somewhere to install the older kotlin but it requires a bit more work.

Feel free to write something up, I think the wiki would be fine for this.

agcom commented 6 months ago

I wrote something up and I was wondering how should I contribute it to the DOMjudge wiki? Should I just create a PR like this one https://github.com/DOMjudge/domjudge/pull/969?

vmcj commented 6 months ago

Posting it here is fine.

agcom commented 6 months ago

Adding a new language

DOMjudge has a default set of languages but it is possible to add your own.

Every submission is ran in an isolated container so the best way to test a new language is to first get the new language installed in the container via the options of dj_make_chroot; start this container with dj_run_chroot and test if you can compile and run some default programs to verify that everything can work; when this works, start to build the run/build scripts by looking at similar languages which already exist in DOMjudge and create the language with you own build/run scripts.

Default DOMjudge will mount only a limited set of directories and those does not include the /tmp/ directory which is used by R for example; therefore, in case of such failures, check if the tmp directory can be set with an environment variable, or extend the chroot-startstop.sh, or look into CREATE_WRITABLE_TEMP_DIR defined here. Be aware that providing writable directories opens up a security hole where the submission may write data into and pass information from one test-case-run to the next.

Kotlin

Kotlin is an officially supported programming language in the programming environment of the ICPC world finals.

To add the Kotlin programming language to a DOMjudge installation, first, the Kotlin command-line compiler should be installed in the chroot environment of the judgehosts (see Installation of the judgehosts - creating a chroot environment | DOMjudge documentation).

Installing Kotlin on existing judgehost

If there is an already running instance of judgehost, access its terminal and unzip the Kotlin compiler in its chroot environment directory (usually /chroot/domjudge/); optionally, for automatic Kotlin compiler discovery in the Kotlin run script, include the extracted bin/kotlinc in the PATH variable of the chroot environment (can be done by creating a symbolic link to the extracted bin/kotlinc inside a directory already included in the PATH); TL;DR, for example:

# A judgehost terminal

CHROOTDIR="/chroot/domjudge"
KOTLIN_VERSION="1.7.21"

# Download the Kotlin compiler zip.
wget -q "https://github.com/JetBrains/kotlin/releases/download/v$KOTLIN_VERSION/kotlin-compiler-$KOTLIN_VERSION.zip"

# Unzip the Kotlin compiler inside the chroot environment.
unzip -qq -d "$CHROOTDIR/usr/local/lib/" "./kotlin-compiler-$KOTLIN_VERSION.zip"

# Include bin/kotlinc in the PATH variable of the chroot environment by creating a symbolic link.
chroot "$CHROOTDIR/" ln -s "/usr/local/lib/kotlinc/bin/kotlinc" "/usr/local/bin/kotlinc"

# Optionally as a cleanup step, remove the downloaded zip file.
rm "./kotlin-compiler-$KOTLIN_VERSION.zip"

Installing Kotlin by editing judgehost install scripts

You can also modify the judgehost install scripts to add Kotlin; simply edit the misc_tools/dj_make_chroot.in script (this raw script can be found a release tarball or git sources) and add something like the above in an appropriate place; for example, after installing debs (this line), add:

...

# Install Kotlin
KOTLIN_VERSION="1.7.21"
wget -q "https://github.com/JetBrains/kotlin/releases/download/v$KOTLIN_VERSION/kotlin-compiler-$KOTLIN_VERSION.zip"
unzip -qq -d "$CHROOTDIR/usr/local/lib/" "./kotlin-compiler-$KOTLIN_VERSION.zip"
in_chroot "ln -s \"/usr/local/lib/kotlinc/bin/kotlinc\" \"/usr/local/bin/kotlinc\""
rm "./kotlin-compiler-$KOTLIN_VERSION.zip"

...

Judgehost Docker image with Kotlin

If you want a judgehost Docker image with Kotlin, modify the docker/judgehost/chroot-and-tar.sh script in the DOMjudge packaging repository, such as:

#!/bin/bash

# Usage: https://github.com/DOMjudge/domjudge/blob/main/misc-tools/dj_make_chroot.in#L58-L87
/opt/domjudge/judgehost/bin/dj_make_chroot

CHROOTDIR="/chroot/domjudge"
KOTLIN_VERSION="1.7.21"

echo "Downloading Kotlin compiler"
wget -q "https://github.com/JetBrains/kotlin/releases/download/v$KOTLIN_VERSION/kotlin-compiler-$KOTLIN_VERSION.zip" -P /
echo "Extracting Kotlin compiler"
unzip -qq -d "/chroot/domjudge/usr/local/lib/" "/kotlin-compiler-$KOTLIN_VERSION.zip"
chroot "$CHROOTDIR" ln -s "/usr/local/lib/kotlinc/bin/kotlinc" "/usr/local/bin/kotlinc"
rm "/kotlin-compiler-$KOTLIN_VERSION.zip"
echo "Done setting up Kotlin"

cd /
echo "[..] Compressing chroot"
tar -czpf /chroot.tar.gz --exclude=/chroot/tmp --exclude=/chroot/proc --exclude=/chroot/sys --exclude=/chroot/mnt --exclude=/chroot/media --exclude=/chroot/dev --one-file-system /chroot
echo "[..] Compressing judge"
tar -czpf /judgehost.tar.gz /opt/domjudge/judgehost

Then run docker/build.sh to build judgehost and domserver images, or docker/build-judgehost.sh after manually downloading the DOMjudge source (first steps of the docker/build.sh), to build only the judgehost image:

# In the DOMjudge packaging repository root.

cd docker/
DOMJUDGE_VERSION="8.2.2"
wget --quiet "https://www.domjudge.org/releases/domjudge-${DOMJUDGE_VERSION}.tar.gz" -O "./domjudge.tar.gz"
./build-judgehost.sh "localhost/domjudge/judgehost:${DOMJUDGE_VERSION}-kotlin"

Enabling Kotlin in the admin dashboard

After successfully installing the Kotlin command-line compiler on judgehosts, Kotlin language should be enabled in the DOMjudge Jury interface for submissions:

A heads-up about the default kt run script: if the DOMjudge version is in range 8.2.0 to 8.2.2 (both inclusive), the automatic Kotlin compiler directory discovery might fail if kotlinc is a symbolic link; one solution is to manually fill-in the KOTLIN_DIR variable in the run script; another solution is to update the run script to its latest commit.

The run script can be edited in the DOMjudge Jury interface: "Languages" - kt - "Compile script / kt" - Content / run.

vmcj commented 6 months ago

@agcom I'll close this issue, I've added your text to the wiki and we'll discuss on the next hackathon what to do in the future with kotlin in the judgehost.

agcom commented 1 month ago

Adding a new language

DOMjudge has a default set of languages but it is possible to add your own.

Every submission is ran in an isolated container so the best way to test a new language is to first get the new language installed in the container via the options of dj_make_chroot; start this container with dj_run_chroot and test if you can compile and run some default programs to verify that everything can work; when this works, start to build the run/build scripts by looking at similar languages which already exist in DOMjudge and create the language with you own build/run scripts.

Default DOMjudge will mount only a limited set of directories and those does not include the /tmp/ directory which is used by R for example; therefore, in case of such failures, check if the tmp directory can be set with an environment variable, or extend the chroot-startstop.sh, or look into CREATE_WRITABLE_TEMP_DIR defined here. Be aware that providing writable directories opens up a security hole where the submission may write data into and pass information from one test-case-run to the next.

Kotlin

...

It just came into my attention 👀, I also revamped the prologue :point_up:, but it wasn't added to the wiki; I thought maybe you haven't noticed it, therefore, commenting.