search

scheme-containers / monorepo

Generators, website, issues
MIT License
6 stars 1 forks source link

Running ecraven's benchmarks #11

Open amirouche opened 4 years ago

amirouche commented 4 years ago

What about porting ecraven's benchmarks and run them using the containers found in this project so that it is easy for anyone to reproduce the results?

lassik commented 4 years ago

Sounds excellent. That benchmark suite is impressive but haven't had time to try and run it. Do you know how to hook it up to our containers?

amirouche commented 4 years ago

It seems straight forward to run the tests with docker. I will look into it.

amirouche commented 4 years ago

ref: https://docs.docker.com/storage/bind-mounts/#start-a-container-with-a-bind-mount

amirouche commented 4 years ago

After cloning the benchmark repository, change directory inside it then:

% docker run -it --mount type=bind,source="$(pwd)",target=/benchmark schemers/chibi bash
$ cd /benchmark/
$ ./bench chibi all

Testing browse under Chibi
Including prelude /benchmark/src/Chibi-prelude.scm
Including postlude /benchmark/src/Chibi-postlude.scm
Compiling...
chibi_comp /tmp/larcenous/Chibi/browse.scm /tmp/larcenous/Chibi/browse.scm
Running...
WARNING: importing undefined variable: chibi-version
Running browse:2000

...
amirouche commented 4 years ago

That is the idea. I tried chicken first, but it seems like the docker container is missing the command chicken-csc or chicken5 has changed its command line. Anyway, I will further investigate.

lassik commented 4 years ago

it seems like the docker container is missing the command chicken-csc

Almost all of our containers should install the Scheme implementation under /usr/local. Try looking in /usr/local/bin. The Chicken compiler may be simply csc.

amirouche commented 4 years ago

The Chicken compiler may be simply csc.

That is exactly the case.

lassik commented 4 years ago

Bind mounts are one way, or you can do git clone --depth 1 or wget/curl https://github.com/ecraven/r7rs-benchmarks/archive/master.tar.gz inside the container. Bind mounts are find for local development, but may not be easy to use reliably in production.

Most of the containers are based on debian, so apt-get update && apt-get install -y curl works to install packages.

amirouche commented 4 years ago

Based on my investigations, we need to fork the benchmarks to adapt it.

What do you think @ecraven?

ecraven commented 4 years ago

Based on my investigations, we need to fork the benchmarks to adapt it. What do you think @ecraven?

I've been planning to containerise things for a long time, but haven't had the time. I've also started work on a chibi scheme test runner, that correctly captures compilation time, stdout, stderr, and so on, but that isn't close to finishing either. I won't be able to finish these projects anytime soon, so feel free to give it a go.

One thing to keep in mind is that containers do have a performance hit, so there will be people who think that benchmarks run inside a container are worse than those run on bare metal ;)

Greetings, Peter