How to use the project

[irevoire]

Hey, I'm struggling to understand how to use the project; I didn't find any documentation or where the integration tests are located.

Is there an example showing what is required to create a new benchmark and what info should I send to add new points?

• The commit sha?
• The date on which it was run?
• The data of all the runs I did?
• And more important, what is the format for all of that?

I found this route in the code but tbh it didn't help me much :sweat_smile:

// curl -X PUT -H "Content-Type: application/json" -d '{"foo":"bar","baz":3}'
//  http://localhost:33333/rebenchdb/results
// DEBUG: koaBody({includeUnparsed: true})
router.put('/rebenchdb/results', koaBody({ jsonLimit: '500mb' }), async (ctx) =>
  acceptResultData(ctx, db)
);

[smarr]

Sorry, the documentation for that is very much lacking, yes.

The tests are here: https://github.com/smarr/ReBenchDB/blob/master/tests/backend/rebench/api.test.ts

Most useful might be the example payload files here:

https://github.com/smarr/ReBenchDB/tree/master/tests/data

specifically small-payload.json and large-payload.json.bz2.

To get an idea of the structure, the type of the data going into the API is BenchmarkData: https://github.com/smarr/ReBenchDB/blob/master/src/shared/api.ts#L121

The only known client to use it is unfortunately https://github.com/smarr/ReBench/ at this point.

Does this give you enough to go on as a start?

[irevoire]

Yeah, that seems really nice; I'll try to look into it in one hour after my meeting!

Also, I forgot to ask, but are there some dependencies I need to install? I've seen that you were using PostgreSQL. Is there anything else I should know? Or even better, is there a script or docker-compose that spawns everything correctly configured for me?

[smarr]

In theory, the Docker image starts the Postgres server and the Node.js. I haven't yet had enough spare time to look into docker-compose to make that all less hacky...

When I say in theory, I mean the Docker image based on Dockerfile.rebench does actually work for our own performance tracking/benchmarking of ReBenchDB.

Though, I used it on podman instead of Docker, and I don't use it to run the rebench.dev instance.

[irevoire]

Huum FYI, the docker image doesn't seem to start the Postgres server (or there is a conf issue):

2024-01-18 15:47:17.598 ERROR /src/index.ts:173 index Unable to connect to database on port 127.0.0.1:5432

I'll try to make it work without docker and keep you updated / fix the docker file if I understand what is wrong

[smarr]

Hm :(

I wonder whether there's a difference in how podman and Docker treat these things. I believe @sophie-kaleba also didn't get the Docker image working.

[irevoire]

Honestly, I don't have any issue with using Podman though, I just want to get the easiest thing that works.

But I don't know which Containerfile you used since there is none in the repo I believe? If I use the dockerfile then podman complains that it doesn't know the specified image; Error: creating build container: short-name "postgres:16-bookworm" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"

[smarr]

The commands, as per our benchmark setup are in .gitlab-ci.yml.

Specifically this:

podman build . -f Dockerfile -t rebenchdb
podman build . -f Dockerfile.rebench -t bench-rdb
podman run --hostname yuria2-podman bench-rdb:latest -c --experiment="CI ID $CI_PIPELINE_ID" --branch="$CI_COMMIT_REF_NAME" rebench.conf

So, the first command should builds the normal image, the second command builds the image for benchmarking, and the last one then runs the thing executing the benchmarks.

And between the two of us, this is the amount of insight I have into docker and podman. The reason to use podman was mostly to avoid requiring root access.

[irevoire]

The reason to use podman was mostly to avoid requiring root access.

Yeep I like that, I'll try to use it as well then

And between the two of us, this is the amount of insight I have into docker and podman.

Same and I just don't get why it works in your CI and not on my computer when it's the very first principle of podman and docker :sob:

% podman build . -f Dockerfile -t rebenchdb
STEP 1/21: FROM postgres:16-bookworm
Error: creating build container: short-name "postgres:16-bookworm" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"

But after some googling it looks like you'll have a bug soon in your CI. You should first run % podman pull docker.io/postgres:16-bookworm which worked on my side.

[irevoire]

Ok, I was happy a little bit too fast because even though the image now builds, it doesn't work. When I run it, nothing happens, and there are no log/errors.

In a nutshell, here's what is what happens when I try to run the image:

irevoire@arch ~/ReBenchDB 🐆 % podman run -it rebenchdb
irevoire@arch ~/ReBenchDB 📛 % podman ps -a
CONTAINER ID  IMAGE                       COMMAND               CREATED        STATUS                    PORTS       NAMES
062ed330dc4d  localhost/rebenchdb:latest  bash ./start-serv...  4 seconds ago  Exited (0) 5 seconds ago              exciting_lumiere
irevoire@arch ~/ReBenchDB 🦅 % podman logs exciting_lumiere
irevoire@arch ~/ReBenchDB 🍰 %

I don't get why because it's pretty clear from your second (rebench) docker file that you don't specify any argument or anything for it to run :thinking:

Any idea why, am I supposed to do something?

[smarr]

In an attempt to "do it the right way", I put a docker-compose version together here: https://github.com/smarr/ReBenchDB/tree/compose

Works on my machine with podman-compose up. I'd hope docker-compose up works the same way.

So far untested other than "on my machine".

[irevoire]

Hey, that's awesome!

The docker-compose worked on mine, but the podman-compose didn't (I think it's linked to the % podman pull docker.io/postgres:16-bookworm command missing).