deploying latest versions of KD and KER v=1.1.3 throws an error: exec /opt/java/openjdk/bin/java: exec format error #412

Closed Sophietje closed 7 months ago

In GitLab by @andraz.andolsek.cyber-grid.com on Mar 8, 2023, 10:55

@luka.zeleznik.sunesis.si FYI

In GitLab by @andraz.andolsek.cyber-grid.com on Mar 8, 2023, 10:59

@barry.nouwt.tno.nl i assume this is related to the platform version. We have deployed this to kubernetes where ubunutu is running on the amd64.

In GitLab by @han.kruiger.tno.nl on Mar 9, 2023, 09:00

This looks like an issue with the Docker image platform/architecture. The images should be multi-platform for a while (I build them on an ARM machine, but they are built for other architectures as well).

@barry.nouwt.tno.nl Can you test the Docker Compose project on your machine, e.g. the one in ./examples/multiple-runtimes?

In GitLab by @barry.nouwt.tno.nl on Mar 9, 2023, 13:06

Hey all,

I've tested the docker-compose project on my Ubuntu Windows Subsystem Linux (WSL) on my amd64 architecture as @han.kruiger.tno.nl suggests and it starts fine. These use the multi-platform ghcr.io docker images version 1.1.3. I also tested both the the multi-platform docker images on gitlab.inesctec.pt for the KE Runtime docker-registry.inesctec.pt/interconnect/knowledge-engine/smart-connector-rest-dist:1.1.3 and Knowledge Directory docker-registry.inesctec.pt/interconnect/knowledge-engine/knowledge-directory:1.1.3 and they also work correctly.

@andraz.andolsek.cyber-grid.com Can you confirm that you indeed use the latest 1.1.3 of the above images (maybe docker pull both explicitly again, because we might have overwritten the non-multi-platform 1.1.3 images with multi-platform ones).

If this does not solve the issue, we should dive a bit deeper to find out what might be causing this. Is there any more information that you can provide?

Best regards,

Barry

In GitLab by @andraz.andolsek.cyber-grid.com on Mar 9, 2023, 21:00

Hi @barry.nouwt.tno.nl ,

I just pulled and deployed. The result is the same:

exec /opt/java/openjdk/bin/java: exec format error

But I can confirm it works, if I run it on my computer, but doesn’t in the AWS EC2 insance.

In GitLab by @andraz.andolsek.cyber-grid.com on Mar 10, 2023, 06:09

@barry.nouwt.tno.nl i wanted to build for my self, but i received the following error:


knowledge-engine git:(master) ✗ docker buildx build ./knowledge-directory --platform linux/amd64 --tag ghcr.io/cyber-grid/knowladge-directory:1.1.3-v2
[+] Building 0.8s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 32B                                                                                                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/adoptopenjdk/openjdk11:alpine-jre                                                                                                                                     0.7s
 => [internal] load build context                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                  0.0s
 => CACHED [1/2] FROM docker.io/adoptopenjdk/openjdk11:alpine-jre@sha256:78d7acff2cac27104a15a563d5a1feb0e1f219d2e545a2af9bd7144ae9186252                                                                        0.0s
 => ERROR [2/2] COPY target/*-with-dependencies.jar /knowledge_directory/knowledge-directory.jar                                                                                                                 0.0s
------
 > [2/2] COPY target/*-with-dependencies.jar /knowledge_directory/knowledge-directory.jar:
------
ERROR: failed to solve: lstat /var/lib/docker/tmp/buildkit-mount2508403150/target: no such file or directory

BTW: i run the mvn clean complie beforehand.

In GitLab by @barry.nouwt.tno.nl on Mar 10, 2023, 08:21

Hey @andraz.andolsek.cyber-grid.com, thanks for the update.

You should use mvn clean package instead of mvn clean compile to actually package the compiled classes.

In GitLab by @han.kruiger.tno.nl on Mar 10, 2023, 08:30

I would recommend adding -DskipTests to prevent waiting for multiple minutes for the tests, so:

mvn clean package -DskipTests

In GitLab by @han.kruiger.tno.nl on Mar 10, 2023, 08:35

But yes, I tested this (on the commit tagged with 1.1.3) and it builds the image correctly on my machine:

mvn clean package -DskipTests
docker buildx build ./knowledge-directory --platform linux/amd64 --tag test-for-andraz --load
docker run --platform linux/amd64 test-for-andraz

@andraz.andolsek.cyber-grid.com can you try that?

In GitLab by @barry.nouwt.tno.nl on Mar 10, 2023, 08:38

@han.kruiger.tno.nl, @andraz.andolsek.cyber-grid.com I also tested it and the above works for me.

In GitLab by @barry.nouwt.tno.nl on Mar 10, 2023, 08:55

@andraz.andolsek.cyber-grid.com Can you compare the docker versions on your local computer and the AWS EC2 instance?

Maybe it has something to do with older docker versions not supporting multi-platform images?

In GitLab by @andraz.andolsek.cyber-grid.com on Mar 10, 2023, 10:36

@barry.nouwt.tno.nl good news, now i managed to build my own containers. Deployment now works. Thank you guys. i would suggest building multiple images for each platform when releasing.

In GitLab by @han.kruiger.tno.nl on Mar 10, 2023, 10:41

Great to read!

I'm still not sure what went wrong with the images, because they are supposed to be multi-platform, so they should work on both ARM and x86.

If you have the time, can you help us debug the issue further by giving:

The full tag of the image that failed to run
The Docker version you used on the machine that could not run the image
- (because it could be the case that multi-platform images don't work on older versions?)

Thanks a bunch!

In GitLab by @andraz.andolsek.cyber-grid.com on Mar 10, 2023, 10:57

@han.kruiger.tno.nl

I used this one: Screenshot_2023-03-10_at_11.55.11

EC2 instance is the amd64, we run in the Kubernetes that has
k8s uses containers, not Docker containerd://1.6.6

In GitLab by @andraz.andolsek.cyber-grid.com on Mar 10, 2023, 11:06

BTW, we were connected with the PT pilot, but due to the different version of KER we were using the connection didn't work. Now we both are running 1.1.3 and it started working. So i think there were some breaking changes.

In GitLab by @barry.nouwt.tno.nl on Mar 10, 2023, 11:15

Which version of the KER did you use previously which did not work with version 1.1.3 of the PT pilot?

In GitLab by @andraz.andolsek.cyber-grid.com on Mar 10, 2023, 11:23

1.1.2

In GitLab by @barry.nouwt.tno.nl on Mar 24, 2023, 09:39

I did a quick test to see if I can reproduce these breaking changes. I started a Knowledge Directory version 1.1.3. Subsequently, I ran a KeRuntime version 1.1.2 with a KB that asks and another KeRuntime version 1.1.3 with a KB that answers and this seems to work fine. I did use the Java Developer API instead of the REST Developer API.

In GitLab by @barry.nouwt.tno.nl on Mar 24, 2023, 09:40

I close this issue, please reopen it if you think it should not be closed.

TNO / knowledge-engine

deploying latest versions of KD and KER v=1.1.3 throws an error: exec /opt/java/openjdk/bin/java: exec format error #412