Open andreas-eberle opened 4 years ago
Are you talking about the Quarkus build or the projects we generate?
@andreas-eberle ping?
Oh, sorry, didn't see your answer.
I was talking about an example for how to build a gitlab-ci.yml for my projects I build with Quarkus. So for a user of Quarkus. It would be especially interesting to know how to make a native build with the docker-build in gitlab.
So we could include that in our project template (and probably something for GH actions too) but we would need someone to handle that and maintain it.
Do you volunteer? :)
Unfortunately, I'm not deep into gitlab. That's why I'm asking for the example.
@andreas-eberle ; hopefully this helps:
install-graal.sh
#!/usr/bin/env bash
echo "Downloading GraalVM"
wget -q https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-19.3.1/graalvm-ce-java8-linux-amd64-19.3.1.tar.gz
tar zxf graalvm-ce-java8-linux-amd64-19.3.1.tar.gz
echo "Installing GraalVM via gu"
${CI_PROJECT_DIR}/graalvm-ce-java8-19.3.1/bin/gu install native-image
.gitlab-ci.yml
image: maven:latest
cache:
paths:
- .m2/repository/
before_script:
- apt-get update -qq
- apt-get install -y -qq build-essential libz-dev zlib1g-dev
build:
stage: build
script:
- chmod +x install-graal.sh && ./install-graal.sh jdk8
- export GRAALVM_HOME=${CI_PROJECT_DIR}/graalvm-ce-java8-19.3.1
- mvn clean install -P native
@gsmet ; happy to raise a PR if you can suggest where it should live.
@garethahealy: I was actually able to use the image: oracle/graalvm-ce:19.3.1-java11
, which has graalvm installed and configured.
Unfortunately, I still haven't found an good way to run the native build as dockerbuild inside gitlab. However, this might also have to do with our Gitlab instance making this a bit tricky. So I cannot really judge that.
AFAIK for Docker build you need "dind", and then you can either run CI on Docker image or use whatever you want and use job-specific Docker image just for the build job.
Another option would be to use JIB, but I cannot figure out how to configure credential properly - it always ends up with 403. I came here because I was wondering did anybody ask for CI guide for jib.
๐hello - about Docker, you don't need "dind", you can use Kaniko: https://docs.gitlab.com/ee/ci/docker/using_kaniko.html
Currently, I am using JIB without neither; it works faster than "dind" and does not require an image with Docker CLI.
quarkus:
container-image:
group: mycompany/myproject
registry: ${CI_REGISTRY}
username: ${CI_REGISTRY_USER}
password: ${CI_REGISTRY_PASSWORD}
@gsmet This is listed as a good first issue, I could give this a shot, maybe add the configuration template to the repo
@sladyn98 sure!
As suggested earlier
image: maven:latest
cache:
paths:
- .m2/repository/
before_script:
- apt-get update -qq
- apt-get install -y -qq build-essential libz-dev zlib1g-dev
build:
stage: build
script:
- chmod +x install-graal.sh && ./install-graal.sh jdk8
- export GRAALVM_HOME=${CI_PROJECT_DIR}/graalvm-ce-java8-19.3.1
- mvn clean install -P native
This could go in the root of the project as gitlabci.yml
I tried the 2 scripts above, unfortunately it doesn't work for me as I got:
/usr/bin/bash: line 99: apt-get: command not found
ERROR: Job failed: exit code 1
Anybody has an update on how to create a pipeline for Quarkus project in GitLab?
@czetsuya what image did you use? i had: maven:latest
@garethahealy when I tried to commit that in GitLab I got:
$ apt-get update -qq
/usr/bin/bash: line 99: apt-get: command not found
I tried using maven:3.6.3-openjdk-11-slim version and apt-get is recognized but I got:
./install-graal.sh: line 4: wget: command not found
After updating both .gitlab-ci.yaml and install-graal.sh, here's the version that works for me:
https://www.czetsuyatech.com/2021/07/javaee-quarkus-on-gitlab.html
@czetsuya can you please update the link to your solution? I have the exact same problem
@ledider updated.
Hi, i saw this discussion and like @k33g said, you can use Kaniko to deploy your image.
Based on the example of @czetsuya, I added deployment and this is available here. This gitlabci file allows to package a project in jvm mode & native mode and use Kaniko to make one image for each mode.
You could provide this as a codestart like we do for github-action: https://github.com/quarkusio/quarkus/tree/main/independent-projects/tools/base-codestarts/src/main/resources/codestarts/quarkus/tooling/github-action
For now the github-action codestart is only used by code.quarkus.io, but we could make that new options in the Quarkus CLI/Maven plugin too
Thanks @ia3andy for this response. I will try to initialize it this week, otherwise it will be at the end of august.
FWIW, I have worked on this and I just want to share:
native image:
stage: build
image: quay.io/quarkus/ubi-quarkus-mandrel:21.2.0.0-Final-java11
variables:
MVNW_VERBOSE: 'true'
script:
- ./mvnw package -Pnative
artifacts:
paths:
- "target/*-runner"
expire_in: 2 days
tags:
- 'xp'
container image:
stage: build
needs:
- job: 'native image'
artifacts: true
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
- mkdir -p /kaniko/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}},\"proxies\":{\"default\":{\"httpProxy\":\"$HTTP_PROXY\",\"httpsProxy\":\"$HTTPS_PROXY\",\"noProxy\":\"$NO_PROXY\"}}}" > /kaniko/.docker/config.json
- wget --proxy off http://corp.org/ca.crt -O - >> /kaniko/ssl/certs/ca-certificates.crt
- "/kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/src/main/docker/Dockerfile.native-distroless --destination $CI_REGISTRY_IMAGE:latest --build-arg HTTP_PROXY=$HTTP_PROXY --build-arg HTTPS_PROXY=$HTTPS_PROXY --build-arg NO_PROXY=$NO_PROXY --build-arg http_proxy=$HTTP_PROXY --build-arg https_proxy=$HTTPS_PROXY --build-arg no_proxy=$NO_PROXY"
There are some gotchas:
Hope this helps...
Thank you for sharing you GitLab-CI.yaml @xasx . It is very helpful. However, when executing the native image job I get the error
Downloading artifacts 00:03
Downloading artifacts for build-application (1621169689)...
Downloading artifacts from coordinator... ok id=1621169689 responseStatus=200 OK token=ndyW1ExL
Executing "step_script" stage of the job script 00:03
Using docker image sha256:f09a86b09374d5324f8cd8b31ccb731962d8ed7b34253e60fcf850f516e78c00 for quay.io/quarkus/ubi-quarkus-mandrel:21.2-java11 with digest quay.io/quarkus/ubi-quarkus-mandrel@sha256:279cd78dde367ba989a6121c762b29aacc4b605169a940e5e7e98c4cb3197e4f ...
Error: Unrecognized option: -c
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1
Do you have any advice how to fix that?
@kvnb93 looks like there's a -c
somewhere it does not belong
Error: Unrecognized option: -c
maybe disclose your .gitlab-ci.yml
should you not find it
Thanks for your quick reply @xasx. I didn't find any -c in my .gitlab-ci.yaml. This is the complete file:
variables:
# This will suppress any download for dependencies and plugins or upload messages which would clutter the console log.
# `showDateTime` will show the passed time in milliseconds. You need to specify `--batch-mode` to make this work.
MAVEN_OPTS: "-Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
# As of Maven 3.3.0 instead of this you may define these options in `.mvn/maven.config` so the same config is used
# when running from the command line.
# `installAtEnd` and `deployAtEnd` are only effective with recent version of the corresponding plugins.
MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"
image: maven:3.6-openjdk-11
cache:
key: "$CI_JOB_NAME"
paths:
- .m2/repository
stages:
- test
- build
- build-native
test:
stage: test
script:
- mvn $MAVEN_CLI_OPTS test
tags:
- netcup
build-application:
stage: build
script:
- mvn $MAVEN_CLI_OPTS package
artifacts:
paths:
- "target/quarkus-app/lib/"
- "target/quarkus-app/*.jar"
- "target/quarkus-app/app/"
- "target/quarkus-app/quarkus/"
expire_in: 2 days
tags:
- netcup
build-jvm-docker-image:
stage: build
needs:
- job: build-application
artifacts: true
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
- mkdir -p /kaniko/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/src/main/docker/Dockerfile.jvm --destination $CI_REGISTRY_IMAGE
tags:
- netcup
build-native-application:
stage: build-native
image: quay.io/quarkus/ubi-quarkus-mandrel:21.2-java11
script:
- ./mvnw $MAVEN_CLI_OPTS package -Pnative
artifacts:
paths:
- "target/*-runner"
expire_in: 2 days
only:
- master
build-native-docker-image:
timeout: 3 hours 30 minutes
stage: build-native
needs:
- job: build-native-application
artifacts: true
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
- mkdir -p /kaniko/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/src/main/docker/Dockerfile.native-distroless --destination $CI_REGISTRY_IMAGE
tags:
- netcup
only:
- master
Hi, maybe in your Dockerfile ?
It is not in the Dockerfile, either. I did not even modify the Dockerfile, I just use the one generated by Quarkus.
What I did modify, however, is the Dockerfile for the JVM (Dockerfile.jvm
) to add the user 1001 to the docker image as it is expected by Kaniko. I paste it here for reference if anyone wants to run my .gitlab-ci.yaml file.
FROM registry.access.redhat.com/ubi8/ubi-minimal:8.4
ARG JAVA_PACKAGE=java-11-openjdk-headless
ARG RUN_JAVA_VERSION=1.3.8
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en'
# Install java and the run-java script
# Also set up permissions for user `1001`
RUN microdnf install curl shadow-utils ca-certificates ${JAVA_PACKAGE} \
&& microdnf update \
&& microdnf clean all \
&& mkdir /deployments \
&& chown 1001 /deployments \
&& chmod "g+rwX" /deployments \
&& chown 1001:root /deployments \
&& curl https://repo1.maven.org/maven2/io/fabric8/run-java-sh/${RUN_JAVA_VERSION}/run-java-sh-${RUN_JAVA_VERSION}-sh.sh -o /deployments/run-java.sh \
&& chown 1001 /deployments/run-java.sh \
&& chmod 540 /deployments/run-java.sh \
&& echo "securerandom.source=file:/dev/urandom" >> /etc/alternatives/jre/conf/security/java.security
RUN useradd -u 1001 user
# Configure the JAVA_OPTIONS, you can add -XshowSettings:vm to also display the heap size.
ENV JAVA_OPTIONS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
# We make four distinct layers so if there are application changes the library layers can be re-used
COPY --chown=1001 target/quarkus-app/lib/ /deployments/lib/
COPY --chown=1001 target/quarkus-app/*.jar /deployments/
COPY --chown=1001 target/quarkus-app/app/ /deployments/app/
COPY --chown=1001 target/quarkus-app/quarkus/ /deployments/quarkus/
EXPOSE 8080
USER 1001
ENTRYPOINT [ "/deployments/run-java.sh" ]
for a demo, i remove this part :) without chown, build a kaniko image works ๐ but i think that's your problem come other thing...
The pipeline fails at the step build-native-application
, not in the JVM build step.
The failing step is run on a GitLab shared runner. Maybe it causes the problem.
You don't have to use tag in your projet. it's specific to each projet. Try without ๐
The failing step is run on a GitLab shared runner. Maybe it causes the problem.
That is what I suspect as well. Some kind of special configuration, I guess.
Hi, i'm starting to work on this issue.
Error: Unrecognized option: -c
+1
but maybe I'm doing it wrong:
relevant .gitlab-ci.yml step:
build_my-fancy-project-core:
tags:
- some-tag-required-to-run-on-the-customer-companys-gitlab-runner-on-aws-eu-central1
image: quay.io/quarkus/ubi-quarkus-mandrel:21.2-java11
stage: le_build_app_stage_name
script:
- cd my-fancy-project-core
- ./mvnw clean package -Dnative -Dmaven.repo.local=./.m2/repository
artifacts:
paths:
- my-fancy-project-core/target/
- my-fancy-project-core/.m2/
expire_in: 4 h
without -Dnative and image: maven:3-jdk-11 it works fine (but has slow cold starts on a lambda, of course)
Here is the config I use with gradle projects, it uses jib to create the container images and works with kubernetes executor without dind:
quarkus-native-build:
variables:
QUARKUS_PACKAGE_TYPE: native
QUARKUS_CONTAINER_IMAGE_BUILD: "true"
QUARKUS_CONTAINER_IMAGE_PUSH: "true"
QUARKUS_CONTAINER_IMAGE_BUILDER: jib
QUARKUS_CONTAINER_IMAGE_USERNAME: $CI_REGISTRY_USER
QUARKUS_CONTAINER_IMAGE_PASSWORD: $CI_REGISTRY_PASSWORD
image: quay.io/quarkus/centos-quarkus-maven:21.3-java11
script:
- ./gradlew build -Dquarkus.container-image.image=${CI_REGISTRY_IMAGE}:latest
Currently, I am using JIB without neither; it works faster than "dind" and does not require an image with Docker CLI.
quarkus: container-image: group: mycompany/myproject registry: ${CI_REGISTRY} username: ${CI_REGISTRY_USER} password: ${CI_REGISTRY_PASSWORD}
I am trying to achieve this myself. I am using a @QuarkusTest
which uses a @QuarkusTestResource
to spin up MySQL and Kafka containers and then I have @QuarkusIntegrationTest
which extends this class. Everything works perfectly locally when I executemvnw -Dquarkus.container-image.build=true clean verify
but the test/failsafe run fails inside Gitlab when it starts running the @QuarkusIntegrationTest
. (however, the @QuarkusTest works fine when the surefire plugin runs)
I can see that the docker image is built and pushed into my Gitlab registry but then the process blows up for some reason.
I don't get any logs from the test at all (I've inserted a log.info line in a @BeforeAll) and I am not sure what is going on. @zeljkot Any ideas what I might be doing wrong?
Using JIB or Kaniko to build docker images and publish to a registry without docker is a way to go.
But how do you just run a quarkus build when dev services are in use for tests and therefore a docker runtime become a requirement ?
Podman & podman-docker added to your base Quarkus build image ?
Edit: forget my question, just found this readme about it with all different images, especially one having everything (maven, graalvm, podman and buildah)
@nicolas-vivot , i have tried using the centos-quarkus-maven image without any luck either. does anyone have any suggestions on how to get this working inside Gitlab?
Hi @nicolas-vivot , according to your dev services, you can try to use GitLab Services.
@einarjohnson As i mentioned here, somebody need to add the podman-docker package to the image + a few configuration so that it works.
I don't have the time right now to do it so if any of you can submit a PR on that topic you're welcome.
@jeanphibaconnais That's an option, but a limited one. Quarkus automatically group tests running with same profile to run them on the same runtime of your application. This also applies to dev services (if i'm not wrong). This is very useful to separate tests that need a certain state in the database to performs and you need to make sure no other tests is altering it (making the test useless) Gitlab Service are external to such system, thus cannot cover the case.
@nicolas-vivot , can you explain further. in order to run @QuarkusIntegration tests inside Gitlab the base image running the tests needs to have podman installed?
@einarjohnson Yes, Podman but also Podman-docker, if you want to get rid of Docker itself (docker daemon)
I'll try to summarize it.
There are two things to distinguish: building an image from a Docker file without Docker, and running an image without Docker in a docker based environment like the Gitlab runners. We already explained & gave a few tools to build images from a Docker file without Docker, so let's jump into the second and more problematic case: running an image without Docker (e.g without accessing the Docker daemon)
First, why running without Docker: to avoid any security issue if you want to use Gitlab runners running on one of your kubernetes cluster. Gitlab forked Docker and are running their own Docker daemon version, so i would say that you're good to continue to use Docker in Docker if you use Gitlab shared runners, but definitely not if you intent to use an on prem gitlab runner running on a Kubernetes cluster.
Now, coming to Quarkus. In Quarkus, you can run @QuarkusTest using dev services to automatically pop a mongdb instance, or a mssql instance along with your Quarkus application to run the tests on. The Quarkus DevServices are using the TestContainer library to run these side car containers, expecting to have access to a local Docker daemon to run the images.
So usually on Gitlab, you would use Docker in Docker to give access to the Docker daemon running on the Gitlab runner to the container building your Quarkus app. In order to avoid the Docker in Docker when your Gitlab runner is executed inside a Kubernetes cluster, you have to go with another technology that is not relying on a daemon to run. This is what Podman is right. You can see podman as a rootless docker.
So one idea to be able to run the Quarkus Dev Services without any Docker is to use Podman as the container runtime. But to do that, you need kind of a bridge between what the Dev services are expecting (a docker socket to connect to) Podman-docker is another library that enable to make that link. Podman will then be able to take over & respond to any request made to a supposed docker socket.
Unfortunately, the current Quarkus image provided as tooling image only embed the GraalVM, Maven, Gradle, Podman and Buildah. This image is originally meant to build images for quarkus native apps and not to run it. (due to Dev Services relying only on Docker and not Podman - for now, maybe this can change in the future, but that may be a change on TestContainer itself, not the Dev Services)
By adding Podman-docker to that existing image, plus a few configuration as demonstrated in this blog, we could use that image not only to build images/containers but also to run it, which is a requirement when it comes to integration tests with Quarkus dev services. To make it clear, with that single image you could now build your maven project including running tests that are using Dev Services without having access to the Docker daemon since Podman would take over in place of Docker.
This would make it possible to build a Quarkus project including the integration tests on a gitlab runner anywhere, without having the access to a docker daemon.
Just adding another example here. I was having trouble tracking down an example that used maven + jib + native image + deployed to aws ecr.
cache:
key: mavencache
paths:
- ./.m2/repository
services:
- docker:dind
variables:
# Instruct Testcontainers to use the daemon of DinD.
DOCKER_HOST: "tcp://docker:2375"
# Instruct Docker not to start over TLS.
DOCKER_TLS_CERTDIR: ""
# Improve performance with overlayfs.
DOCKER_DRIVER: overlay2
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
AWS_ECR: "<account_id>.dkr.ecr.<region>.amazonaws.com/<repo_name>"
MAVEN_OPTS: "-Djava.awt.headless=true -Dmaven.repo.local=./.m2/repository"
MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version"
stages:
- login
- package
aws:
image:
name: amazon/aws-cli
entrypoint: [""]
stage: login
script:
- aws ecr get-login-password --region $AWS_DEFAULT_REGION >> password.txt
artifacts:
paths:
- password.txt
package:
needs:
- job: aws
artifacts: true
image:
name: softinstigate/graalvm-maven
entrypoint: [""]
stage: package
script:
- ls -ltr
- source /root/.bashrc
- mvn install $MAVEN_CLI_OPTS -DskipTests
- mvn package $MAVEN_CLI_OPTS -Pnative -DskipTests -Dquarkus.container-image.image=$AWS_ECR:latest -Dquarkus.container-image.push=true -Dquarkus.container-image.username=AWS -Dquarkus.container-image.password=$(cat password.txt)
Error: Unrecognized option: -c
+1but maybe I'm doing it wrong:
relevant .gitlab-ci.yml step: [...]
If somebody stumbles over this -c
-Error as well: The image ubi-quarkus-mandrel
-image has a default entrypoint enabled, you need to override this, then the -c
-Error disappears.:
image:
name: quay.io/quarkus/ubi-quarkus-mandrel:22.3.0.1-Final-java17
entrypoint:
- ""
Hi everyone, I am posting my solution for gitlab cd-ci, hope it helps
variables:
GRAALVM_IMG_TAG: ol7-java17-22.3.0
GRAALVM_HOME_DIR: /opt/graalvm-ce-java17-22.3.0/
build-jar:
image: ghcr.io/graalvm/graalvm-ce:$GRAALVM_IMG_TAG
needs:
- install-dependencies
- test
before_script:
- yum install wget -y
- wget https://dlcdn.apache.org/maven/maven-3/3.9.2/binaries/apache-maven-3.9.2-bin.tar.gz --no-check-certificate
- tar -xvf apache-maven-3.9.2-bin.tar.gz
- mv apache-maven-3.9.2 /opt/
- export M2_HOME='/opt/apache-maven-3.9.2'
- PATH="$M2_HOME/bin:$PATH"
- export PATH
- mvn -version
- export GRAALVM_HOME=$GRAALVM_HOME_DIR
- gu install native-image
script:
- mvn package -Pnative -DskipTests
artifacts:
paths:
- ${CI_PROJECT_DIR}/target
@Lory1990 Nice, but unfortunately these ghcr.io/graalvm/graalvm-ce images have been deprecated
Alternative use: quay.io/quarkus/ubi-quarkus-mandrel-builder-image and mvnw Ref: https://maven.apache.org/wrapper/
Can you add an example gitlab-ci.yml file to the documentation? I haven't been able to find such an example and it would be especially interesting to include tasks for tests, native tests, jvm builds and native builds. Including such an example allows an easier start into Quarkus.