search

paketo-buildpacks / opentelemetry

Apache License 2.0
19 stars 0 forks source link

Unable to add agent to spring boot app with lifecycle/creator #117

Closed wpater closed 9 months ago

wpater commented 9 months ago

I am trying to add OTEL agent to the working Spring Boot application using GitLab CI on the runner with kubernetes executor (no docker socket).

Currently I am able to build whole application and it is working on the cluster, but now I need to add agent and I don't know how to to it:

dockerize:
  image: paketobuildpacks/builder-jammy-base
  variables:
    REGISTRY_GROUP_PROJECT: $CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
    IMAGE: $REGISTRY_GROUP_PROJECT/app
    CNB_PLATFORM_API: '0.12'
    BP_OPENTELEMETRY_ENABLED: 'true'
  before_script:
    - mkdir ~/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" >> ~/.docker/config.json
  script:
    - /cnb/lifecycle/creator -buildpacks urn:cnb:builder/paketo-buildpacks/opentelemetry -app . -cache-image $REGISTRY_GROUP_PROJECT/paketo-build-cache:latest $IMAGE:$CI_COMMIT_SHORT_SHA

and output:

===> ANALYZING
Restoring data for SBOM from previous image
===> DETECTING
ERROR: failed to initialize detector: open /builds/<gitlab-group>/<gitlab-project-name>/urn:cnb:builder/paketo-buildpacks/opentelemetry/paketo-buildpacks_ruby/0.42.1/buildpack.toml: no such file or directory

It looks like /cnb/lifecycle/creator is expecting this buildpack on the local file system mounted to the pod which executes job and I think I am using/defining it incorrectly - I thought that it will be downloaded from the registry.

Expected Behavior

OTEL agent is added to the app

Current Behavior

Job is failing with ERROR: failed to initialize detector: open /builds/<gitlab-group>/<gitlab-project-name>/urn:cnb:builder/paketo-buildpacks/opentelemetry/paketo-buildpacks_ruby/0.42.1/buildpack.toml: no such file or directory

Possible Solution

Different naming for this buildpack?

Steps to Reproduce

  1. Use /cnb/lifecycle/creator on paketobuildpacks/builder-jammy-base image
  2. Build app with /cnb/lifecycle/creator -buildpacks urn:cnb:builder/paketo-buildpacks/opentelemetry -app . -cache-image $REGISTRY_GROUP_PROJECT/paketo-build-cache:latest $IMAGE:$CI_COMMIT_SHORT_SHA

Motivations

I am not able to add OTEL agent to the application

wpater commented 9 months ago

Now I see that -buildbpacks flag is a path to buildpacks directory - https://github.com/buildpacks/rfcs/blob/main/text/0026-lifecycle-all.md#usage which explains why it is trying to find this directory on the FS.

How I can pass opentelemetry buildpack to the cnb/lifecycle/creator?

dmikusa commented 9 months ago

I'm closing this because it's not a buildpack issue. This is a problem with the way you are running buildpacks.

Normally, when you use pack (or the Spring Boot Build tools) to build your application, they will take care of creating the build environment. Because you are running the lifecycle directly, you are 100% responsible for creating the build environment. This includes downloading and installing buildpacks in the proper location.

In the simplest case, you don't need to download any buildpacks because they are included with the builder image & if you use that builder image as your base image for the environment where you run the lifecycle then the buildpacks will just be present. Because the OpenTelemetry buildpack is not included in the builder, you must download and add the buildpack to the build environment manually.

Doing all that is beyond the help I can provider here. I would suggest checking out this article which walks through things more in detail, or reaching out on the Buildpacks Slack (the #buildpacks-* channels) for more help.

wpater commented 9 months ago

Since there is currently no way to add this buildpack to the java buildpack, here is a short summary of how I was able to solve this problem (I was not able to find any end-to-end tutorial for this, and certainly this approach still needs some cleanup and improvement).

  1. Own Docker image with pack CLI Official Docker image will fail with unauthorized CA for Docker Hub (index.docker.io) because it does not include standard certificates - extending buildpacksio/pack:0.33.1-base with

    RUN apt-get update && \
  apt-get install -y xz-utils ca-certificates && \
  rm -rf /var/lib/apt/lists/*

    should do the trick

  2. Own buildpack You need to create own buildpack which contains two dependencies - paketo-buildpacks/java and paketo-buildpacks/opentelemetry, buildpack.toml:

    api = "0.7"

[buildpack]
    description = "A Cloud Native Buildpack with an order definition suitable for Java applications with OTEL Agent - it's a paketo-buildpacks/java with paketo-buildpacks/opentelemetry"
    id = "registry.gitlab.com/<gitlab-group>/devops/buildpacks/java:0.0.1"
    keywords = ["java", "otel"]
    name = "My buildpack for Java"
    version = "0.0.1"

[metadata]
    include-files = ["buildpack.toml"]

[[order]]
    [[order.group]]
        id = "paketo-buildpacks/java"
        version = "11.1.0"
    [[order.group]]
        id = "paketo-buildpacks/opentelemetry"
        version = "1.7.0"

    and package.toml:

    [buildpack]
    uri = "."

[platform]
    os = "linux"

[[dependencies]]
    uri = "docker://gcr.io/paketo-buildpacks/java:11.1.0"

[[dependencies]]
    uri = "docker://gcr.io/paketo-buildpacks/opentelemetry:1.7.0"

  3. Pipeline for building buildpack

    buildpack:
  image:
    name: $CI_REGISTRY/<gitlab-group>/devops/pack:0.33.1
    entrypoint: [ "" ]
  stage: build
  before_script:
    - mkdir ~/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" >> ~/.docker/config.json
  script:
    - pack buildpack package $CI_REGISTRY_IMAGE/<gitlab-group>/devops/buildpacks/java:0.0.1 --config buildpack/java/package.toml --publish

  4. Own builder: Create own builder with own buildpack - builder.toml:

    description = "Ubuntu 22.04 Jammy Jellyfish base image with buildpacks for Java and OTEL Agent - based on https://github.com/paketo-buildpacks/builder-jammy-base"

[[buildpacks]]
    uri = "docker://registry.gitlab.com/<gitlab-group>/devops/buildpacks/java:0.0.1"
    version = "0.0.1"

[lifecycle]
    version = "0.18.4"

[[order]]
    [[order.group]]
        id = "registry.gitlab.com/<gitlab-group>/devops/buildpacks/java"
        version = "0.0.1"

[build]
    image = "cnbs/sample-base-build:jammy"
    [[build.env]]
        name = "BP_OPENTELEMETRY_ENABLED"
        value = "1"

[run]
    [[run.images]]
        image = "cnbs/sample-base-run:jammy"

  5. Pipeline for building builder

    builder:
  image:
    name: $CI_REGISTRY/<gitlab-group>/devops/pack:0.33.1
    entrypoint: [ "" ]
  stage: build
  before_script:
    - mkdir ~/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" >> ~/.docker/config.json
  script:
    - pack builder create $CI_REGISTRY_IMAGE/<gitlab-group>/devops/buildpacks/builder:0.0.1 --config buildpack/builder.toml --publish

  6. Pipeline for building app with own builder

    dockerize:
  image: registry.gitlab.com/<gitlab-group>/devops/buildpacks/builder:0.0.1
  variables:
    REGISTRY_GROUP_PROJECT: $CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
    IMAGE: $REGISTRY_GROUP_PROJECT/app
    CNB_PLATFORM_API: '0.12'
  before_script:
    - mkdir ~/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" >> ~/.docker/config.json
  script:
    - /cnb/lifecycle/creator -app ./api/$API -cache-image $REGISTRY_GROUP_PROJECT/paketo-build-cache:latest $IMAGE:$CI_COMMIT_SHORT_SHA

Using this I was able to take this buildpack into the app:

11 of 27 buildpacks participating
paketo-buildpacks/ca-certificates   3.6.7
paketo-buildpacks/bellsoft-liberica 10.5.2
paketo-buildpacks/syft              1.44.0
paketo-buildpacks/gradle            7.8.0
paketo-buildpacks/executable-jar    6.8.3
paketo-buildpacks/apache-tomcat     7.14.3
paketo-buildpacks/apache-tomee      1.8.0
paketo-buildpacks/liberty           4.0.0
paketo-buildpacks/dist-zip          5.6.8
paketo-buildpacks/spring-boot       5.27.8
paketo-buildpacks/opentelemetry     1.7.0

Again, a lot depends on the setup (Kubernetes executor without Docker socket in my case) and it could certainly be improved, but at least I managed to make it work and maybe it will be helpful for someone.

ThomasVitale commented 9 months ago

@wpater thanks for sharing! If you have a Kubernetes environment, you can also use kpack to publish your custom builder using convenient Kubernetes-native APIs. Here you can find an example of Kubernetes custom resources used to make a builder including the OpenTelemetry buildpack as well as all the other standard ones.

wpater commented 9 months ago

@ThomasVitale yeah, I saw kpack and probably I will use it in the future as production solution, just wanted to verify this approach :)

dmikusa commented 9 months ago

Thanks for sharing. A couple of notes...

  1. For others coming across this, if you can use pack, this is all simpler. Pack can make dynamic builders, so you can just run pack build -b paketo-buildpacks/java -b paketo-buildpacks/opentelementry ...<rest of your args>... and it will dynamically create a builder image with both buildpacks. If you can use pack (or the Spring Boot Build Tools), that is the shortest option.

  2. If you're making your own builder, you can shave off a step by not making a composite buildpack. Your steps 2.) and 3.) above. You can just inline the two buildpacks from your composite buildpack directly into the builder.toml. In the end, there's no real difference. It's just about how you structure and organize the buildpacks. For Paketo, having composite buildpacks makes sense, but it is more overhead.

  3. I would caution against using the buildpacks.io build/run images. For two reasons, 1.) I don't think they intend them to be production images and so I don't believe they have policies on things like when/how often they patch and update them, and 2.) Paketo buildpacks are designed and tested to work with the Paketo base images. Some Paketo buildpacks will work on other base images, but they're not tested in those scenarios. You might hit some additional problems by going that route. The Java related buildpacks don't have a lot of base-os dependencies so they tend to work fine across a lot of different scenarios, but it's still worth keeping in mind.