search

buildpacks / tekton-integration

Buildpacks + Tekton
22 stars 3 forks source link

Unable to setup python runtime using google buildpack #32

Closed dmitry-mightydevops closed 2 years ago

dmitry-mightydevops commented 2 years ago

Incorrect python runtime version selected despite values in project.toml or via `stepTemplate':

part of my tekton task has this

 stepTemplate:
    env:
    - name: CNB_PLATFORM_API
      value: "0.4"
    - name: GOOGLE_RUNTIME_VERSION
      value: 3.8.2
    envFrom:
    - configMapRef:
        name: $(params.application)-build-pipeline-config
    name: ""
    resources: {}

....

  - image: $(params.builder_image)
    imagePullPolicy: Always
    name: build
    resources: {}
    script: |
      #!/bin/bash

      aws ecr get-login-password --region $AWS_REGION | docker login -u AWS --password-stdin $(params.docker_registry)

      # build image utilizing backpack
      /cnb/lifecycle/creator \
        -app=/workspace/app/$(params.source_subpath) \
        -project-metadata=/workspace/app/project.toml \
        -cache-dir=/cache \
        -layers=/layers \
        -platform=/platform \
        -report=/layers/report.toml \
        -cache-image=$(params.cache_image) \
        -uid=$(params.user_id) \
        -gid=$(params.group_id) \
        -process-type=$(params.process_type) \
        -skip-restore=$(params.skip_restore) \
        -previous-image=$(resources.outputs.image.url) \
        -run-image=$(params.run_image) \
        $(resources.outputs.image.url)
    securityContext:
      runAsGroup: 1000
      runAsUser: 1000
    volumeMounts:
    - mountPath: /layers
      name: layers-dir
    - mountPath: /cache
      name: $(params.cache)
    - mountPath: /platform
      name: $(params.platform_dir)

project.toml

✗  cat project.toml
[[build.env]]
name =  "GOOGLE_RUNTIME_VERSION"
value = "3.8.2"

and yet during detection I get Python 3.9.5 installed. So how do I set a proper Python runtime?

✗  tknlo vp-backend-prod-build-pipeline-run-r4zhm                                        

[buildpack : git-source-app-z8mgb] {"level":"info","ts":1630821410.015329,"caller":"git/git.go:169","msg":"Successfully cloned git@github.com:saritasa-nest/vp-backend.git @ 4206f69c065e9cbb187a70758507d2bc691726ef (grafted, HEAD) in path /workspace/app"}
[buildpack : git-source-app-z8mgb] {"level":"info","ts":1630821410.0349324,"caller":"git/git.go:207","msg":"Successfully initialized and updated submodules in path /workspace/app"}

[buildpack : python] Found requirements.txt

[buildpack : build] WARNING! Your password will be stored unencrypted in /home/cnb/.docker/config.json.
[buildpack : build] Configure a credential helper to remove this warning. See
[buildpack : build] https://docs.docker.com/engine/reference/commandline/login/#credentials-store
[buildpack : build] 
[buildpack : build] Login Succeeded
[buildpack : build] ===> DETECTING
[buildpack : build] google.python.runtime    0.9.1
[buildpack : build] google.python.pip        0.9.2
[buildpack : build] google.config.entrypoint 0.9.0
[buildpack : build] google.utils.label       0.0.1
[buildpack : build] ===> ANALYZING
[buildpack : build] Previous image with name "xxx.dkr.ecr.us-west-2.amazonaws.com/vp/prod/backend:prod-4206f69" not found
[buildpack : build] Restoring metadata for "google.python.pip:pipcache" from cache
[buildpack : build] ===> RESTORING
[buildpack : build] Restoring data for "google.python.pip:pipcache" from cache
[buildpack : build] ===> BUILDING
[buildpack : build] === Python - Runtime (google.python.runtime@0.9.1) ===
[buildpack : build] Using latest runtime version: 3.9.5
[buildpack : build] Installing Python v3.9.5
[buildpack : build] Upgrading pip to the latest version and installing build tools
[buildpack : build] ----------------------------------------------------------------
dmitry-mightydevops commented 2 years ago

At the same time if I just issue pack command, a proper runtime is selected as shown below

➜ pack build vp-backend --builder xxx.dkr.ecr.us-west-2.amazonaws.com/vp/prod/buildpacks/google/builder:v1 --path .
v1: Pulling from vp/prod/buildpacks/google/builder
Digest: sha256:99885c2e4d529acb865613cb3546ad895fcdd53002afd2a51d0ab336d4132e3e
Status: Image is up to date for xxx.dkr.ecr.us-west-2.amazonaws.com/vp/prod/buildpacks/google/builder:v1
v1: Pulling from buildpacks/gcp/run
a94600045bce: Pull complete 
92ed1a359aed: Pull complete 
3c2cba919283: Pull complete 
d96e3c6b1e3b: Pull complete 
7d84a02405d5: Pull complete 
Digest: sha256:4c7a95e9f39fac91b12357b834c7aab723c14a75ba23b0a92a38ed01a7fb0fc8
Status: Downloaded newer image for gcr.io/buildpacks/gcp/run:v1
0.11.1: Pulling from buildpacksio/lifecycle
Digest: sha256:2ca9870d9472400792bdc2d43ded8df2642ffcf937e8cd01685836678c28780b
Status: Image is up to date for buildpacksio/lifecycle:0.11.1
===> DETECTING
[detector] 3 of 4 buildpacks participating
[detector] google.python.runtime    0.9.1
[detector] google.config.entrypoint 0.9.0
[detector] google.utils.label       0.0.1
===> ANALYZING
[analyzer] Previous image with name "vp-backend" not found
===> RESTORING
===> BUILDING
[builder] === Python - Runtime (google.python.runtime@0.9.1) ===
[builder] Using runtime version from GOOGLE_RUNTIME_VERSION: 3.8.2
[builder] Installing Python v3.8.2
[builder] Upgrading pip to the latest version and installing build tools
[builder] --------------------------------------------------------------------------------
[builder] Running "/layers/google.python.runtime/python/bin/python3 -m pip install --upgrade pip setuptools wheel"
^C^CERROR: failed to build: executing lifecycle. This may be the result of using an untrusted builder: context canceled
dmitry-mightydevops commented 2 years ago 
✗  pack inspect-builder xxx.dkr.ecr.us-west-2.amazonaws.com/vp/prod/buildpacks/google/builder:v1         
Inspecting builder: xxx.dkr.ecr.us-west-2.amazonaws.com/vp/prod/buildpacks/google/builder:v1

REMOTE:

Description: Ubuntu 18 base image with buildpacks for .NET, Go, Java, Node.js, and Python

Created By:
  Name: Pack CLI
  Version: 0.13.1+git-4134cc6.build-1135

Trusted: No

Stack:
  ID: google

Lifecycle:
  Version: 0.11.1
  Buildpack APIs:
    Deprecated: (none)
    Supported: 0.2, 0.3, 0.4, 0.5, 0.6
  Platform APIs:
    Deprecated: (none)
    Supported: 0.3, 0.4, 0.5, 0.6

Run Images:
  gcr.io/buildpacks/gcp/run:v1

Buildpacks:
  ID                                       NAME        VERSION        HOMEPAGE
  google.config.entrypoint                 -           0.9.0          -
  google.cpp.clear_source                  -           0.1.0          -
  google.cpp.functions-framework           -           0.1.0          -
  google.dotnet.functions-framework        -           0.0.1          -
  google.dotnet.publish                    -           0.9.0          -
  google.dotnet.runtime                    -           0.9.1          -
  google.go.build                          -           0.9.0          -
  google.go.clear_source                   -           0.9.0          -
  google.go.functions-framework            -           0.9.4          -
  google.go.gopath                         -           0.9.0          -
  google.go.runtime                        -           0.9.1          -
  google.java.clear_source                 -           0.9.0          -
  google.java.entrypoint                   -           0.9.0          -
  google.java.exploded-jar                 -           0.9.0          -
  google.java.functions-framework          -           1.1.0          -
  google.java.graalvm                      -           0.1.0          -
  google.java.gradle                       -           0.9.0          -
  google.java.maven                        -           0.9.0          -
  google.java.native-image                 -           0.1.0          -
  google.java.runtime                      -           0.9.1          -
  google.nodejs.functions-framework        -           0.9.3          -
  google.nodejs.npm                        -           0.9.0          -
  google.nodejs.runtime                    -           0.9.2          -
  google.nodejs.yarn                       -           0.9.0          -
  google.python.functions-framework        -           0.9.5          -
  google.python.missing-entrypoint         -           0.9.0          -
  google.python.pip                        -           0.9.2          -
  google.python.runtime                    -           0.9.1          -
  google.utils.label                       -           0.0.1          -

Detection Order:
 ├ Group #1:
 │  ├ google.dotnet.functions-framework    (optional)
 │  ├ google.dotnet.runtime
 │  ├ google.dotnet.publish
 │  ├ google.config.entrypoint             (optional)
 │  └ google.utils.label
 ├ Group #2:
 │  ├ google.dotnet.runtime
 │  ├ google.config.entrypoint
 │  └ google.utils.label
 ├ Group #3:
 │  ├ google.go.runtime
 │  ├ google.go.functions-framework
 │  ├ google.go.build
 │  ├ google.config.entrypoint         (optional)
 │  ├ google.go.clear_source           (optional)
 │  └ google.utils.label
 ├ Group #4:
 │  ├ google.go.runtime
 │  ├ google.go.gopath            (optional)
 │  ├ google.go.build
 │  ├ google.config.entrypoint    (optional)
 │  ├ google.go.clear_source      (optional)
 │  └ google.utils.label
 ├ Group #5:
 │  ├ google.java.graalvm
 │  ├ google.java.maven
 │  ├ google.java.functions-framework    (optional)
 │  ├ google.java.native-image
 │  ├ google.config.entrypoint           (optional)
 │  ├ google.java.clear_source           (optional)
 │  └ google.utils.label
 ├ Group #6:
 │  ├ google.java.runtime
 │  ├ google.java.maven
 │  ├ google.java.functions-framework
 │  ├ google.java.clear_source           (optional)
 │  └ google.utils.label
 ├ Group #7:
 │  ├ google.java.runtime
 │  ├ google.java.gradle                 (optional)
 │  ├ google.java.functions-framework
 │  ├ google.java.clear_source           (optional)
 │  └ google.utils.label
 ├ Group #8:
 │  ├ google.java.runtime
 │  ├ google.java.exploded-jar
 │  └ google.utils.label
 ├ Group #9:
 │  ├ google.java.runtime
 │  ├ google.java.maven
 │  ├ google.config.entrypoint
 │  ├ google.java.clear_source    (optional)
 │  └ google.utils.label
 ├ Group #10:
 │  ├ google.java.runtime
 │  ├ google.java.maven
 │  ├ google.java.entrypoint
 │  ├ google.java.clear_source    (optional)
 │  └ google.utils.label
 ├ Group #11:
 │  ├ google.java.runtime
 │  ├ google.java.gradle          (optional)
 │  ├ google.config.entrypoint
 │  ├ google.java.clear_source    (optional)
 │  └ google.utils.label
 ├ Group #12:
 │  ├ google.java.runtime
 │  ├ google.java.gradle          (optional)
 │  ├ google.java.entrypoint
 │  ├ google.java.clear_source    (optional)
 │  └ google.utils.label
 ├ Group #13:
 │  ├ google.python.runtime
 │  ├ google.python.functions-framework
 │  ├ google.python.pip                    (optional)
 │  ├ google.config.entrypoint             (optional)
 │  └ google.utils.label
 ├ Group #14:
 │  ├ google.python.runtime
 │  ├ google.python.pip           (optional)
 │  ├ google.config.entrypoint
 │  └ google.utils.label
 ├ Group #15:
 │  ├ google.nodejs.runtime
 │  ├ google.nodejs.yarn
 │  ├ google.nodejs.functions-framework    (optional)
 │  ├ google.config.entrypoint             (optional)
 │  └ google.utils.label
 ├ Group #16:
 │  ├ google.nodejs.runtime
 │  ├ google.nodejs.npm
 │  ├ google.nodejs.functions-framework    (optional)
 │  ├ google.config.entrypoint             (optional)
 │  └ google.utils.label
 ├ Group #17:
 │  ├ google.nodejs.runtime
 │  ├ google.nodejs.functions-framework
 │  ├ google.config.entrypoint             (optional)
 │  └ google.utils.label
 ├ Group #18:
 │  ├ google.nodejs.runtime
 │  ├ google.config.entrypoint
 │  └ google.utils.label
 ├ Group #19:
 │  ├ google.python.runtime
 │  ├ google.python.missing-entrypoint
 │  └ google.utils.label
 └ Group #20:
    ├ google.cpp.functions-framework
    ├ google.cpp.clear_source           (optional)
    └ google.utils.label

LOCAL:

Description: Ubuntu 18 base image with buildpacks for .NET, Go, Java, Node.js, and Python

Created By:
  Name: Pack CLI
  Version: 0.13.1+git-4134cc6.build-1135

Trusted: No

Stack:
  ID: google

Lifecycle:
  Version: 0.11.1
  Buildpack APIs:
    Deprecated: (none)
    Supported: 0.2, 0.3, 0.4, 0.5, 0.6
  Platform APIs:
    Deprecated: (none)
    Supported: 0.3, 0.4, 0.5, 0.6

Run Images:
  gcr.io/buildpacks/gcp/run:v1

Buildpacks:
  ID                                       NAME        VERSION        HOMEPAGE
  google.config.entrypoint                 -           0.9.0          -
  google.cpp.clear_source                  -           0.1.0          -
  google.cpp.functions-framework           -           0.1.0          -
  google.dotnet.functions-framework        -           0.0.1          -
  google.dotnet.publish                    -           0.9.0          -
  google.dotnet.runtime                    -           0.9.1          -
  google.go.build                          -           0.9.0          -
  google.go.clear_source                   -           0.9.0          -
  google.go.functions-framework            -           0.9.4          -
  google.go.gopath                         -           0.9.0          -
  google.go.runtime                        -           0.9.1          -
  google.java.clear_source                 -           0.9.0          -
  google.java.entrypoint                   -           0.9.0          -
  google.java.exploded-jar                 -           0.9.0          -
  google.java.functions-framework          -           1.1.0          -
  google.java.graalvm                      -           0.1.0          -
  google.java.gradle                       -           0.9.0          -
  google.java.maven                        -           0.9.0          -
  google.java.native-image                 -           0.1.0          -
  google.java.runtime                      -           0.9.1          -
  google.nodejs.functions-framework        -           0.9.3          -
  google.nodejs.npm                        -           0.9.0          -
  google.nodejs.runtime                    -           0.9.2          -
  google.nodejs.yarn                       -           0.9.0          -
  google.python.functions-framework        -           0.9.5          -
  google.python.missing-entrypoint         -           0.9.0          -
  google.python.pip                        -           0.9.2          -
  google.python.runtime                    -           0.9.1          -
  google.utils.label                       -           0.0.1          -

Detection Order:
 ├ Group #1:
 │  ├ google.dotnet.functions-framework    (optional)
 │  ├ google.dotnet.runtime
 │  ├ google.dotnet.publish
 │  ├ google.config.entrypoint             (optional)
 │  └ google.utils.label
 ├ Group #2:
 │  ├ google.dotnet.runtime
 │  ├ google.config.entrypoint
 │  └ google.utils.label
 ├ Group #3:
 │  ├ google.go.runtime
 │  ├ google.go.functions-framework
 │  ├ google.go.build
 │  ├ google.config.entrypoint         (optional)
 │  ├ google.go.clear_source           (optional)
 │  └ google.utils.label
 ├ Group #4:
 │  ├ google.go.runtime
 │  ├ google.go.gopath            (optional)
 │  ├ google.go.build
 │  ├ google.config.entrypoint    (optional)
 │  ├ google.go.clear_source      (optional)
 │  └ google.utils.label
 ├ Group #5:
 │  ├ google.java.graalvm
 │  ├ google.java.maven
 │  ├ google.java.functions-framework    (optional)
 │  ├ google.java.native-image
 │  ├ google.config.entrypoint           (optional)
 │  ├ google.java.clear_source           (optional)
 │  └ google.utils.label
 ├ Group #6:
 │  ├ google.java.runtime
 │  ├ google.java.maven
 │  ├ google.java.functions-framework
 │  ├ google.java.clear_source           (optional)
 │  └ google.utils.label
 ├ Group #7:
 │  ├ google.java.runtime
 │  ├ google.java.gradle                 (optional)
 │  ├ google.java.functions-framework
 │  ├ google.java.clear_source           (optional)
 │  └ google.utils.label
 ├ Group #8:
 │  ├ google.java.runtime
 │  ├ google.java.exploded-jar
 │  └ google.utils.label
 ├ Group #9:
 │  ├ google.java.runtime
 │  ├ google.java.maven
 │  ├ google.config.entrypoint
 │  ├ google.java.clear_source    (optional)
 │  └ google.utils.label
 ├ Group #10:
 │  ├ google.java.runtime
 │  ├ google.java.maven
 │  ├ google.java.entrypoint
 │  ├ google.java.clear_source    (optional)
 │  └ google.utils.label
 ├ Group #11:
 │  ├ google.java.runtime
 │  ├ google.java.gradle          (optional)
 │  ├ google.config.entrypoint
 │  ├ google.java.clear_source    (optional)
 │  └ google.utils.label
 ├ Group #12:
 │  ├ google.java.runtime
 │  ├ google.java.gradle          (optional)
 │  ├ google.java.entrypoint
 │  ├ google.java.clear_source    (optional)
 │  └ google.utils.label
 ├ Group #13:
 │  ├ google.python.runtime
 │  ├ google.python.functions-framework
 │  ├ google.python.pip                    (optional)
 │  ├ google.config.entrypoint             (optional)
 │  └ google.utils.label
 ├ Group #14:
 │  ├ google.python.runtime
 │  ├ google.python.pip           (optional)
 │  ├ google.config.entrypoint
 │  └ google.utils.label
 ├ Group #15:
 │  ├ google.nodejs.runtime
 │  ├ google.nodejs.yarn
 │  ├ google.nodejs.functions-framework    (optional)
 │  ├ google.config.entrypoint             (optional)
 │  └ google.utils.label
 ├ Group #16:
 │  ├ google.nodejs.runtime
 │  ├ google.nodejs.npm
 │  ├ google.nodejs.functions-framework    (optional)
 │  ├ google.config.entrypoint             (optional)
 │  └ google.utils.label
 ├ Group #17:
 │  ├ google.nodejs.runtime
 │  ├ google.nodejs.functions-framework
 │  ├ google.config.entrypoint             (optional)
 │  └ google.utils.label
 ├ Group #18:
 │  ├ google.nodejs.runtime
 │  ├ google.config.entrypoint
 │  └ google.utils.label
 ├ Group #19:
 │  ├ google.python.runtime
 │  ├ google.python.missing-entrypoint
 │  └ google.utils.label
 └ Group #20:
    ├ google.cpp.functions-framework
    ├ google.cpp.clear_source           (optional)
    └ google.utils.label
dmitry-mightydevops commented 2 years ago

If I print available ENV vars in the build step, I get the following


[buildpack : build] GOOGLE_RUNTIME_VERSION=3.8.2
[buildpack : build] CNB_PLATFORM_API=0.4
[buildpack : build] HOSTNAME=vp-backend-prod-build-pipeline-run-255dk-buildpack-44qzc--b2s2n
[buildpack : build] ARGOCD_ROOT_PATH=/
[buildpack : build] CNB_STACK_ID=google
[buildpack : build] KUBERNETES_PORT_443_TCP_PROTO=tcp
[buildpack : build] KUBERNETES_PORT_443_TCP_ADDR=172.20.0.1
[buildpack : build] EL_BUILD_PIPELINE_EVENT_LISTENER_PORT_8080_TCP=tcp://172.20.145.135:8080
[buildpack : build] KUBERNETES_PORT=tcp://172.20.0.1:443
[buildpack : build] EL_BUILD_PIPELINE_EVENT_LISTENER_PORT=tcp://172.20.145.135:8080
[buildpack : build] PWD=/layers
[buildpack : build] APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1
[buildpack : build] HOME=/home/cnb
[buildpack : build] EL_BUILD_PIPELINE_EVENT_LISTENER_SERVICE_HOST=172.20.145.135
[buildpack : build] KUBERNETES_SERVICE_PORT_HTTPS=443
[buildpack : build] DEBIAN_FRONTEND=noninteractive
[buildpack : build] CNB_USER_ID=1000
[buildpack : build] KUBERNETES_PORT_443_TCP_PORT=443
[buildpack : build] EL_BUILD_PIPELINE_EVENT_LISTENER_SERVICE_PORT_HTTP_LISTENER=8080
[buildpack : build] AWS_REGION=us-west-2
[buildpack : build] EL_BUILD_PIPELINE_EVENT_LISTENER_SERVICE_PORT=8080
[buildpack : build] TEKTON_PIPELINE_NAME=buildpack-build-pipeline
[buildpack : build] KUBERNETES_PORT_443_TCP=tcp://172.20.0.1:443
[buildpack : build] EL_BUILD_PIPELINE_EVENT_LISTENER_PORT_8080_TCP_ADDR=172.20.145.135
[buildpack : build] CNB_GROUP_ID=1000
[buildpack : build] SHLVL=1
[buildpack : build] KUBERNETES_SERVICE_PORT=443
[buildpack : build] EL_BUILD_PIPELINE_EVENT_LISTENER_PORT_8080_TCP_PROTO=tcp
[buildpack : build] EL_BUILD_PIPELINE_EVENT_LISTENER_PORT_8080_TCP_PORT=8080
[buildpack : build] PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[buildpack : build] KUBERNETES_SERVICE_HOST=172.20.0.1
[buildpack : build] _=/usr/bin/env
[buildpack : build] WARNING! Your password will be stored unencrypted in /home/cnb/.docker/config.json.
[buildpack : build] Configure a credential helper to remove this warning. See
[buildpack : build] https://docs.docker.com/engine/reference/commandline/login/#credentials-store

So the ENV is there, but no luck getting 3.8.2 to be "detected".

jromero commented 2 years ago

Hi @dmitry-mightydevops,

Based on the spec, user-provided environment variables must be passed via the file system at /platform/env. You can see how our tasks currently do this by looking at the source here.

Please re-open if this doesn't solve/answer your question.