search

buildpacks / pack

CLI for building apps using Cloud Native Buildpacks
https://buildpacks.io
Apache License 2.0
2.56k stars 286 forks source link

Environment variables not available while running `pack build` #1376

Closed marcusball closed 2 years ago

marcusball commented 2 years ago

Summary

The builder I'm trying to use requires some environment variables to be set while it is building. I can run the builder directly though docker like this, docker run -t -e "MY_ENV=example" --rm example/builder, and it works fine. However, if I try to run the builder with Pack, pack build example-image -B example/builder --env "MY_ENV=example" the detector phase throws an error saying the environment variable is not defined. If I configure Pack to trust the builder, it also fails.

I've tried defining the variables via --env, I've tried putting them in a file an passing it via --env-file, and I've tried putting them in a .env file just to see if that would do anything (even though I wasn't expecting it to).

Based on this issue, I'm under the impression that these variables should be present during the detector phase. I'll admit, I could also be completely misunderstanding the meaning of "build-time" if the builder image running is the "image runtime" during which the variables are not available. If I am misunderstanding this, when is build-time if not when the builder is running?

The builder I'm trying to use is digitaloceanapps/apps-builder with the required GIT_SOURCE_URL variable.

Reproduction

Steps
  1. pack build problem-test -B digitaloceanapps/apps-builder --verbose --env "GIT_SOURCE_URL=https://github.com/buildpacks/pack"; the git URL and the working directory do not really matter for this issue, I'm just using the Pack repository as an example.
Current behavior

If the builder is untrusted: 

 > pack build problem-test -B digitaloceanapps/apps-builder --verbose --env "GIT_SOURCE_URL=https://github.com/buildpacks/pack"                         

Builder digitaloceanapps/apps-builder is untrusted
As a result, the phases of the lifecycle which require root access will be run in separate trusted ephemeral containers.
For more information, see https://medium.com/buildpacks/faster-more-secure-builds-with-pack-0-11-0-4d0c633ca619
Pulling image index.docker.io/digitaloceanapps/apps-builder:latest
latest: Pulling from digitaloceanapps/apps-builder
Digest: sha256:d41801860d149d931bca2333131784e0133c8cfa7ca81f200443a432eb6549c7
Status: Image is up to date for digitaloceanapps/apps-builder:latest
Selected run image digitaloceanapps/apps-run:d6df4f3
Pulling image digitaloceanapps/apps-run:d6df4f3
d6df4f3: Pulling from digitaloceanapps/apps-run
Digest: sha256:94d0e3a9a35bd8e51c8a240f4be9d5f43bc71c61dc89098e60472f544c037891
Status: Image is up to date for digitaloceanapps/apps-run:d6df4f3
Creating builder with the following buildpacks:
-> digitalocean/custom@0.1.0
-> digitalocean/go-appdetect@0.0.7
-> digitalocean/hugo@0.0.7
-> digitalocean/node@0.3.4
-> digitalocean/php-appdetect@0.0.3
-> digitalocean/procfile@0.0.3
-> digitalocean/python-appdetect@0.0.2
-> digitalocean/ruby-appdetect@0.0.2
-> digitalocean/static@0.0.2
-> heroku/go@0.156.4
-> heroku/nodejs-engine@0.5.0
-> heroku/php@0.210.4
-> heroku/python@0.205.4
-> heroku/ruby@0.223.1
Provided Environment Variables
  GIT_SOURCE_URL=https://github.com/buildpacks/pack
Pulling image buildpacksio/lifecycle:0.10.2
0.10.2: Pulling from buildpacksio/lifecycle
Digest: sha256:c3a070ed0eaf8776b66f9f7c285469edccf5299b3283c453dd45699d58d78003
Status: Image is up to date for buildpacksio/lifecycle:0.10.2
Using build cache volume pack-cache-library_problem-test_latest-32bac1d1c73a.build
===> DETECTING
Running the detector on OS linux with:
Container Settings:
  Args: /cnb/lifecycle/detector -app /workspace -log-level debug
  System Envs: CNB_PLATFORM_API=0.5
  Image: pack.local/builder/6c766e75667a766b6a61:latest
  User: 
  Labels: map[author:pack]
Host Settings:
  Binds: pack-layers-bonvpvrsnb:/layers pack-app-qyizqyqgmk:/workspace
  Network Mode: 
[detector] => Initializing build
[detector]  ! $GIT_SOURCE_URL is not set
ERROR: failed to build: executing lifecycle. This may be the result of using an untrusted builder: failed with status code: 1

If the builder is trusted: 

pack build problem-test -B digitaloceanapps/apps-builder --verbose --env "GIT_SOURCE_URL=https://github.com/buildpacks/pack" --trust-builder
Builder digitaloceanapps/apps-builder is trusted
Pulling image index.docker.io/digitaloceanapps/apps-builder:latest
latest: Pulling from digitaloceanapps/apps-builder
Digest: sha256:d41801860d149d931bca2333131784e0133c8cfa7ca81f200443a432eb6549c7
Status: Image is up to date for digitaloceanapps/apps-builder:latest
Selected run image digitaloceanapps/apps-run:d6df4f3
Pulling image digitaloceanapps/apps-run:d6df4f3
d6df4f3: Pulling from digitaloceanapps/apps-run
Digest: sha256:94d0e3a9a35bd8e51c8a240f4be9d5f43bc71c61dc89098e60472f544c037891
Status: Image is up to date for digitaloceanapps/apps-run:d6df4f3
Creating builder with the following buildpacks:
-> digitalocean/custom@0.1.0
-> digitalocean/go-appdetect@0.0.7
-> digitalocean/hugo@0.0.7
-> digitalocean/node@0.3.4
-> digitalocean/php-appdetect@0.0.3
-> digitalocean/procfile@0.0.3
-> digitalocean/python-appdetect@0.0.2
-> digitalocean/ruby-appdetect@0.0.2
-> digitalocean/static@0.0.2
-> heroku/go@0.156.4
-> heroku/nodejs-engine@0.5.0
-> heroku/php@0.210.4
-> heroku/python@0.205.4
-> heroku/ruby@0.223.1
Provided Environment Variables
  GIT_SOURCE_URL=https://github.com/buildpacks/pack
Using build cache volume pack-cache-library_problem-test_latest-32bac1d1c73a.build
Running the creator on OS linux with:
Container Settings:
  Args: /cnb/lifecycle/creator -daemon -launch-cache /launch-cache -log-level debug -app /workspace -cache-dir /cache -run-image digitaloceanapps/apps-run:d6df4f3 -process-type web problem-test
  System Envs: CNB_PLATFORM_API=0.5
  Image: pack.local/builder/6367676f6f70696e6678:latest
  User: root
  Labels: map[author:pack]
Host Settings:
  Binds: pack-cache-library_problem-test_latest-32bac1d1c73a.build:/cache /var/run/docker.sock:/var/run/docker.sock pack-cache-library_problem-test_latest-32bac1d1c73a.launch:/launch-cache pack-layers-yhwqsuiipm:/layers pack-app-zjvbisxlrh:/workspace
  Network Mode: 
=> Initializing build
 ! $GIT_SOURCE_URL is not set
ERROR: failed to build: executing lifecycle: failed with status code: 1
Expected behavior

The GIT_SOURCE_URL should be defined in the builder, and the builder should be able to proceed past the "initializing build" step. Ex: 

> docker run -t -e "GIT_SOURCE_URL=https://github.com/buildpacks/pack" --rm  digitaloceanapps/apps-builder
=> Initializing build
=> Retrieving source code to /workspace

Environment

pack info
pack report                                                       
Pack:
  Version:  0.24.0+git-79a40b7.build-3148
  OS/Arch:  linux/amd64

Default Lifecycle Version:  0.13.3

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8

Config:
  default-builder-image = "[REDACTED]"
docker info
docker info                                                          ✔ 
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  compose: Docker Compose (Docker Inc., 2.2.3)

Server:
 Containers: 13
  Running: 12
  Paused: 0
  Stopped: 1
 Images: 62
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1407cab509ff0d96baa4f0eb6ff9980270e6e620.m
 runc version: v1.1.0-0-g067aaf85
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.21-1-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.5GiB
 Name: [REDACTED]
 ID: [REDACTED]
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: [REDACTED]
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
jromero commented 2 years ago

@marcusball having a discussion on Slack about this here: https://cloud-native.slack.com/archives/C0331B61A1Y/p1646156479825739?thread_ts=1646156011.522369&cid=C0331B61A1Y

jromero commented 2 years ago

@marcusball an update here:

@natalieparellano dug into the digitaloceanapps/apps-builder and found that there's a custom entrypoint ("ENTRYPOINT [\"/.app_platform/build.sh\"]"). It's then that script that expects an environment variable.

As per the spec, pack only provides the environment variables through a file system interface and doesn't set the environment variables on execution of detector. If I'm not mistaken, detector then reads those environment variables from the file system and provides them to the buildpack's detect execution and that's what being referred to in this issue.

Relevant spec wording:

User-provided environment variables MUST be supplied by the platform as files in the /env/ directory. Each file SHALL define a single environment variable, where the file name defines the key and the file contents define the value.

User-provided environment variables MAY be modified by prior buildpacks before they are provided to a given buildpack.

The platform SHOULD NOT set user-provided environment variables directly in the lifecycle execution environment.

To summarize, it appears that the builder image you are using is expected to run in a very precise way and may arguably be non-compliant with the specification.

Please let me know if something I mentioned is still unclear.

Also, I was unable to find any reference online to the builder. If you can, could you relay this information to the builder author(s)?

marcusball commented 2 years ago

Thank you for the reply!

As per the spec, pack only provides the environment variables through a file system interface and doesn't set the environment variables on execution of detector. If I'm not mistaken, detector then reads those environment variables from the file system and provides them to the buildpack's detect execution and that's what being referred to in https://github.com/buildpacks/pack/issues/83.

Alright, that's the conclusion I eventually came to after filing the issue, but this is my first time trying to use Pack, so I wasn't confident I was interpreting the spec correctly.

Luckily, I ended up being able to work around once I dug into the builder image and realized it was basically just a bundle of Heroku builders with the additional custom entrypoint scripts.

natalieparellano commented 2 years ago

I'll close this issue as it seems resolved. Thanks for your feedback @marcusball !