`yj` is not working properly at times

fgrehm commented 2 years ago

What happened?

What were you attempting to do?

Run yj during our image builds to convert toml to JSON, just so we can manipulate / query with jq

What did you expect to happen?

yj should always work

What was the actual behavior? Please provide log output, if possible.

At times it seem to me that it's failing to download:

docker run -ti --rm docker.io/paketobuildpacks/build:full-cnb yj -h
# results in "standard_init_linux.go:228: exec user process caused: exec format error"

docker run -ti --rm docker.io/paketobuildpacks/build:full-cnb cat /usr/local/bin/yj
# Returns "Not Found"

docker run -ti --rm docker.io/paketobuildpacks/build:1.3.33-full-cnb cat /usr/local/bin/yj
# Returns "Not Found"

docker run -ti --rm docker.io/paketobuildpacks/build:1.3.31-full-cnb cat /usr/local/bin/yj
# Gibberish since it's the executable

docker run -ti --rm docker.io/paketobuildpacks/build:1.3.31-full-cnb yj -h
# Usage: yj [-][ytjcrneikhv] ....

We have automated tests for our custom builder which includes pulling the upstream full-cnb img and adding "our stuff" on top of it. We noticed this last friday and I have no idea why things were intermittent for me, we run this on CircleCI so it could be some layer cache since we don't lock it to a specific version (we want to always pull in latest)

Here's the docker inspect for what I have on my computer right now.

Build Configuration

What platform (pack, kpack, tekton buildpacks plugin, etc.) are you using? Please include a version.

We have our own wrapper on top of lifecycle which uses yj, but shouldn't matter in this case IMO

What buildpacks are you using? Please include versions.

Should not matter

What builder are you using? If custom, can you provide the output from pack inspect-builder <builder>?

Should not matter

Can you provide a sample app or relevant configuration (buildpack.yml, nginx.conf, etc.)?

Should not be necessary

Checklist

[x] I have included log output.
[x] The log output includes an error message.
[x] I have included steps for reproduction.

Starting point

https://github.com/sclevine/yj/releases/latest/download/yj-linux from https://github.com/paketo-buildpacks/stacks/blob/554003b821598c1d24415ffae4d0759acb60d14b/bionic/dockerfile/build/Dockerfile#L19

Redirects me to a 404 page. I wonder if we should lock it to a specific release on the Dockerfile

$ curl -LI https://github.com/sclevine/yj/releases/latest/download/yj-linux
HTTP/2 302
server: GitHub.com
date: Mon, 18 Apr 2022 21:50:58 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Accept-Encoding, Accept, X-Requested-With
permissions-policy: interest-cohort=()
location: https://github.com/sclevine/yj/releases/download/v5.1.0/yj-linux
cache-control: no-cache
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 0
referrer-policy: no-referrer-when-downgrade
expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubusercontent.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events translator.github.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com online.visualstudio.com/api/v1/locations github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src render.githubusercontent.com viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com github-cloud.s3.amazonaws.com secured-user-images.githubusercontent.com/ *.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; worker-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/
set-cookie: _gh_sess=mP4t5BfJNy6pli4yARLzRv6C1u0cy0VfQPBVz95eK9A9QnGbs0yUAAuFnG4VOX9cBGq30EJx6Vw6y0zeD%2B8yCaDKdlbEohVywiyZdmS1VS7Cy3yQI%2BgekXs7oykxZOjn9ebZzd87PlDHM5%2F0WXOUUSNCrC3fJq4sd4KwRNva%2FM%2FA8zxyz4BR%2BpXLLmXKBo4Llw6kuFBjMAJ1otfi6nTOeXkxLwO5v%2Br1awk9x45bTNGb0gHIpV9AfCqVSILCAHFhu%2BUbdOBP9pTfuDv0LtRJzQ%3D%3D--A1YEbvhIjWntarMJ--eYOiSsYTNHYU%2Bq68MI81Cw%3D%3D; Path=/; HttpOnly; Secure; SameSite=Lax
set-cookie: _octo=GH1.1.1204816530.1650318658; Path=/; Domain=github.com; Expires=Tue, 18 Apr 2023 21:50:58 GMT; Secure; SameSite=Lax
set-cookie: logged_in=no; Path=/; Domain=github.com; Expires=Tue, 18 Apr 2023 21:50:58 GMT; HttpOnly; Secure; SameSite=Lax
content-length: 130
x-github-request-id: 42E0:1571:55F615:670056:625DDD42

HTTP/2 404
server: GitHub.com
date: Mon, 18 Apr 2022 21:50:58 GMT
content-type: text/plain; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Accept-Encoding, Accept, X-Requested-With
permissions-policy: interest-cohort=()
cache-control: no-cache
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 0
referrer-policy: no-referrer-when-downgrade
expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
content-security-policy: default-src 'none'; base-uri 'self'; connect-src 'self'; form-action 'self'; img-src 'self' data:; script-src 'self'; style-src 'unsafe-inline'
x-github-request-id: 42E0:1571:55F629:670076:625DDD42

Another idea would be to change that curl to fail if HTTP status != 200

robdimsdale commented 2 years ago

The reason that https://github.com/sclevine/yj/releases/latest/download/yj-linux is returning a 404 is because the name of the binary on the release pages has changed from yj-linux to yj-linux-amd64 in release v5.1.0, so the correct URL is now: https://github.com/sclevine/yj/releases/latest/download/yj-linux-amd64.

I've opened #130 to fix this.

sophiewigmore commented 2 years ago

@fgrehm the code has been merged in, so the change should take effect in the next release of the stacks. Unfortunately, the stacks aren't released until there's a change in the packages in either build or run image. Is this blocking for you? There might be a viable workaround until the stacks are released if needed.

fgrehm commented 2 years ago

@sophiewigmore sorry, missed your msg here, I wrote this on the PR https://github.com/paketo-buildpacks/stacks/pull/131#issuecomment-1102728325

paketo-buildpacks / stacks