search

kubernetes-sigs / kustomize

Customization of kubernetes YAML configurations
Apache License 2.0
10.72k stars 2.22k forks source link

hack/install_kustomize.sh fails intermittently #4769

Open bewing opened 1 year ago

bewing commented 1 year ago

Describe the bug It seems like a change in how GitHub API text is returned is causing issues with the bash script hack/install_kustomize.sh. Instead of pretty printing the JSON output, it is returned as a single line. As a result, the incorrect download link is selected ( there is only a single line for grep to search, it returns it as true as the pattern is in that line, and cut selects the wrong URL for download). Platform

Tested on Ubuntu 20.04.4 LTS with curl 7.68.0

Additional context

Some output from curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash -xs -- 3.8.7 ~/bin (taken from operator SDK Makefile)

+ set -e
+ unset CDPATH
+ where=/home/bewing
+ release_url=https://api.github.com/repos/kubernetes-sigs/kustomize/releases
+ '[' -n 3.8.7 ']'
+ [[ 3.8.7 =~ ^[0-9]+(\.[0-9]+){2}$ ]]
+ version=v3.8.7
+ release_url=https://api.github.com/repos/kubernetes-sigs/kustomize/releases/tags/kustomize%2Fv3.8.7
<output omitted>
++ curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases/tags/kustomize%2Fv3.8.7
+ releases='{"url":"https://api.github.com/repos/kubernetes-sigs/kustomize/releases/33829673","assets_url":"https://api.github.com/repos/kubernetes-sigs/kustomize/releases/33829673/assets","upload_url":"https://uploads.github.com/repos/kubernetes-sigs/kustomize/releases/33829673/assets{?name,label}","html_url":"https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize/v3.8.7","id":33829673,"author":{"login":"monopole","id":2928188,"node_id":"MDQ6VXNlcjI5MjgxODg=","avatar_url":"https://avatars.githubusercontent.com/u/2928188?v=4","gravatar_id":"","url":"https://api.github.com/users/monopole","html_url":"https://github.com/monopole","followers_url":"https://api.github.com/users/monopole/followers","following_url":"https://api.github.com/users/monopole/following{/other_user}","gists_url":"https://api.github.com/users/monopole/gists{/gist_id}","starred_url":"https://api.github.com/users/monopole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/monopole/subscriptions","organizations_url":"https://api.github.com/users/monopole/orgs","repos_url":"https://api.github.com/users/monopole/repos","events_url":"https://api.github.com/users/monopole/events{/privacy}", <long line truncated>
<long output again removed>
++ cut -d '"' -f 4
++ sort -V
++ tail -n 1
+ RELEASE_URL=https://api.github.com/repos/kubernetes-sigs/kustomize/releases/33829673
+ '[' '!' -n https://api.github.com/repos/kubernetes-sigs/kustomize/releases/33829673 ']'
+ curl -sLO https://api.github.com/repos/kubernetes-sigs/kustomize/releases/33829673
+ tar xzf './kustomize_v*_linux_amd64.tar.gz'
tar (child): ./kustomize_v*_linux_amd64.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now

If I download install_kustomize.sh and modify the script to pass the output from the release_url through | jq . to pretty-print it, it works as expected.

k8s-ci-robot commented 1 year ago

@bewing: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
KnVerey commented 1 year ago

A bunch of us tried to reproduce this during today's bug scrub and we were not able to. Can you please confirm whether you are still experiencing this issue?

/triage needs-information

bewing commented 1 year ago

I am still experiencing this issue.

Here is a gist with the full output from executing the script with -x set: https://gist.github.com/bewing/62f50954a6a8cb9e6ca1e3c6a100fcb7

The machine in question is colocated with ServerCentral in Chicago, IL

bewing commented 1 year ago

As additional information: This is non-deterministic. Once, it did manage to download/install the binary successfully. I think Github or its cache might be doing some A/B testing of removing newlines from API output?

bewing commented 1 year ago

Curious if in your testing performing curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases generates the output as one extremely long line, or as multi-line?

$ curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases | wc -l
0

edit:
again, this is non-deterministic. I tried again just now and got the multi-line response:

$ curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases | wc -l
3572
bewing commented 1 year ago

If newlines are not present, adding jq into the invocation can restore them, and make this work properly:

# non-working
curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases | grep browser_download.*linux_amd64 |  cut -d '"' -f 4 |  sort -V | tail -n 1
https://api.github.com/repos/kubernetes-sigs/kustomize/releases/73452880

# working
$ curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases | jq | grep browser_download.*linux_amd64 |  cut -d '"' -f 4 |  sort -V | tail -n 1
https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v4.5.7/kustomize_v4.5.7_linux_amd64.tar.gz
david-martin commented 1 year ago

I'm also seeing this problem only recently. It definitely worked a few days ago. In my case it's on macos.

+ tar xzf './kustomize_v*_darwin_arm64.tar.gz'
tar: Error opening archive: Failed to open './kustomize_v*_darwin_arm64.tar.gz'

The pipe to jq workaround works for me.

oxr463 commented 1 year ago

So should we modify the script to use jq? What's the current status here?

bewing commented 1 year ago

So should we modify the script to use jq? What's the current status here?

I'm not sure. How can you ensure that jq is present? It looks like the current script tries very hard to use only common binaries (curl, cut, grep). Adding in a new dependency might break a lot of things downstream (operator-sdk, for example) that attempt to use this script o install kustomize for build steps if not present in the path

https://github.com/operator-framework/operator-sdk/blob/7ff900717f9179665dea414ac54ccdedaef2b4fe/internal/plugins/ansible/v1/scaffolds/internal/templates/makefile.go#L127-L139

oxr463 commented 1 year ago

That's a good point. Maybe finding a solution without adding an additional dependency would be the best way forward.

david-martin commented 1 year ago

Maybe json_pp is common enough to use?

curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases | json_pp -json_opt pretty,canonical | grep browser_download.*linux_amd64 | cut -d '"' -f 4 |  sort -V | tail -n 1
KnVerey commented 1 year ago

Indeed, we want to avoid introducing extra dependencies to this script. Although I'd also prefer to avoid adding complexity, if we have no better option, we could check if the most commonly available tool is present, and continue attempting to use the output raw if it isn't (perhaps with a warning message).

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

oxr463 commented 1 year ago

/remove-lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

oxr463 commented 1 year ago

/remove-lifecycle-stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

oxr463 commented 1 year ago

/remove-lifecycle-rotten

oxr463 commented 1 year ago

Ping @bewing

bewing commented 1 year ago

Haven't thought about this in a bit. Might the best approach be to:

k8s-triage-robot commented 12 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 12 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/kustomize/issues/4769#issuecomment-1621851006): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
bewing commented 12 months ago

/reopen

/remove-lifecycle rotten

k8s-ci-robot commented 12 months ago

@bewing: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/kustomize/issues/4769#issuecomment-1621858826): >/reopen > >/remove-lifecycle rotten Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
k8s-triage-robot commented 5 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 3 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/kustomize/issues/4769#issuecomment-2016621597): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
bewing commented 3 months ago

/reopen

/remove-lifecycle rotten

k8s-ci-robot commented 3 months ago

@bewing: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/kustomize/issues/4769#issuecomment-2017896898): >/reopen > >/remove-lifecycle rotten Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
k8s-triage-robot commented 1 week ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale