search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
28.05k stars 2.35k forks source link

Latest `install.sh` exits without error message on airgapped nodes #7247

Closed djpbessems closed 1 year ago

djpbessems commented 1 year ago

Environmental Info: K3s Version: master-branch

Node(s) CPU architecture, OS, and Version: Ubuntu 20.04 LTS

Cluster Configuration: N/A

Describe the bug: When running the latest install.sh with below command, it exits right after showing [INFO] Skipping k3s download and verify without an error message (exit code is not 0 though):

INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC="server --cluster-init --token <token> --tls-san <ipaddress> --disable local-storage --config /etc/rancher/k3s/config.yaml" ./install.sh

Steps To Reproduce: See above

Expected behavior: Installation starts.

Actual behavior: See above

Additional context / logs: The install.sh from tag v1.26.3+k3s1 still works correctly.

/kind bug

brandond commented 1 year ago

cc @galal-hussein

galal-hussein commented 1 year ago

@djpbessems I am not able to reproduce with these following steps:

# curl https://get.k3s.io | INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC="server --cluster-init --token test --tls-san xxxxx --disable local-storage --config /etc/rancher/k3s/config.yaml" sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 29027  100 29027    0     0   383k      0 --:--:-- --:--:-- --:--:--  388k
[INFO]  Skipping k3s download and verify
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

can you make sure that the steps are the same, or am I missing something

brandond commented 1 year ago

@djpbessems You might try running the script with -x to show what command is failing:

INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC="server --cluster-init --token <token> --tls-san <ipaddress> --disable local-storage --config /etc/rancher/k3s/config.yaml" sh -x ./install.sh
djpbessems commented 1 year ago

I realised I forgot to share that this is happening on an air-gapped system.

Since k3s (and kube-vip) needs a default route, I've configured the system with a bogus default route. When I then run the latest version of the install script, it stops with a silent error (last few lines of output when running with -x):

[...]
+ curl+ grep -oP (?<="browser_download_url": ")[^"]*
 -s https://api.github.com/repos/k3s-io/k3s-selinux/releases/latest
+ grep -oE [^\/]+el8\.noarch\.rpm
+ available_version=

When I then replace the default route with a working gateway, the command succeeds again. So the latest changes make k3s unable to be installed in an air-gapped environment despite the INSTALL_K3S_SKIP_DOWNLOAD setting.

brandond commented 1 year ago

@galal-hussein it looks like the change at https://github.com/k3s-io/k3s/commit/027cc187ce9f21157b8d37d62e67ee1c42968b4b#diff-043df5bdbf6639d7a77e1d44c5226fd7371e5259a1e4df3a0dd5d64c30dca44fR500-R504 makes it call out to the GitHub API to retrieve the latest release of the k3s-selinux package, even if it's not going to be shown.

While I do like the idea of having the script call out to get the latest version to suggest, we probably need to make that an optional check with a short timeout, and fall back to a known-good default value. It also shouldn't happen when we're not downloading anything.

Rudedog9d commented 1 year ago

+1 to this issue on CoreOS on versions 37.20230218.3.0 (CoreOS) and 37.20230322.3.0.

Full OS-Release for our primary image

``` NAME="Fedora Linux" VERSION="37.20230218.3.0 (CoreOS)" ID=fedora VERSION_ID=37 VERSION_CODENAME="" PLATFORM_ID="platform:f37" PRETTY_NAME="Fedora CoreOS 37.20230218.3.0" ANSI_COLOR="0;38;2;60;110;180" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:37" HOME_URL="https://getfedora.org/coreos/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/" SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/" BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=37 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=37 SUPPORT_END=2023-11-14 VARIANT="CoreOS" VARIANT_ID=coreos OSTREE_VERSION='37.20230218.3.0' ```

We're also seeing a silent failure at the available_versions step. We were confused by the grep working outside of the script but not in the script - and we realized it's the rpm_target variable causing it to fail.

Here's a bunch of debug info. Let me know if you need something additional.

Last lines of script with `-x`

``` [INFO] Installing k3s to /usr/local/bin/k3s + chown root:root /tmp/k3s-install.dkFHc26FAG/k3s.bin + mv -f /tmp/k3s-install.dkFHc26FAG/k3s.bin /usr/local/bin/k3s + setup_selinux + case ${INSTALL_K3S_CHANNEL} in + rpm_channel=stable + rpm_site=rpm.rancher.io + '[' stable = testing ']' + '[' -r /etc/os-release ']' + . /etc/os-release ++ NAME='Fedora Linux' ++ VERSION='37.20230218.3.0 (CoreOS)' ++ ID=fedora ++ VERSION_ID=37 ++ VERSION_CODENAME= ++ PLATFORM_ID=platform:f37 ++ PRETTY_NAME='Fedora CoreOS 37.20230218.3.0' ++ ANSI_COLOR='0;38;2;60;110;180' ++ LOGO=fedora-logo-icon ++ CPE_NAME=cpe:/o:fedoraproject:fedora:37 ++ HOME_URL=https://getfedora.org/coreos/ ++ DOCUMENTATION_URL=https://docs.fedoraproject.org/en-US/fedora-coreos/ ++ SUPPORT_URL=https://github.com/coreos/fedora-coreos-tracker/ ++ BUG_REPORT_URL=https://github.com/coreos/fedora-coreos-tracker/ ++ REDHAT_BUGZILLA_PRODUCT=Fedora ++ REDHAT_BUGZILLA_PRODUCT_VERSION=37 ++ REDHAT_SUPPORT_PRODUCT=Fedora ++ REDHAT_SUPPORT_PRODUCT_VERSION=37 ++ SUPPORT_END=2023-11-14 ++ VARIANT=CoreOS ++ VARIANT_ID=coreos ++ OSTREE_VERSION=37.20230218.3.0 ++ expr '' : '.*suse.*' + '[' 0 '!=' 0 ']' + '[' 37 = 7 ']' + '[' '' = coreos ']' + '[' coreos = coreos ']' + rpm_target=coreos + rpm_site_infix=coreos + package_installer=rpm-ostree + '[' rpm-ostree = rpm-ostree ']' + '[' -x /bin/yum ']' + '[' rpm-ostree = yum ']' + '[' stable = testing ']' ++ curl -s https://api.github.com/repos/k3s-io/k3s-selinux/releases/latest ++ grep -oP '(?<="browser_download_url": ")[^"]*' ++ grep -oE '[^\/]+coreos\.noarch\.rpm' + available_version= + cleanup + code=1 + set +e + trap - EXIT + rm -rf /tmp/k3s-install.dkFHc26FAG + exit 1 ```

curl -s https://api.github.com/repos/k3s-io/k3s-selinux/releases/latest

``` { "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/74469641", "assets_url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/74469641/assets", "upload_url": "https://uploads.github.com/repos/k3s-io/k3s-selinux/releases/74469641/assets{?name,label}", "html_url": "https://github.com/k3s-io/k3s-selinux/releases/tag/v1.2.stable.2", "id": 74469641, "author": { "login": "brandond", "id": 370103, "node_id": "MDQ6VXNlcjM3MDEwMw==", "avatar_url": "https://avatars.githubusercontent.com/u/370103?v=4", "gravatar_id": "", "url": "https://api.github.com/users/brandond", "html_url": "https://github.com/brandond", "followers_url": "https://api.github.com/users/brandond/followers", "following_url": "https://api.github.com/users/brandond/following{/other_user}", "gists_url": "https://api.github.com/users/brandond/gists{/gist_id}", "starred_url": "https://api.github.com/users/brandond/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/brandond/subscriptions", "organizations_url": "https://api.github.com/users/brandond/orgs", "repos_url": "https://api.github.com/users/brandond/repos", "events_url": "https://api.github.com/users/brandond/events{/privacy}", "received_events_url": "https://api.github.com/users/brandond/received_events", "type": "User", "site_admin": false }, "node_id": "RE_kwDODnar2c4EcFEJ", "tag_name": "v1.2.stable.2", "target_commitish": "master", "name": "v1.2.stable.2", "draft": false, "prerelease": false, "created_at": "2022-06-10T21:43:16Z", "published_at": "2022-08-16T00:40:26Z", "assets": [ { "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818004", "id": 74818004, "node_id": "RA_kwDODnar2c4EdaHU", "name": "k3s-selinux-1.2-2.el7.noarch.rpm", "label": "", "uploader": { "login": "ks3serviceaccount", "id": 74435438, "node_id": "MDQ6VXNlcjc0NDM1NDM4", "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4", "gravatar_id": "", "url": "https://api.github.com/users/ks3serviceaccount", "html_url": "https://github.com/ks3serviceaccount", "followers_url": "https://api.github.com/users/ks3serviceaccount/followers", "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}", "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}", "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions", "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs", "repos_url": "https://api.github.com/users/ks3serviceaccount/repos", "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}", "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events", "type": "User", "site_admin": false }, "content_type": "application/x-rpm", "state": "uploaded", "size": 16240, "download_count": 1014, "created_at": "2022-08-16T00:41:49Z", "updated_at": "2022-08-16T00:41:50Z", "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el7.noarch.rpm" }, { "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818006", "id": 74818006, "node_id": "RA_kwDODnar2c4EdaHW", "name": "k3s-selinux-1.2-2.el7.src.rpm", "label": "", "uploader": { "login": "ks3serviceaccount", "id": 74435438, "node_id": "MDQ6VXNlcjc0NDM1NDM4", "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4", "gravatar_id": "", "url": "https://api.github.com/users/ks3serviceaccount", "html_url": "https://github.com/ks3serviceaccount", "followers_url": "https://api.github.com/users/ks3serviceaccount/followers", "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}", "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}", "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions", "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs", "repos_url": "https://api.github.com/users/ks3serviceaccount/repos", "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}", "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events", "type": "User", "site_admin": false }, "content_type": "application/x-rpm", "state": "uploaded", "size": 18725, "download_count": 69, "created_at": "2022-08-16T00:41:50Z", "updated_at": "2022-08-16T00:41:50Z", "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el7.src.rpm" }, { "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818087", "id": 74818087, "node_id": "RA_kwDODnar2c4EdaIn", "name": "k3s-selinux-1.2-2.el8.noarch.rpm", "label": "", "uploader": { "login": "ks3serviceaccount", "id": 74435438, "node_id": "MDQ6VXNlcjc0NDM1NDM4", "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4", "gravatar_id": "", "url": "https://api.github.com/users/ks3serviceaccount", "html_url": "https://github.com/ks3serviceaccount", "followers_url": "https://api.github.com/users/ks3serviceaccount/followers", "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}", "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}", "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions", "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs", "repos_url": "https://api.github.com/users/ks3serviceaccount/repos", "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}", "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events", "type": "User", "site_admin": false }, "content_type": "application/x-rpm", "state": "uploaded", "size": 20696, "download_count": 3931, "created_at": "2022-08-16T00:43:04Z", "updated_at": "2022-08-16T00:43:04Z", "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el8.noarch.rpm" }, { "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818088", "id": 74818088, "node_id": "RA_kwDODnar2c4EdaIo", "name": "k3s-selinux-1.2-2.el8.src.rpm", "label": "", "uploader": { "login": "ks3serviceaccount", "id": 74435438, "node_id": "MDQ6VXNlcjc0NDM1NDM4", "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4", "gravatar_id": "", "url": "https://api.github.com/users/ks3serviceaccount", "html_url": "https://github.com/ks3serviceaccount", "followers_url": "https://api.github.com/users/ks3serviceaccount/followers", "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}", "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}", "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions", "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs", "repos_url": "https://api.github.com/users/ks3serviceaccount/repos", "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}", "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events", "type": "User", "site_admin": false }, "content_type": "application/x-rpm", "state": "uploaded", "size": 23061, "download_count": 116, "created_at": "2022-08-16T00:43:04Z", "updated_at": "2022-08-16T00:43:04Z", "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el8.src.rpm" }, { "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818174", "id": 74818174, "node_id": "RA_kwDODnar2c4EdaJ-", "name": "k3s-selinux-1.2-2.sle.noarch.rpm", "label": "", "uploader": { "login": "ks3serviceaccount", "id": 74435438, "node_id": "MDQ6VXNlcjc0NDM1NDM4", "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4", "gravatar_id": "", "url": "https://api.github.com/users/ks3serviceaccount", "html_url": "https://github.com/ks3serviceaccount", "followers_url": "https://api.github.com/users/ks3serviceaccount/followers", "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}", "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}", "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions", "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs", "repos_url": "https://api.github.com/users/ks3serviceaccount/repos", "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}", "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events", "type": "User", "site_admin": false }, "content_type": "application/x-rpm", "state": "uploaded", "size": 20958, "download_count": 96, "created_at": "2022-08-16T00:44:42Z", "updated_at": "2022-08-16T00:44:42Z", "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.sle.noarch.rpm" }, { "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818196", "id": 74818196, "node_id": "RA_kwDODnar2c4EdaKU", "name": "k3s-selinux-1.2-2.sle.src.rpm", "label": "", "uploader": { "login": "ks3serviceaccount", "id": 74435438, "node_id": "MDQ6VXNlcjc0NDM1NDM4", "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4", "gravatar_id": "", "url": "https://api.github.com/users/ks3serviceaccount", "html_url": "https://github.com/ks3serviceaccount", "followers_url": "https://api.github.com/users/ks3serviceaccount/followers", "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}", "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}", "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions", "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs", "repos_url": "https://api.github.com/users/ks3serviceaccount/repos", "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}", "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events", "type": "User", "site_admin": false }, "content_type": "application/x-rpm", "state": "uploaded", "size": 23404, "download_count": 60, "created_at": "2022-08-16T00:44:42Z", "updated_at": "2022-08-16T00:44:43Z", "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.sle.src.rpm" }, { "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818007", "id": 74818007, "node_id": "RA_kwDODnar2c4EdaHX", "name": "sha256sum-centos7-noarch.txt", "label": "", "uploader": { "login": "ks3serviceaccount", "id": 74435438, "node_id": "MDQ6VXNlcjc0NDM1NDM4", "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4", "gravatar_id": "", "url": "https://api.github.com/users/ks3serviceaccount", "html_url": "https://github.com/ks3serviceaccount", "followers_url": "https://api.github.com/users/ks3serviceaccount/followers", "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}", "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}", "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions", "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs", "repos_url": "https://api.github.com/users/ks3serviceaccount/repos", "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}", "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events", "type": "User", "site_admin": false }, "content_type": "text/plain; charset=utf-8", "state": "uploaded", "size": 195, "download_count": 185, "created_at": "2022-08-16T00:41:50Z", "updated_at": "2022-08-16T00:41:51Z", "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/sha256sum-centos7-noarch.txt" }, { "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818089", "id": 74818089, "node_id": "RA_kwDODnar2c4EdaIp", "name": "sha256sum-centos8-noarch.txt", "label": "", "uploader": { "login": "ks3serviceaccount", "id": 74435438, "node_id": "MDQ6VXNlcjc0NDM1NDM4", "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4", "gravatar_id": "", "url": "https://api.github.com/users/ks3serviceaccount", "html_url": "https://github.com/ks3serviceaccount", "followers_url": "https://api.github.com/users/ks3serviceaccount/followers", "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}", "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}", "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions", "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs", "repos_url": "https://api.github.com/users/ks3serviceaccount/repos", "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}", "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events", "type": "User", "site_admin": false }, "content_type": "text/plain; charset=utf-8", "state": "uploaded", "size": 195, "download_count": 83, "created_at": "2022-08-16T00:43:04Z", "updated_at": "2022-08-16T00:43:05Z", "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/sha256sum-centos8-noarch.txt" }, { "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818199", "id": 74818199, "node_id": "RA_kwDODnar2c4EdaKX", "name": "sha256sum-microos-noarch.txt", "label": "", "uploader": { "login": "ks3serviceaccount", "id": 74435438, "node_id": "MDQ6VXNlcjc0NDM1NDM4", "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4", "gravatar_id": "", "url": "https://api.github.com/users/ks3serviceaccount", "html_url": "https://github.com/ks3serviceaccount", "followers_url": "https://api.github.com/users/ks3serviceaccount/followers", "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}", "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}", "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions", "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs", "repos_url": "https://api.github.com/users/ks3serviceaccount/repos", "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}", "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events", "type": "User", "site_admin": false }, "content_type": "text/plain; charset=utf-8", "state": "uploaded", "size": 195, "download_count": 62, "created_at": "2022-08-16T00:44:43Z", "updated_at": "2022-08-16T00:44:43Z", "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/sha256sum-microos-noarch.txt" } ], "tarball_url": "https://api.github.com/repos/k3s-io/k3s-selinux/tarball/v1.2.stable.2", "zipball_url": "https://api.github.com/repos/k3s-io/k3s-selinux/zipball/v1.2.stable.2", "body": "## What's Changed\r\n* Use SHA256 to sign packages instead of default SHA1 by @brandond in https://github.com/k3s-io/k3s-selinux/pull/32\r\n* Bump pip/setuptools version; switch to https for git clone by @brandond in https://github.com/k3s-io/k3s-selinux/pull/33\r\n\r\n**Full Changelog**: https://github.com/k3s-io/k3s-selinux/compare/v1.1.stable.1...v1.2.stable.2", "mentions_count": 1 } ```

grep -oP '(?<="browser_download_url": ")[^"]*'

``` curl -s https://api.github.com/repos/k3s-io/k3s-selinux/releases/latest | grep -oP '(?<="browser_download_url": ")[^"]*' https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el7.noarch.rpm https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el7.src.rpm https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el8.noarch.rpm https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el8.src.rpm https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.sle.noarch.rpm https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.sle.src.rpm https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/sha256sum-centos7-noarch.txt https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/sha256sum-centos8-noarch.txt https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/sha256sum-microos-noarch.txt ```

All together now

``` curl -s https://api.github.com/repos/k3s-io/k3s-selinux/releases/latest | grep -oP '(?<="browser_download_url": ")[^"]*' | grep -oE '[^/]+coreos.noarch.rpm' # no return, exit status 1 ```

Rudedog9d commented 1 year ago

We fell back to the previous version of the script for now, as we need to spin up a cluster for a competition this weekend (new cluster each week). Will look out for the fix on this ticket!

Link to previous version of script for anyone looking to do the same: https://raw.githubusercontent.com/k3s-io/k3s/7e59376bb91d451d3eaf16b9a3f80ae4d711b2bc/install.sh

Rudedog9d commented 1 year ago

Also as a note - the script appears to work as is if exporting the variable INSTALL_K3S_CHANNEL=testing, although our automated cluster join step didn't appear to work.

Logs from script

``` [INFO] Using v1.25.4+k3s1 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt [INFO] Skipping binary downloaded, installed k3s matches hash Inactive requests: k3s-selinux (already provided by k3s-selinux-0.5-1.el8.noarch) Checking out tree d34ff10... done Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora rancher-k3s-common-testing updates-archive Updating metadata for 'rancher-k3s-common-testing'... done Importing rpm-md... done rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2022-10-06T11:01:40Z solvables: 4 rpm-md repo 'fedora-modular' (cached); generated: 2022-11-05T07:58:03Z solvables: 1454 rpm-md repo 'updates-modular' (cached); generated: 2023-02-22T11:28:37Z solvables: 1464 rpm-md repo 'updates' (cached); generated: 2023-04-07T02:52:47Z solvables: 28764 rpm-md repo 'fedora' (cached); generated: 2022-11-05T08:04:38Z solvables: 66822 rpm-md repo 'rancher-k3s-common-testing'; generated: 2023-02-09T22:23:18Z solvables: 2 rpm-md repo 'updates-archive' (cached); generated: 2023-04-07T03:57:22Z solvables: 36528 Resolving dependencies... done Checking out packages... done Running pre scripts... done Running post scripts... done Running posttrans scripts... done Writing rpmdb... done Writing OSTree commit... done Staging deployment... done Changes queued for next boot. Run "systemctl reboot" to start a reboot [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service [INFO] systemd: Enabling k3s-agent unit Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service. ```

fieldju commented 1 year ago

Not sure if this is realated but I am seeing today that curl -sfL https://get.k3s.io | bash - started to fail intermittently but its not exiting 1

When I SSH'd on to the EC2 instance I had to run the install script by hand a few times before it worked image

I'm trying to work around it with some retries in my script, but if this is round robin'ing to some old script thats in cache or what not then retry'ing will start failing

echo "INSTALLING_K3S" > /home/admin/status

install_k3s() {
  echo "installing k3s"
  curl -sfL https://get.k3s.io | bash -
  sudo chown -R admin:admin /etc/rancher/k3s
  echo "finished installing k3s"
}

# Install K3s with 20 retries
for i in {1..20}; do install_k3s && break || sleep 1; done

echo "INSTALLING_ADMISSION_CONTROLLER" > /home/admin/status

I might look into https://github.com/k3s-io/k3s/issues/7247#issuecomment-1500711907 and see if I can pin to a known working script.

fieldju commented 1 year ago

We fell back to the previous version of the script for now, as we need to spin up a cluster for a competition this weekend (new cluster each week). Will look out for the fix on this ticket!

Link to previous version of script for anyone looking to do the same: https://raw.githubusercontent.com/k3s-io/k3s/7e59376bb91d451d3eaf16b9a3f80ae4d711b2bc/install.sh

FWIW I tried pinning to this version and was still experiencing my issue.

Rudedog9d commented 1 year ago

@fieldju sounds like you're having a different issue. Our experienced of it has been completely consistent thus far.

brandond commented 1 year ago

@fieldju that is unrelated. The K3s channel server is on infrastructure that is occasionally overloaded when Rancher releases chart updates.

terrytangyuan commented 1 year ago

Not sure if this is realated but I am seeing today that curl -sfL https://get.k3s.io | bash - started to fail intermittently but its not exiting 1

I am seeing the same issue. https://github.com/argoproj/argo-workflows/actions/runs/4641097132/jobs/8217213836

Any suggestions?

What I did was as usual:

Run curl -sfL https://get.k3s.io/ | INSTALL_K3S_VERSION= INSTALL_K3S_CHANNEL=stable INSTALL_K3S_EXEC=--docker K3S_KUBECONFIG_MODE=644 sh -
[INFO]  Finding release for channel stable
[INFO]  Using stable as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/stable/sha256sum-amd64.txt
Error: Process completed with exit code 22.
brandond commented 1 year ago

@terrytangyuan see my message directly above yours.

sradigan commented 1 year ago

I just pushed up a commit with some changes that I think will help with some if the issues here.

I'll test it and make a pull request if this at least resolves the coreos issue.

https://github.com/k3s-io/k3s/commit/1b1594bffc27309cfa8b56869b02c61c2c2219e6

Edit: Note that the boolean on the "testing" check is != (not equals) rather than = (equals). The block was nested in the else so that the ENV var could be set on the air-gapped system to skip the curl to the net.

terrytangyuan commented 1 year ago

Thanks. Looks like stable is no longer available. I pinned the CI to a specific version.

jgbright commented 1 year ago

I am seeing an error that may have the same root cause, related to using the -P option with grep. I started seeing this on Friday, April 7. The timing of when the error started seems to coincide with the code change referenced earlier in the thread.

root@qemuarm64:~# INSTALL_K3S_EXEC="--snapshotter=native" INSTALL_K3S_SKIP_DOWNLOAD=true ./k3s_install.sh
[INFO]  Skipping k3s download and verify
grep: invalid option -- 'P'
BusyBox v1.31.1 (2022-12-13 23:09:41 UTC) multi-call binary.

Usage: grep [-HhnlLoqvsriwFE] [-m N] [-A/B/C N] PATTERN/-e PATTERN.../-f FILE [FILE]...
brandond commented 1 year ago

cc @galal-hussein we should also not expect that any non-posix grep options are available, such as -P which is a gnu extension to support PCREs.

bguzman-3pillar commented 1 year ago

Validated on v1.26.4-rc1+k3s1

$ k3s -v
k3s version v1.26.4-rc1+k3s1 (257fa2c5)
go version go1.19.8

Environment Details

Infrastructure

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="9.1 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.1 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.1
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"

Cluster Configuration:

1 server

Config.yaml:

selinux: true

Testing Steps

  1. Create Airgapped environment
  2. Get all the relevant files from bastion: 
    - Install script: wget -O k3s_install.sh [get.k3s.io](http://get.k3s.io/)
- Version used: wget https://github.com/k3s-io/k3s/releases/download/v1.26.4-rc1%2Bk3s1/k3s-airgap-images-amd64.tar.gz
- K3S binary: wget https://github.com/k3s-io/k3s/releases/download/v1.26.4-rc1%2Bk3s1/k3s
  3. Copy all of that to airgapped node
  4. Run this (Steps here https://docs.k3s.io/installation/airgap#installing-k3s-in-an-air-gapped-environment) 
    sudo mkdir -p /var/lib/rancher/k3s/agent/images/
sudo cp ./k3s-airgap-images-amd64.tar.gz /var/lib/rancher/k3s/agent/images/
  5. Copy the config.yaml file 
    sudo mkdir -p /etc/rancher/k3s
sudo cp config.yaml /etc/rancher/k3s/config.yaml
  6. Install k3s using this INSTALL_K3S_SKIP_DOWNLOAD=true ./k3s_install.sh
  7. Check everything is up and running kubectl get node -A -o wide

Validation Results:

$ cat /etc/rancher/k3s/config.yaml
selinux: true
$ INSTALL_K3S_SKIP_DOWNLOAD=true ./k3s_install.sh
[INFO]  Skipping k3s download and verify
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
$ kubectl get node,all -A 
NAME                                               STATUS   ROLES                  AGE   VERSION
node/ip-172-31-34-209.us-east-2.compute.internal   Ready    control-plane,master   20m   v1.26.4-rc1+k3s1

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE
kube-system   pod/coredns-59b4f5bbd5-c9nhj                  1/1     Running     0          20m
kube-system   pod/local-path-provisioner-76d776f6f9-qf9tb   1/1     Running     0          20m
kube-system   pod/helm-install-traefik-crd-tfsq2            0/1     Completed   0          20m
kube-system   pod/svclb-traefik-0ef1a7a1-fk2pq              2/2     Running     0          20m
kube-system   pod/traefik-56b8c5fb5c-8bb4z                  1/1     Running     0          20m
kube-system   pod/helm-install-traefik-sc8sz                0/1     Completed   1          20m
kube-system   pod/metrics-server-7b67f64457-hnkjp           1/1     Running     0          20m

NAMESPACE     NAME                     TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
default       service/kubernetes       ClusterIP      10.43.0.1      <none>          443/TCP                      20m
kube-system   service/kube-dns         ClusterIP      10.43.0.10     <none>          53/UDP,53/TCP,9153/TCP       20m
kube-system   service/metrics-server   ClusterIP      10.43.13.87    <none>          443/TCP                      20m
kube-system   service/traefik          LoadBalancer   10.43.97.234   172.31.34.209   80:31066/TCP,443:30999/TCP   20m

NAMESPACE     NAME                                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/svclb-traefik-0ef1a7a1   1         1         1       1            1           <none>          20m

NAMESPACE     NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns                  1/1     1            1           20m
kube-system   deployment.apps/local-path-provisioner   1/1     1            1           20m
kube-system   deployment.apps/traefik                  1/1     1            1           20m
kube-system   deployment.apps/metrics-server           1/1     1            1           20m

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-59b4f5bbd5                  1         1         1       20m
kube-system   replicaset.apps/local-path-provisioner-76d776f6f9   1         1         1       20m
kube-system   replicaset.apps/traefik-56b8c5fb5c                  1         1         1       20m
kube-system   replicaset.apps/metrics-server-7b67f64457           1         1         1       20m

NAMESPACE     NAME                                 COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik-crd   1/1           18s        20m
kube-system   job.batch/helm-install-traefik       1/1           21s        20m