Permissions to add minikube packages to Kubernetes repos (.deb, .rpm)

tstromberg commented 5 years ago

Hello!

We (minikube) would like to publish our .deb and .rpm packages to the official Kubernetes repositories as part of our release process. How do we get started?

Currently our release scripts publish to a Google Storage bucket:

https://github.com/kubernetes/minikube/blob/551b164017d2649a3857facb79ba80d00569fa18/hack/jenkins/release_build_and_upload.sh#L45

tstromberg commented 5 years ago

/cc @afbjorklund @justaugustus

afbjorklund commented 5 years ago

https://storage.googleapis.com/minikube/releases/v1.3.0/minikube_1.3.0.deb https://storage.googleapis.com/minikube/releases/v1.3.0/docker-machine-driver-kvm2_1.3.0.deb

https://storage.googleapis.com/minikube/releases/v1.3.0/minikube-1.3.0.rpm https://storage.googleapis.com/minikube/releases/v1.3.0/docker-machine-driver-kvm2-1.3.0.rpm

tstromberg commented 5 years ago

Clarifying title. I could use some help with getting the appropriate permissions. Alternatively, we could create our own repo, but it would be nice to preserve the maintainability benefits of having a single Kubernetes package repository.

tstromberg commented 5 years ago

/cc @justaugustus

Now that we are post-v1.16 release, does the release team have the cycles to help make this happen?

tstromberg commented 5 years ago

If #sig-release prefers, an alternative approach would be pulling files from our GCS or GitHub file stores. This would introduce a post-build race condition, but I'm happy with any solution so long as the deb files get published before we make public announcements.

medyagh commented 4 years ago

any update on this ?

afbjorklund commented 4 years ago

I guess there is no update on this, and it won't be possible to easily install minikube 1.8.0 ?

Only change since last time, was that we added the architecture to the package file names:

https://github.com/kubernetes/minikube/releases/download/v1.7.0/minikube_1.7.0-0_amd64.deb

https://github.com/kubernetes/minikube/releases/download/v1.7.0/minikube-1.7.0-0.x86_64.rpm

tstromberg commented 4 years ago

Is there anything I or others can do to help with the signing infrastructure? This issue has now been open for 244 days without a response on GitHub, though I did have a chance to briefly bring this up with @justaugustus at KubeCon in November.

Even if it takes a Googler to push a button to sign the key, that's fine with me to get this thing going.

justaugustus commented 4 years ago

From #sig-release Slack thread:

@tstromberg -- Sorry for letting this slip. The tl;dr:

SIG Release does not control the apt/yum repos that we publish packages to. Google does. So deb/rpm publishing that happens every cycle is done happens on the Google side.
The current package signing key is also Google-owned
Today, we don't have a process for enabling arbitrary Kubernetes projects to push to those repos
If we did, I would suggest using a signing key that doesn't imply SIG Release/Release Engineering is on the hook for issues in publication
We're planning to move Kubernetes to community-owned infra, including the package repos, but apt/yum repos are lower priority compared to other improvements we're trying to make

ref on the current work: https://github.com/kubernetes/release/issues/911 ref on future work: https://github.com/kubernetes/release/issues/913

Unfortunately, I don't have a good answer for you right now, given the other more pressing projects.

Potential solutions:

rolling your own
requesting a separate Google-owned repo for minikube

cc: @kubernetes/build-admins

afbjorklund commented 4 years ago

The main downside is that user would have to install two separate keys, but that seems to be the same situation for both solutions since if it was the same key used for new repo it could go into the main...

https://packages.cloud.google.com/apt/doc/apt-key.gpg https://packages.cloud.google.com/yum/doc/yum-key.gpg

pub   2048R/A7317B0F 2015-04-03 [expired: 2018-04-02]
uid                  Google Cloud Packages Automatic Signing Key <gc-team@google.com>

pub   2048R/BA07F4FB 2018-04-01 [expires: 2021-03-31]
uid                  Google Cloud Packages Automatic Signing Key <gc-team@google.com>

So I guess minikube will have to invent some local process, that will be a part of the release build. And then find some hosting for both the (public) gpg keys as well as the rest of the apt/yum index files...

afbjorklund commented 4 years ago

The workaround is still the same as before, user will have to download and install unsigned packages:

https://github.com/kubernetes/minikube/releases/download/v1.9.2/minikube_1.9.2-0_amd64.deb

https://github.com/kubernetes/minikube/releases/download/v1.9.2/minikube-1.9.2-0.x86_64.rpm

Usually this leads to a warning from the web browser, and then another one from the pkg installer...

So in that sense it will be the same as the Mac and Win experience, outside their "walled gardens":

Apple Gatekeeper
Microsoft Smartscreen

They will also have to repeat the process for every minor version, since there is no update/upgrade.

afbjorklund commented 4 years ago

Now it is blocked in the Ubuntu Software as well, so users will need to use the Terminal...

https://en.wikipedia.org/wiki/Snapcraft (https://snapcraft.io/)

Still works in Fedora, but not sure for how long. It is possible that they will move to Flathub.

https://en.wikipedia.org/wiki/Flathub (https://flathub.org/)

It is getting harder and harder to distribute software, especially without being able to sign it. We still have a workaround in the command-line and in the independent package managers*.

* Like Homebrew for Mac and Chocolatey for Win

Might have to find something similar for Linux as well...

medyagh commented 4 years ago

any update on this issue ?

afbjorklund commented 4 years ago

Can we close this ?

It's obviously not going anywhere, and minikube is not in of the upstream release or even a "supported" kubeadm distribution.

Potential solutions:

rolling your own

Until we have fixed the issues with all other package managers, I guess the users are back to curl and the GitHub releases ?

 curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
 sudo install minikube-linux-amd64 /usr/local/bin/minikube

tstromberg commented 4 years ago

I believe this is still something we should aim for, though ultimately, I agree that the goal is to ensure that the minikube .deb & .rpm packages are hosted somewhere that supports an automated update mechanism. If we have to create our own repo, it'll just take longer to achieve.

Shared infra for package hosting would be wonderful.

blueelvis commented 4 years ago

We also have the choice of removing the installations. Basically, removing all these installers and then just maintaining the script based installation like Anders has mentioned above. For Windows, we could switch exclusively to choco or PowerShell script.

afbjorklund commented 3 years ago

We also have the choice of removing the installations.

a.k.a. "give up"

Ultimately this comes down to how we want users to install and update minikube. Currently, the process is quite awful. The benefit of the packages is that they are compressed and checksummed, without requiring too much from the user.

Currently using LinuxBrew is actually a nicer experience for doing the installation, than the system package manager... This is because there are no minikube packages available from the system (Ubuntu) nor from the vendor (Kubernetes)

brew install minikube

We should probably just close these improvement issues...

And have the user use the command-line forever to install them, without the package signing and the automatic updates. I used https://bintray.com for my own personal packages, but I'm not sure that we want to go there for the official releases.

For next release we want to start promoting the arm64 binaries and packages, in addition to the existing amd64 binaries. This will only make the situation worse, since now that is one more step that the poor user will have to get involved in...

https://github.com/kubernetes/minikube/releases/download/v1.16.0/minikube_1.16.0-0_amd64.deb https://github.com/kubernetes/minikube/releases/download/v1.16.0/minikube_1.16.0-0_arm64.deb

https://github.com/kubernetes/minikube/releases/download/v1.16.0/minikube-1.16.0-0.x86_64.rpm https://github.com/kubernetes/minikube/releases/download/v1.16.0/minikube-1.16.0-0.aarch64.rpm

Using apt and yum would have fixed this too, automatically.

mgabeler-lee-6rs commented 3 years ago

We should probably just close these improvement issues...

It is not a good look to your users for a project of this scale and large scale backing to conclude it is unable to stand up a simple pair of apt & yum repositories. I understand that things are complicated, esp. when dealing with large enterprises & code signing systems there, but having a cloud hosting technology project give up on ... setting up cloud hosting is incredibly bad optics.

Currently using LinuxBrew is actually a nicer experience for doing the installation

Maybe if you already use LinuxBrew. If you don't, asking users to install a new package manager just to install your one package is still pretty awful, esp. when they may already be annoyed by having to have 3-4 other package managers already installed as part of their distro (apt/yum, snap, flatpak, ...)

I used https://bintray.com for my own personal packages, but I'm not sure that we want to go there for the official releases.

Lots of open source projects leverage that and similar services for this purpose. If getting things hooked up with the Google processes is going to be slow, using that as at least an intermediate solution seems completely reasonable to me as a user. My only question would be whether minikube fits within the 1TB/month download quota they have, but a quick skim through https://api.github.com/repos/kubernetes/minikube/releases suggests that this project would be well within those limits.

afbjorklund commented 3 years ago

Currently using LinuxBrew is actually a nicer experience for doing the installation

Maybe if you already use LinuxBrew

I meant that the experience for users using brew, is nicer than the experience for users of apt and yum. This is mostly because they cannot just use "apt" or "yum", but have to download and use "dpkg" or "rpm".

If it had a reasonable amount of effort to add the repository and required keys, then it would compare... Like https://kubernetes.io/docs/tasks/tools/install-kubectl/ when compared to brew install kubectl ?

Other than that, there is no way to compare - without getting minikube accepted into all the distributions. But that has other implications, when it comes to update frequency and availability due to e.g. go versions

For the time being, promoting the binary as the default distribution method seems to be the "easiest". Then we can dream of something different, like with https://apps.0install.net/kubernetes/minikube.xml

mgabeler-lee-6rs commented 3 years ago

I meant that the experience for users using brew, is nicer than the experience for users of apt and yum. This is mostly because they cannot just use "apt" or "yum", but have to download and use "dpkg" or "rpm".

This is, at best, an extremely misleading statement

apt uses dpkg under the hood. Nobody would ever have to "download" dpkg. You cannot remove dpkg from a Debian-based system
Similarly I don't think you can remove rpm from a RedHat-based system. yum fronts for rpm here AFAIK, much like apt fronts for dpkg
I don't know if you can use yum to directly install a downloaded .rpm file, but you can definitely use apt to install downloaded .deb files, benefitting from all of apt's dependency resolution logic. So, e.g., being a Debian user I typically update minikube via sudo apt install ~/Downloads/minikube-....deb

mgabeler-lee-6rs commented 3 years ago

If it had a reasonable amount of effort to add the repository and required keys, then it would compare... Like https://kubernetes.io/docs/tasks/tools/install-kubectl/ when compared to brew install kubectl ?

Check out how Chrome, VS-Code, and many other "third party" apps do this. The package itself has a post-install script which handles adding the repo, and a crontab to ensure the repo config is kept up to date. The user only needs to download and install the package ones, and thereafter the OS-driven regular system updates keep it up to date. :tada:

afbjorklund commented 3 years ago

I meant that the experience for users using brew, is nicer than the experience for users of apt and yum. This is mostly because they cannot just use "apt" or "yum", but have to download and use "dpkg" or "rpm".

This is, at best, an extremely misleading statement

I meant that you have to 1) download the package file yourself and 2) run a command on the file. Instead of just being able to install (and upgrade!), if we offered actual apt and yum repositories.

I think you are right that we could use apt installand yum localinstall instead of dpkg and rpm. I mostly meant the two-step CLI process, unfortunately using "double-click" doesn't work anymore.

mgabeler-lee-6rs commented 3 years ago

I meant that you have to 1) download the package file yourself and 2) run a command on the file.

Most users will have a DE installed and they'll just be able to (double) click to open the downloaded package which will open it in a graphical package manager with a nice big "install" button, or so I would think. That's certainly how things work on Gnome-ish systems, I'm assuming KDE behaves similarly.

Instead of just being able to install (and upgrade!), if we offered actual apt and yum repositories.

Definitely agree this is preferred ... my earlier notes were more than the one-time install can configure the repo for a user automatically, so the complex steps you linked from the main kubernetes installation instructions are not necessarily something a user has to do themselves.

unfortunately using "double-click" doesn't work anymore.

Why? It works here. Granted I'm a terminal person so I don't use that capability, but it does work on my system. The .deb file opens with gnome-software which shows the package info (version, description, license) and a big friendly button to act on it ... in my case it's a big Remove button because I already have it installed, but if I didn't it'd be a big friendly Install button.

afbjorklund commented 3 years ago

The feature was disabled in Ubuntu 20.04, in order to promote the Ubuntu Software app store...

It used to work like you say. We have similar issues with the Mac and Win downloads not working.

mgabeler-lee-6rs commented 3 years ago

Aah, I wasn't aware of that change in Ubuntu. Seems rather user-hostile, which ... well that's part of why I use Debian :)

At any rate, if the package post-install script can setup the apt/yum repo, then the user would only need to do the CLI steps once, which would be a big improvement :)

afbjorklund commented 3 years ago

Aah, I wasn't aware of that change in Ubuntu. Seems rather user-hostile, which ... well that's part of why I use Debian :)

Getting the software accepted into Debian is a much longer journey, especially with the insane amount of dependencies.

For other distributions we get away with static linking and providing binaries, but that's just not the way that Debian rolls.

Not that I think that anyone actually tried (to add minikube) ?

mgabeler-lee-6rs commented 3 years ago

Getting the software accepted into Debian is a much longer journey, especially with the insane amount of dependencies.

Yes, was not suggesting trying to get this accepted into Debian, just that the type of user-hostile change you described from Ubuntu is why I don't use Ubuntu.

afbjorklund commented 3 years ago

... the type of user-hostile change you described from Ubuntu is why I don't use Ubuntu.

And of course the Snap wasn't updated either, because they were promoting Microk8s instead. :roll_eyes:

So in the end we ended up deleting it (put it out of its misery). See ~~https://github.com/kubernetes/minikube/issues/7977~~

Which might be the way to "fix" the .deb and .rpm installations as well. Instead of fighting windmills.

The actual binary is the same either way, so we will just focus on producing those - "upstream".

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

afbjorklund commented 3 years ago

@tstromberg Can we close this ticket ? ("Wont Do")

We have called out for the web designers to make our workarounds look slightly less awful, rather than hoping for support for any official packages. The old documentation was removed*.

https://minikube.sigs.k8s.io/docs/start/

https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb https://storage.googleapis.com/minikube/releases/latest/minikube-latest.x86_64.rpm

https://storage.googleapis.com/minikube/releases/latest/minikube_latest_arm64.deb https://storage.googleapis.com/minikube/releases/latest/minikube-latest.aarch64.rpm

BTW; Bintray is closing down services (May 1, 2021).

* The installation has been removed from the main docs:

~~https://v1-18.docs.kubernetes.io/docs/tasks/tools/install-minikube/~~ ~~https://kubernetes.io/docs/setup/learning-environment/~~

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/ https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/ https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/

tstromberg commented 3 years ago

While I still feel like minikube (as well as kubectl or other command-line utilities) should have an apt/yum repository to support automatic upgrades, it's clear that this request isn't going anywhere, so I'm going to close it for now.

I believe the right path going forward will be for minikube to establish its own repository.

dims commented 3 years ago

@tstromberg if you want to use any GCP assets for your own repository, do please ask in kubernetes/k8s.io and we'll figure out a way to support you.

justaugustus commented 2 years ago

Another request: https://kubernetes.slack.com/archives/CJH2GBF7Y/p1647903600618439

afbjorklund commented 2 years ago

@klaases we already gave up on this

kubernetes / release

Permissions to add minikube packages to Kubernetes repos (.deb, .rpm) #839