search

jupyterhub / team-compass

A repository for team interaction, syncing, and handling meeting notes across the JupyterHub ecosystem.
http://jupyterhub-team-compass.readthedocs.io
62 stars 33 forks source link

Move from dockerhub to quay.io (and consider we got an 800$ charge!) #688

Open yuvipanda opened 1 year ago

yuvipanda commented 1 year ago

@choldgraf's card is apparently attached to the jupyterhub org on dockerhub, and it just got an 800$ charge!

image

This is not supposed to happen, as we have 'sponsored oss' status (I'm not sure who got that for us, but THANK YOU to whoever that was).

While I guess this charge is a mistake on their part, it at least feels sufficient motivation for me to move us completely over to quay.io. We already moved repo2docker (https://github.com/jupyterhub/repo2docker/issues/1076#issuecomment-920700356), and while I can't find the issue right now, the 'sponsored OSS' status was why we didn't move everything else. Time to move it now I think.

Repos to move

sgibson91 commented 1 year ago

Cc @consideRatio and @mathbunnyru who lead that work

consideRatio commented 1 year ago

Was @choldgraf's card charged and is it now removed so it can't be charged going onards?

image

image

minrk commented 1 year ago

👍 to moving everything to quay and leaving docker. I don't think anything's been charged, but let's make sure we don't give them a chance to charge in the future.

mathbunnyru commented 1 year ago

I see this charge on the jupyter DockerHub account as well (also unsuccessful), so we will also have to move it.

I think I understand what's going on.

https://www.docker.com/community/open-source/application/

Core contributors of your project namespace get access to a year-long Docker Team subscription

https://github.com/jupyterhub/team-compass/issues/559#issuecomment-1281606540

And, based on this message, it was exactly one year ago when the upgrade to Docker Team happened. I didn't think that it would only last a year, and then Docker would charge us for the next year - doesn't really sound like a "Docker-Sponsored Open Source Program", more like "Docker Team One Year Trial". I missed it a year ago, I'm so sorry. Hope no one was charged (at least DockerHub says the payment failed, which is good).

mathbunnyru commented 1 year ago

👍 to moving the jupyter account to quay as well.

I will work on jupyter/docker-stacks move this or next week (I suppose it will be the most difficult repo to move, so it will take a bit of time).

manics commented 1 year ago

Could we push the important images (mostly jupyterhub) to both docker hub and quay.io for a release cycle or two, but update all docs, config and example repos to use quay.io? That'll ensure anyone pulling/checking latest will still get updates in the short term.

choldgraf commented 1 year ago

I've sent a support request to Docker asking them WTF is going on. I'll let you all know if I hear back. I definitely agree we should just leave the platform because this is too much hassle if there are alternatives that work just as well.

I'd suggest we:

manics commented 1 year ago

I don't think we need another year of service (though there's no harm if someone wants to apply). It means people may hit the rate limits when pulling, but hopefully that means they'll look at the readme and see that they should move to quay.io

yuvipanda commented 1 year ago

I built a small tool that'll migrate some but not all tagged images across the registries! https://github.com/yuvipanda/move-off-dockerhub.

I've moved k8s-hub image's released versions only over to quay.io now, so if people do end up being throttled by dockerhub, they can simply point to quay.io and get the exact same image (as it's copied).

image

I'll do this for all the z2jh and chp images, and the JupyterHub ones as well.

@manics we can do this in reverse for a bit to dockerhub as well.

mathbunnyru commented 1 year ago

@yuvipanda please, change the email of the jupyterhub quay.io account to mathbunnyru+jupyterhub@gmail.com. It will only change the icon for the organization (and the images).

mathbunnyru commented 1 year ago

After 2 days of work with ~25 commits, I've finally switched http://github.com/jupyter/docker-stacks/ to Quay.io. An example: https://quay.io/repository/jupyter/base-notebook Only 1 of these 25 commits is actually renaming (mostly, sed-like), all the others are gonna stay if we ever decide to change the registry again.

Cool features work fine too - automatic registry description update, docker manifest to create multi-arch images, testing contributed recipes, and automatic wiki update.

Nice features of Quay.io I noticed:

Things I don't like/don't work:

choldgraf commented 1 year ago

I went ahead and applied for another year of Open Source account status with DockerHub, and got an e-mail that it was approved today. Could somebody confirm that we do in-fact have this status now?

consideRatio commented 1 year ago

image

Not yet, but I recall that this wasn't reflected instantly last time either.

manics commented 1 year ago

I received an email yesterday:

Hello jupyterhub!

We’re sad to see you go. This email confirms that your Docker Team - Annual subscription for your account has been canceled.

🤷🏽

consideRatio commented 1 year ago

Dockerhub status looks like this:

image

I received this email:

image

yuvipanda commented 1 year ago

At least once the JupyterHub image moves are complete, we should go to all the READMEs in the jupyterhub dockerhub and write them out as redirects.

yuvipanda commented 1 year ago

Ok, everything that needs a PR now has a PR :) Reviews would be appreciated!

mathbunnyru commented 1 year ago

Ok, everything that needs a PR now has a PR :) Reviews would be appreciated!

Done. Please, ping me in PRs if I missed something.

Carreau commented 1 year ago

I would also suggest contact the Jupyter Media strategy team to have them broadcast on various channels that you are moving to Quay

chris-bateman commented 1 year ago

This is obviously still a WIP but this just caused issues on a few of our pipelines. There really needs to be a widespread announcement and migration guide put out.

minrk commented 1 year ago

@chris-bateman we're working on it, but I'm curious how this caused any issues for you. It shouldn't have affected any existing images as far as I can tell.

mathbunnyru commented 1 year ago

We'll have to search our repos for read-usages of Docker images and update them as well. An example: jupyterhub/jupyterhub-deploy-docker: https://github.com/jupyterhub/jupyterhub-deploy-docker/pull/133

choldgraf commented 1 year ago

I wonder if the issues that @chris-bateman ran into were because of the momentary lapse in our DockerHub account status, because of the rejected credit card payment the tried to make.

Either way, right now we should just have another year of DockerHub's service, so nothing should be changed for the time being. Agreed that we should communicate the decision to migrate (within our capacity to sustainably do this...)

chris-bateman commented 1 year ago

I'm sorry for the lack of detail in my above comment. The issue did turn out to be something else related to this PR https://github.com/jupyter/docker-stacks/pull/2006 Resolved on our end due to some hard coded assumptions. But we did see a clone issue of the docker image during one of our builds. It might have been due to the above DockerHub status but I can't comment on the exact cause.

mathbunnyru commented 1 year ago

I applied for open source program for jupyter Docker Hub organization a few days ago (https://github.com/jupyterhub/team-compass/issues/694) and today received an approval.

I see that our images still show a Sponsored OSS badge, but we're on a free Docker Free Team plan.

Screenshot 2023-11-07 at 13 15 01
yuvipanda commented 1 year ago

Great to hear that both jupyter and jupyterhub are still part of the program.

I do think we should still migrate the 'official' source to be quay.io - so we don't have to do this every year, and at the risk that at some point soon we'll just get back a 'no'

yuvipanda commented 1 year ago

ok, everything has moved now! \o/

Next:

yuvipanda commented 1 year ago

Spoke too soon - mybinder.org-deploy also needs to move, but there are no external consumers of that image.

mathbunnyru commented 1 year ago

ok, everything has moved now! \o/

Next:

  • [ ] Make a note in each dockerhub description that the image has moved to quay.io
  • [ ] Copy the descriptions for these images from dockerhub to quay.io

@yuvipanda I highly recommend not doing it manually, because we might delete/add some badges in the READMEs and sometimes text is changed. There is a update-container-description-action. It works with both Docker Hub and Quay.io with slightly different setup. Here is an example how Jupyter Docker Stacks use it: https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/registry-overviews.yml

miraculixx commented 1 year ago

ok, everything has moved now! \o/

I appreciate the move to quay.io and everyone's effort to make this happen. I just came across this update by chance, noticing the reference to quay.io in the docs, and I'm wondering about the scope of the migration strategy.

Specifically I noticed that, as of writing this, only the images tagged with python-3.11 are available from the quay.io repo. For example, docker.io/jupyter/datascience-notebook:python-3.10.11 does not seem to be available on quay.io.

Is this intended? If yes, what is the policy going forward as to which Python versions will be built/provided on quay.io v.s. docker hub, or will the latter be shut down, that is all previous images would be removed? Thank you

yuvipanda commented 1 year ago

@mathbunnyru just as I moved old jupyterhub images to quay.io, I want to move old jupyter/docker-stacks images as well. Do you have objections to that?

mathbunnyru commented 1 year ago

@mathbunnyru just as I moved old jupyterhub images to quay.io, I want to move old jupyter/docker-stacks images as well. Do you have objections to that?

I have a few:

  1. We don’t have classical releases. And we build at least weekly, meaning we have much more images than other projects.
  2. Some of our images are quite large. I suppose combining with first issue it will be difficult to move all of them.
  3. Our current tags in Quay.io definitely clash with our previous tags with Docker Hub. We shouldn’t rewrite these tags with older images.
  4. There were lots of incorrect tags in DockerHub till 2022-07-05.
  5. Till that date our images were sometimes not inherited properly one from another. So, a new image was inherited from an old one, for example.

I didn’t want this mess to be present in Quay.io.

@miraculixx answering your questions.

I intentionally didn’t push old images to Quay.io, mostly for the reasons mentioned above. All new updates will only be provided on Quay.io. At the same time, we do not plan to shut down Docker Hub - it will definitely affect our users in a bad way and we don’t want that. At the same time I expect most of our users to move to Quay.io images in a few years, especially because of python eol.

manics commented 1 year ago

@mathbunnyru what do you think about migrating just the "old" images mentioned on https://jupyter-docker-stacks.readthedocs.io/en/latest/#using-old-images?

mathbunnyru commented 1 year ago

@mathbunnyru what do you think about migrating just the "old" images mentioned on https://jupyter-docker-stacks.readthedocs.io/en/latest/#using-old-images?

I think that’s a good idea.

mathbunnyru commented 1 year ago

@mathbunnyru what do you think about migrating just the "old" images mentioned on https://jupyter-docker-stacks.readthedocs.io/en/latest/#using-old-images?

@manics @yuvipanda @miraculixx done in https://github.com/jupyter/docker-stacks/pull/2032 There is also a small workflow - so if we ever need to move something, we can reuse it.

In a few days I will update the table and the information about using Docker Hub for old images.

Update: done

yuvipanda commented 1 year ago

Can someone else move the manifests / descriptions from dockerhub to quay.io? I don't think I'll have time to do that, manually or automatically, for at least another week or so :(

mathbunnyru commented 1 year ago

Can someone else move the manifests / descriptions from dockerhub to quay.io? I don't think I'll have time to do that, manually or automatically, for at least another week or so :(

@yuvipanda done for 4 images in jupyterhub repo, please, take a look: https://github.com/jupyterhub/jupyterhub/pull/4634

manics commented 1 year ago

Did you intentionally omit the arm64 images when copying the old docker-stack images?

mathbunnyru commented 1 year ago

Did you intentionally omit the arm64 images when copying the old docker-stack images?

Nope, that's a bug, thank you!

Do you maybe know an easy way to fix this? Afaik, I can specify arch, when pulling, but only one, and that doesn't help much.

I know this solution:

  1. Pull aarch64-<tag> and x86_64-<tag> single arch images
  2. Push these images with only changing the registry
  3. Use docker manifest to merge these images and create one multi-platform image <tag>. The previous step is needed otherwise docker manifest tells something about security.

Update: Implemented this solution: https://github.com/jupyter/docker-stacks/pull/2035 But I don't really like it - it doesn't handle a case where an image exists only for one arch

manics commented 1 year ago

@yuvipanda suggested using skopeo copy --multi-arch all in https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/3254#issuecomment-1767746273

mathbunnyru commented 1 year ago

@manics thanks. I fixed the problem.

consideRatio commented 12 months ago

I'd like to publish to both DockerHub and Quay.io instead of switching over fully, is that okay?

Both of these container registries are free services under no obligation to stay functional and free for us to publish to and users to pull from, so it could make sense to publish images to more than one container registry to allow us to switch over if there are issues for some or all users.

Publishing to both seems like a kinder transition than to stop publishing to docker hub, as that doesn't break workflows like building and using a custom image for each release.

As an example, here is a PR for configurable-http-proxy that accomplishes this, its a minimal change that will only add the extra upload time.

image

consideRatio commented 12 months ago

If there are no objections, I'd like to help all repos publish to Docker Hub alongside Quay.io.

Repos to publish to both Docker Hub and Quay.io

manics commented 12 months ago

I think it makes sense to publish JupyterHub and CHP to both since they're the base images that people use directly. Z2JH and perhaps BinderHub makes sense from the perspective of redundancy.

I don't think we should bother with repo2docker though since we made the switch a long time ago and we're beyond the point where people still expect to find it on Docker Hub.

I also don't see the point of pushing mybinder.org-deploy to both registries. It's our production deployment which we fully control, and although it's a good example of how to go about deploying a mybinder type service there's no expectation that it'll work for anyone else. The registry serves the purpose of being an internal container distribution service amongst the mybinder federation members which just happens to be public.

consideRatio commented 12 months ago

:+1: thanks for the feedback on this @manics and @minrk! There are now four open PRs opened to review about this.

minrk commented 12 months ago

I agree with @manics on most points. I don't really see the benefit for the helm chart images, since those aren't images that users use directly, the URLs are internal to the chart. So my inclination would be just the standalone images that haven't migrated long ago, (i.e. just jupyterhub and chp, I think).

consideRatio commented 12 months ago

I'd like to retain publishing to z2jh because the jupyterhub/k8s-hub image is not seldom built from, for example 2i2c is maintaining a self-managed image we bump to and rebuuild whenever we bump the chart. Its simple to migrate of course but still also easy to publish to both still.

Also, its a failsafe for the future. With charts relying on access to various images, if quay.io is down but we have published all images to docker hub as well, the fix is just a config update for all users with issues.

From 2i2c's meta chart depending on JupyterHub, but building its own custom hub pod image: image

yuvipanda commented 12 months ago

+1 to what @minrk and @manics said. I think it will help move people away from dockerhub, towards quay.io. For people who are inheriting from one of the k8s images, to bump up version numbers they have to change the FROM line anyway to specify new tags, so I think it's fine that they have to put a quay.io on front.

I'm also personally kinda pissed off and disappointed at dockerhub, given this is like the 3rd or 4th time (since this) I've been part of a community effort scrambling to try to figure out if we need to move off it or not, and it eventually 'working out' that an imminent move is not exactly needed - feels a little bit like hostage taking, and I don't wanna do that again. So, from an ideological perspective, I would like us to maintain as minimal a presence on dockerhub as possible, and make sure that our primary registry is quay.io.

manics commented 12 months ago

How about if we only push Z2JH releases to Docker Hub? I think that's a nice compromise between not supporting Docker Hub, whilst still allowing some flexibility and redundancy for admins.

yuvipanda commented 12 months ago

Given @consideRatio has already done the work in a bunch of PRs, I want to respect that. Perhaps what we can do is:

  1. Merge those, so we simultaneously publish to both dockerhub and quay.io
  2. Set a 1 year deadline - don't renew the dockerhub open source program
  3. Write a blog post about moving our primary host from dockerhub to quay.io, and recommend people move. Describe that in 1 year, there will be pull limits from dockerhub, and new images may not be published there. Describe how to move.
  4. In all public materials, refer to quay.io as our primary host, and link to Dockerhub as a 'secondary host' available as backup.
  5. Push to more secondary hosts before the end of the year as redundancy. I think this would mean GitHub to start with.
  6. Once the year is up, publish another blog post about how users should be using images on quay.io instead.

I think this eases users out, which I think is @consideRatio's original goal in making these PRs. I think any org that allows pulling from dockerhub will also allow pulling from quay.io (or GitHub registry), so I think the long term flexibility is not that limited.

How does this sound?