search

jupyterhub / team-compass

A repository for team interaction, syncing, and handling meeting notes across the JupyterHub ecosystem.
http://jupyterhub-team-compass.readthedocs.io
62 stars 33 forks source link

Prepare for end of gke.mybinder.org #642

Open minrk opened 1 year ago

minrk commented 1 year ago

Google cloud credits are ending at the end of this month. Some tasks to track dealing with this:

choldgraf commented 1 year ago

How much of our current capacity can we re-direct to OVH? If we assume that no other services are spun up (e.g., on AWS), what is the total expected drop in capacity?

I'll ask NumFocus about our Binder funds now and see what they say.

manics commented 1 year ago

As previously mentioned I've got some Terraform for deploying an AWS EKS cluster, and I've deployed a BinderHub example using ECR. I haven't looked at scaling it up, but it should be enough to get started. Happy to help once we've got an admin access to an AWS account.

choldgraf commented 1 year ago

@manics I've opened an issue to track deploying on AWS at the link below (and added it above), wanna discuss and track there?

minrk commented 1 year ago

OVH hasn't been as robust, I think largely due to issues in the Harbor registry, which may be fixed (https://github.com/jupyterhub/mybinder.org-deploy/issues/2514#issuecomment-1506693566). It seems fairly stable now with ~100 users, and ought to cover everything we have on GKE (funding-wise), but it seems to struggle with larger load. I've bumped capacity to 200 in https://github.com/jupyterhub/mybinder.org-deploy/pull/2557 so we can see how that goes.

Current typical peak users is ~500, so it's a pretty big drop.

We've also had some disk-filling-up issues, which can be solved short-term by recycling nodes, but needs to be addressed somehow.

Two possible ways we can address capacity in OVH, since funding isn't short right now:

sgibson91 commented 1 year ago

Also, I have access to Scott H's AWS account that I have been testing https://github.com/jupyterhub/mybinder.org-deploy/pull/2467 on

Update: Pangeo have actually pulled down their AWS deployments, announced in this Discourse https://discourse.pangeo.io/t/aws-pangeo-jupyterhubs-to-shut-down-friday-march-17/3228 Stating the reason "Credits have depleted" - so potentially Scott's account is no longer the best place to be testing/host the new AWS deployment

minrk commented 1 year ago

I realized another significant change that we may have to deal with more directly: staging is also run on GCP. We probably need to shut this down as well to save costs, though it's drastically less costly than running prod.

We could also launch a staging cluster on OVH, since we have room there, but I think it can't run everything it currently does (analytics publisher, matomo). Essentially, everything that we do want to keep on GKE after shutting down gke.mybinder.org because it isn't easily relocated.

Perhaps notably, we don't actually run tests for these components, so the staging tests will pass just fine if it's only a single federation member and not a full staging 'federation'

choldgraf commented 1 year ago

Do we have an estimate of how much staging would cost on GKE?

minrk commented 1 year ago

Cost calculator suggests it should be around ~$250 for the current cluster and cloud sql (I'm assuming registry costs are negligible, since registry costs are ~all pulls, and staging hardly does any pulls). I could probably save ~$100 further by using a smaller cluster (e.g. 4 cores with 200GB disks or even less). I think I can comfortably fit it in $200/month, and if we push it, fit it in $100.

minrk commented 1 year ago

Without the prod BinderHub, I think I can probably aim for a fairly similar cost for running only the non-Hub parts. Those are all pretty low-resource.

We could even run both staging and the prod entrypoint on the same cluster in different namespaces. That's probably not the best idea, though.

choldgraf commented 1 year ago

So is this right then:

Or around $2,400 - $4,800 / yr ?

And then user clusters are all fully modular?

As a stop-gap, maybe this is what we use the funds that @arnim mentioned from the GESIS work that we are doing (especially if we could get it on the lower end). That'd keep the core going for at least a year.

minrk commented 1 year ago

I believe we can get to that, yes.

choldgraf commented 1 year ago

In that case, my feeling is that this should be our target for "core funding" that we pay for with cash rather than bunches of credits, and try to run all of this on GKE. Then we can boost capacity of mybinder.org in a more opportunistic way, where we just spin up a cluster somewhere and connect it to the federation.

AKA, credits can power user capacity, but we should have $$ to cover at least 1 year's worth of core operation, or something like that.

arnim commented 1 year ago

Ensuring that at least the core-infrastructure can continue to operate is important. As stop-gap we can use for this 5k USD from the GESIS-2i2c contract.

rgaiacs commented 1 year ago

I'm trying to complete the setup of BinderHub on GESIS premises. Trying to navigate the sea of Helm chart.

@minrk and @sgibson91 do you have 1h to help me with online pair programming debugging next week? Maybe Tuesday?

minrk commented 1 year ago

Yes, I think I can do Tuesday. Any time between 10-15 CEST.

rgaiacs commented 1 year ago

Thanks @minrk. I sent an invitation to your email.

manics commented 1 year ago

We could even run both staging and the prod entrypoint on the same cluster in different namespaces. That's probably not the best idea, though.

What are your main concerns with running staging on the same cluster? It might not be "best-practice" but it'd save $74/month, with more savings from not paying to run an additional set of K8s management pods.

The Prod cluster has a 2TB volume for prometheus which is around $80/month. I presume this was originally setup for long term storage of data- in which case we could save a lot by archiving it to an object store ($40/month for 2TB, even cheaper for the long-term archive class)?

However if this was the intention we're not doing it! We're only storing the metrics for 30 days, and we're using < 50G:

$ kubectl -nprod exec $(kubectl -nprod get pod -lapp=prometheus -o name) -c prometheus-server -- df -h /data
Filesystem                Size      Used Available Use% Mounted on
/dev/sdc                  1.9T     46.5G      1.9T   2% /data

https://github.com/jupyterhub/mybinder.org-deploy/blob/5a710c1159b18fadc56402b27aaa853626f1fb21/config/prod.yaml#L121-L125

minrk commented 1 year ago

It might not be "best-practice" but it'd save $74/month, with more savings from not paying to run an additional set of K8s management pods.

It's mainly the 'best practice' bit. We do deploy one cluster-wide resource (cert-manager), which would become shared. So any upgrades related to that could only be deployed to prod, not tested on staging before updating prod. It actually wouldn't save any on management costs, because staging is a zonal cluster, which costs $0, while prod is regional (benefits from zero downtime upgrades). It would mostly save on node costs, which is around $100-$150 for staging.

The Prod cluster has a 2TB volume for prometheus which is around $80/month. I

I think prod prometheus was never meant to be long-term. With more than a couple months of data, performance gets too slow and too memory intensive. It's also going to dramatically reduce its needs when the prod binderhub stops, because there will be no launch events or running notebooks, which is the main resource prometheus tracks.

minrk commented 1 year ago

The simplest way to reduce the cost of prod prometheus is to reduce the size and just delete the current prometheus data and deploy a new PV. This should be fine, since prometheus data is by definition ephemeral, but it might be good to do that sooner rather than later, so we have some data from before the transition.

minrk commented 1 year ago

What do folks think of deploying staging in a namespace on the prod cluster? I think this would likely save ~$100/month, but it would increase the risk of downtime on a bad deploy to staging.

manics commented 1 year ago

What do folks think of deploying staging in a namespace on the prod cluster?

For the cost saving I think that's justified. Ultimately we're asking are we better off spending $100/month on staging to reduce the chance of an outage (do we know how often that happens in practice?), or to spend it on ensuring the production cluster keeps running for longer. Given the current situation I think saving the $100 is worth it, we can always switch back later.

We can reduce the risk by enforcing a namespace resource limit https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/quota-memory-cpu-namespace/

minrk commented 1 year ago

Having looked at it, I think the effort of merging staging into prod safely (things like ingress controllers, node exporters, etc. might have issues), plus the fact that I think we can trial running staging on spot instances for a total of ~$20/month, makes the effort/cost not worth merging them this week. We can look at it later, though, since I think it would be on the order of saving $1/day.

choldgraf commented 1 year ago

2i2c is ready to use funds to pay for the core Binder infrastructure

We discussed this in a team meeting yesterday and agreed that 2i2c can use its funds to power the "core" mybinder.org infrastructure, and pass through this cost to GESIS up to $5,000.

Here are the things I think we need to do:

@minrk can you provide guidance on the proper projects to look for to connect 2i2c's credit card?

I'd appreciate if we could mention this at least in a blog post or something, so that this arrangement and the work going into it isn't invisible to the outside world (for both GESIS and for 2i2c). Are others OK with that?

manics commented 1 year ago

Sounds great. In addition to the blog we could give the funders more prominence on https://mybinder.readthedocs.io/ ?

For the avoidance of doubt, it'd be good to mention somewhere that the core funding provided by GESIS (for GKE) is in addition to the compute provided by GESIS (https://notebooks.gesis.org/binder/) as I was quite confused at first- I thought the core infra was moving to GESIS 😄

choldgraf commented 1 year ago

Yep good point, maybe we should make all of this explicit in those docs. e.g. a page that breaks it down by:

and so on?

minrk commented 1 year ago

That sounds like a great plan! It's all in project binderhub-288415, where everything has been this year. I believe this is currently already backed by @choldgraf's 2i2c card (billing account: gke.mybinder.org) once the credits stop (it's labeled "Ramp cloud card", I don't know what that is). I set up a separate 'mybinder-gke' project last funding-pressure round, and that's where I'd like to deploy the next GKE cluster, if there's ever funding for that, to more easily separate the federation member from the core infrastructure, and more easily bring cluster costs all the way to zero. There is nothing there now, though.

I'll have to keep an eye on costs over the weekend and into next week, but looking at what we're doing currently, I think around $300/month is within reach.

choldgraf commented 1 year ago

Ah perfect, OK a few responses:

Any objections to either of those? I don't think that there are unintended consequences if we rename a project, since it still keeps its unique ID, but wanted to double check.

minrk commented 1 year ago

A blog post is a great idea, maybe aim for next week when some dust has settled?

core infrastructure project

Costs are unclear for now for logs, which I expect will go to zero without binderhub deployed (first 50GB is free and we used 400GB for $175 last month). I think we can do some log exclusions to make sure that happens (nginx ingress logs are pretty verbose and we don't need all of them)

minrk commented 1 year ago

Any objections to either of those? I don't think that there are unintended consequences if we rename a project, since it still keeps its unique ID, but wanted to double check.

Sounds good! Yes, the name is used exclusively for display purposes. You can't actually use it anywhere, it's always the immutable project ID that makes things work.