search

operate-first / support

This repo should serve as a central source for users to raise issues/questions/requests for Operate First.
GNU General Public License v3.0
15 stars 25 forks source link

OCP on CNV on OCP (nested clusters) #277

Closed durandom closed 2 years ago

durandom commented 3 years ago

Is your feature request related to a problem? Please describe. To support ephemeral clusters, we'd like to explore deploying OCP clusters on CNV Ultimately having a cluster bot

Describe the solution you'd like

Additional context

cc @rbo @goern @fabiand can be asked for more pointers

rbo commented 3 years ago

With OpenShift 4.7 kubevirt/cnv is available as Dev Preview

$ openshift-install explain installconfig.platform.kubevirt
KIND:     InstallConfig
VERSION:  v1

RESOURCE: <object>
  Kubevirt is the configuration used when installing on kubevirt.

FIELDS:
.....
  • pin down the minimal requirements

I assume three node compact cluster is the smallest cluster we can deploy. 3x 16GB RAM + 4 vCPU's

  • is either Hetzner or the zero cluster capable of running this

I will check Hetzner cluster after final installation/sizing.

  • how can we declaratively describe and install a cluster
  • would ACM be able to deploy a cluster

@fabiand do you know if Hive supports kubevirt/cnv?

goern commented 3 years ago

@arminmw FYI

fabiand commented 3 years ago

@rbo there is partial support in OCP - it will likely never be complete. @ravidbro can possibly help to leverage the partial support.

Otherwise the cluster can be rought up with AI or UPI.

ArminMW commented 3 years ago

How is this different from Scott White's & Arvin Amirian's Morgan Stanley's presentation on Red Hat Summit 2021 OpenShift On OpenShift - nested OpenShift ?

ravidbro commented 3 years ago

What they showed in the summit wasn't using IPI with KubeVirt provider, that effort wasn't released eventually although it mentioned in the installer 'explain'.

durandom commented 3 years ago

@ravidbro do you have any guides/docs on how to get a POC installed? My naive approach would be

  1. create appropriate VMs via CNV
  2. use UPI to deploy into the VMs (not sure if that would work with ACM)

But I'm sure the devil's in the detail

durandom commented 3 years ago

https://github.com/openshift/hypershift-toolkit

Currently examining the path of using Hypershift with kubevirt provisioner

sesheta commented 2 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

fabiand commented 2 years ago

@durandom @rmohr and others are also looking into this.

There is now also https://github.com/okd-virtualization/release - where we prep a consumable version of kubevirt for OKD. This is something we will be able to leverage in this context.

/remove-lifecycle stale

fabiand commented 2 years ago

Can we also rename this issue to "KubeVirt/HCO on OKD"

rbo commented 2 years ago

Can we also rename this issue to "KubeVirt/HCO on OKD"

But we running OpenShift Virtualization on OpenShift, so really the product not the upstream project.

fabiand commented 2 years ago

Oh! I was not aware of this. I thought that it was based on OKD

durandom commented 2 years ago

Right now we exclusively use OpenShift - but we're open and welcome OKD as well :) @rmohr @fabiand if any of you are interested in bringing this to the community we can talk about resources.

fabiand commented 2 years ago

If it's OCP then it should be pretty straight forward to simply install CNV, as long as we have some BM workers.

Is there any process to enable an operator?

durandom commented 2 years ago

We have CNV installed on the rick cluster on BM workers. As documented on https://github.com/operate-first/hetzner-baremetal-openshift

Do you have links how to install OCP on those CNV vms?

Operators are installed via gitops

fabiand commented 2 years ago

You say you have CNV installed? But then also how to install it? :) I am confused.

If you have OCP with BM workers today, then simply install CNV (it's an operator), we could help with a PR. If you have OCP in VMs on a BM cluster, then we need to enable nesting on the BM nodes, and simply install CNV on OCP inside the VMs.

WHat siutation do we have today?

durandom commented 2 years ago

LOL - maybe things progressed since we started....

If you have OCP in VMs on a BM cluster, then we need to enable nesting on the BM nodes, and simply install CNV on OCP inside the VMs.

We have CNV installed on Rick https://github.com/operate-first/apps/pull/1101 I don't think we explicitly enabled 'nesting on BM nodes'

How would you recommend deploying an OCP instance on VMs running on the Rick cluster? We have RHACM available and ArgoCD. The goal should be to deploy small clusters via gitops or an API

fabiand commented 2 years ago

Ah, now I get you:

  1. You have CNV installed on OCP BM.
  2. You would like to use CNV to deploy tenant clusters in VMs

For 2 we should indeed wait for the hypershift work to converge

durandom commented 2 years ago

@fabiand got it. Do you think "KubeVirt/HCO on OKD" will be available sooner?

fabiand commented 2 years ago

Yes and no. "KubeVirt/HCO on OKD" is simply the upstream pendant of CNV on OCP - thus not a nested cluster.

sesheta commented 2 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

rbo commented 2 years ago

Meanwhile, Hypershift is available as TechPreview with OpenShift 4.10 / ACM 2.5

image

sesheta commented 2 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

sesheta commented 2 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

/close

sesheta commented 2 years ago

@sesheta: Closing this issue.

In response to [this](https://github.com/operate-first/support/issues/277#issuecomment-1109846068): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.