search

coreos / fedora-coreos-tracker

Issue tracker for Fedora CoreOS
https://fedoraproject.org/coreos/
260 stars 60 forks source link

Vagrant Cloud images for Fedora CoreOS #144

Open mcandre opened 5 years ago

mcandre commented 5 years ago

Hey, how soon can we get some beta VM and Docker images published for Fedora CoreOS, so that developers can prepare their applications to run on this lean environment? I see some entries for Fedora, CoreOS, and Atomic, but no official images for Fedora CoreOS just yet.

https://app.vagrantup.com/boxes/search

https://hub.docker.com/

dustymabe commented 5 years ago

Hi @mcandre. We are not producing any OCI container images for Fedora CoreOS. For the vagrant images we have plans to create them but we don't have an estimate yet.

Thanks for the interest! We'll try to have those soon.

pcgeek86 commented 4 years ago

I found this while searching for container images for Fedora CoreOS.

We are not producing any OCI container images for Fedora CoreOS

@dustymabe Would you mind sharing why you aren't planning to produce these? Just curious, thanks!

bgilbert commented 4 years ago

@pcgeek86 Fedora CoreOS is intended as a host operating system for running containers, not as a base image for building containers. In other words, containers run inside it, not the other way around. Do you have a use case where a Fedora CoreOS container image would be helpful?

pcgeek86 commented 4 years ago

At the moment, I don't have a specific use case. It just seems like a common trend that operating system distributions also have a corresponding Docker image, so you can explore the environment without having to create a virtual machine.

I don't use Vagrant, because on Mac you're pretty much stuck with VirtualBox, which is incredibly slow and buggy. They (Hashicorp) don't have a supported xhyve plug-in for Vagrant, which would be the most preferred option, in my mind.

With that in mind, would you guys be interested in providing a tool to spin up xhyve virtual machines, similar to the multipass tool that Canonical provides for Ubuntu? If you're not familiar with multipass, I'd highly recommend checking it out. It's painless to create and destroy virtual machines, using the native hypervisor, with no frills.

It'd be neat if an equivalent tool existed for Fedora CoreOS, to ease local development and testing.

bgilbert commented 4 years ago

Convenience tools of that kind aren't on our short-term roadmap, but I'm certainly in favor of making it easier to try out Fedora CoreOS.

Note that Fedora CoreOS requires there to be some way to pass a userdata blob to a VM, so that users can specify an Ignition config to customize the machine. Last time I looked, xhyve didn't provide a userdata mechanism, so we'd need to investigate adding one.

till commented 4 years ago

I can add a use-case:

I want to use Fedora CoreOS through ignite, and locally through footloose.

Ignite uses OCI-compatible images to start (micro) VMs using the AWS Firecracker project. Footloose provides the abstraction, to e.g. bootstrap the environment using docker locally, and then roll it out with Firecracker "online".

I came here looking for an image to be able to spin up a test bed locally as well. Providing an image would be much appreciated. Or maybe a couple steps? I'd be happy to contribute a Dockerfile in the end.

stormobile commented 4 years ago

I can add a use-case:

I want to use Fedora CoreOS through ignite, and locally through footloose.

Ignite uses OCI-compatible images to start (micro) VMs using the AWS Firecracker project. Footloose provides the abstraction, to e.g. bootstrap the environment using docker locally, and then roll it out with Firecracker "online".

I came here looking for an image to be able to spin up a test bed locally as well. Providing an image would be much appreciated. Or maybe a couple steps? I'd be happy to contribute a Dockerfile in the end.

I have the same use case. I am building K8s deployment engine around ClusterAPI on baremetal with FCOS as a base image for hypervisor hosts and I really need CI environment that is as close to BM servers as possible. One of the nicest options is to spawn ignite VMs in CI and E2E tests with the target OS inside and for that we need OCI images.

lucab commented 4 years ago

@till @stormobile your usecases are different than the initial request, it would be nice to split them to a separate ticket.

In particular, you have a hypervisor/platform (firecraker) that we don't currently support. In order to do that we'd need somebody knowledgeable on that platform to figure out:

From a quick look around, I didn't find much on those topics.

till commented 4 years ago

@lucab to start, we need Docker images, that's all. :) Same request.

LorbusChris commented 4 years ago

FCOS-in-containers in something I'd very much like to have as well :) I think this doesn't exist yet and would need a new build target for coreos-assembler

jlebon commented 4 years ago

Highly related is that cosa does build OCI containers today to ship OSTree content on OCP. Would be interesting to just repurpose that capability so that it ships a full usable tree as a hardlink-checkout of the OSTree repo.

Since we wouldn't ship /boot, it'd just be /etc that would be extra, and that's tiny. Hmm, and we'd probably bake in a run of systemd-tmpfiles to populate /var? The repo wouldn't have any protection though but I think that's fine if updates are not supported anyway.

gabibbo97 commented 4 years ago

Has anyone managed to get a way to reproduce a multi-node setup locally?

spikeburton commented 4 years ago

Is there any update on this at this time? I would really like to be able to setup a FCOS cluster with Vagrant for local development purposes.

jseguillon commented 4 years ago

Please, if possible, dont' forget to generate Hyper-V compatible box :)

SemanticBeeng commented 3 years ago

Do you have a use case where a Fedora CoreOS container image would be helpful?

@bgilbert To start, there are the use cases that the images "for Fedora, CoreOS, and Atomic ..." are used in.

, would you guys be interested in providing a tool to spin up ignite virtual machines, similar to the multipass tool that Canonical provides for Ubuntu? If you're not familiar with multipass, I'd highly recommend checking it out.

Then, we indeed have uses with firecracker and ignite.

With Ignite, you pick an OCI-compliant image (Docker image) that you want to run as a VM, and then just execute ignite run instead of docker run.

Resources

This would allow VMs based on Fedora CoreOS participate in multi-OS kubernetes implementations.

Lastly, one would hope that Fedora CoreOS will be another willing player in the ecosystem of OSes which can be containerized in standard OCI images.

If there is a reason this is not possible currently or not desired for some reasons related to Red Hat marketing plans then it would fair to declare officially so we can make plans accordingly for our clients. The gap between OpenShift / OKD and the rest of the similar distributions already hurts.

Please advise if, for the short term, is there any reason why latest Fedora CoreOS cannot be made into an OCI image ?

travier commented 3 years ago

If there is a reason this is not possible currently or not desired for some reasons related to Red Hat marketing plans then it would fair to declare officially so we can make plans accordingly for our clients. The gap between OpenShift / OKD and the rest of the similar distributions already hurts.

Please advise if, for the short term, is there any reason why latest Fedora CoreOS cannot be made into an OCI image ?

I don't think there is any hidden reason blocking this issue here. It just needs to be worked on, tested and integrated into the CI and that takes time.

bgilbert commented 3 years ago

To start, there are the use cases that the images "for Fedora, CoreOS, and Atomic ..." are used in.

In general, user software running in Fedora CoreOS should run in containers, not directly in the host. So the main use of a Fedora CoreOS container would be to run another container inside it. It sounds as though that'd be useful in CI for some people, so we might want to explore it. But I suspect it wouldn't be as generally useful as you're implying.

In addition, Fedora CoreOS expects to be deployed as a disk image, not as a filesystem image. AFAICT there are two cases:

SemanticBeeng commented 3 years ago

Grateful tor sharing this detailed thought process and design insights.

While am digesting that, let me ask this: given what ignite and firecracker can do, intuition suggests that OKD could be run without forcing Centos or RHEL as host OS... For production envs we can ask clients for RHEL since OpenShift requires it but in development it is tough sell to mandate overall change of client OS (from Debian) just to do development with OKD. Is there a way to achieve this and make OKD more palatable ?

I have some interesting things to do with Spark and Cloudflow in development but cannot justify starting from scratch with CentOS atm ...

Hope I have not made myself too obscure... Thanks in advance.

travier commented 3 years ago
SemanticBeeng commented 3 years ago

having a Fedora CoreOS Vagrant image

  1. My understanding is that firecracker / ignite actually need OCI images and run mcroVMs not full VMs.

OKD is based on Fedora CoreOS, not RHEL or CentOS

  1. https://computingforgeeks.com/setup-openshift-origin-local-cluster-on-centos/

    This guide will dive to the installation of OpenShift Origin (OKD) 3.x on a CentOS 7 VM.

Has this become not possible with latest OKD ?

support for passing Ignition configs to VMs

  1. Why is passing ignition configs to an OCI image to run would still be required ? My limited understanding says that ignition is only used at setup time.

Please advise.

travier commented 3 years ago

Has this become not possible with latest OKD ?

Not as far as I know. See https://docs.okd.io/latest/architecture/architecture.html#architecture-custom-os_architecture:

In OKD Latest, you must use FCOS for all control plane machines, but you can use Red Hat Enterprise Linux (RHEL) as the operating system for compute machines, which are also known as worker machines. If you choose to use RHEL workers, you must perform more system maintenance than if you use FCOS for all of the cluster machines.

Please ask OKD specific questions at https://github.com/openshift/okd to make sure that you get the best answer.

travier commented 3 years ago

support for passing Ignition configs to VMs

  1. Why is passing ignition configs to an OCI image to run would still be required ? My limited understanding says that ignition is only used at setup time.

Much like Vagrant current support for cloud-init to provision VMs at first boot, I think we would need support for the Fedora CoreOS Vagrant images to be provisioned via an Ignition config (even if it's a default empty config with only the Vagrant insecure ssh key if nothing is provided). This would make the FCOS experience with Vagrant similar to other platforms.

If you need pre-provisioned FCOS Vagrant images you should be able to create them with packer and automatic installation with the ISO installer.

travier commented 3 years ago
  1. My understanding is that firecracker / ignite actually need OCI images and run mcroVMs not full VMs.

I took a deeper look at firecracker and ignite more specifically and due to the way it handles kernel and kernel arguments, I think it does not make sense to run FCOS when you can directly run Fedora there. With Ignite, you would loose most of the benefit of rpm-ostree managing the root image, kernel, kernel args and boot order as you would have to extract that from the image and pass it as command line args. Enabling FCOS support directly in ignite is a possibility but that would require upstream work there.

I am renaming this issue to focus on Vagrant images as this is the most concrete step forward here while other potential use cases should be tracked in other issues.

till commented 3 years ago

@travier that's too bad. Again, one of the advantages of an OCI image (which is what ignite/firecracker uses) would be using Fedora CoreOS with footloose. Which I mentioned above, think local dev env or CI. Anything is faster than bootstrapping vagrant.

SemanticBeeng commented 3 years ago

I am renaming this issue to focus on Vagrant images

While am starting/trying to see the point of not running FCOS from OCI/Docker images but in a full VMs, I think it would be more considerate and fair to leave the name of the task as it was. We really meant OCI images and not Vagrant / VM. :-)

If it is not possible atm then fine, but better to keep track of the request as formulated.

till commented 3 years ago

I just hope the work is not in vain and someone actually wants to use Vagrant/Virtualbox (still). ;)

travier commented 3 years ago

I have created https://github.com/coreos/fedora-coreos-tracker/issues/624 to keep track of the Ignite / Footloose support request while we keep this issue focused on Vagrant support only. Thanks

travier commented 3 years ago

https://github.com/basvdlei/fedora-coreos-vagrant-box-builder > This projects looks like a good start for those interested in unofficial Vagrant QEMU based images.

travier commented 2 years ago

See https://github.com/coreos/fedora-coreos-tracker/issues/1008 for progress regarding full VirtualBox support which would make this significantly easier.

buckaroogeek commented 2 years ago

Still very interested in vagrant-libvirt support.

bgilbert commented 2 years ago

Starting with this week's releases, FCOS ships VirtualBox images. We'd still need to add Vagrant integration though.

For the record, there's some old code for supporting Ignition on VirtualBox with Vagrant. It will not work as is, but some of it may be salvageable.