search

ClusterHQ / flocker

Container data volume manager for your Dockerized application
https://clusterhq.com
Apache License 2.0
3.39k stars 290 forks source link

Document PoC for getting ZFS on Linux running on CoreOS #667

Closed lukemarsden closed 9 years ago

lukemarsden commented 10 years ago

Replaced by https://clusterhq.atlassian.net/browse/FLOC-667

It would be nice to be able to run Flocker on top of CoreOS. There has already been some initial demand for this sort of collaboration: https://groups.google.com/forum/#!topic/flocker-users/taBOUyX3W8A

The first step towards doing this is getting ZFS on Linux working on a CoreOS kernel and userland. This issue is to produce a tutorial documenting our initial work towards that goal, so that we can share it with the CoreOS team.

Future work may include running Flocker in a system container - in order not to require installation of the Flocker userland dependencies in the CoreOS base system. There is then also the question of deeper integration with Fleet/Kubernetes.

lukemarsden commented 10 years ago

Got this working (and tested) as a tutorial here: https://github.com/ClusterHQ/flocker/blob/zfs-on-coreos-tutorial-667/docs/experimental/zfs-on-coreos.rst

Feedback welcome!

thaJeztah commented 10 years ago

Awesome work! CoreOS looks to be the OS for Docker (and, thus, flocker), so any effort on making this work is a step to a brighter future!

lukemarsden commented 10 years ago

Thanks @thaJeztah! Credit where it's due, @ryao did all the hard work on this...

ryao commented 10 years ago

@thaJeztah The technique that I used allows for the Gentoo ZFSOnLinux packaging to be reused on arbitrary distributions. The tutorial skips a few steps as the full technique uses scripts to build from source. My plan is to make it as easy as possible to use. I will likely write a blog post on the technique, how to use it on arbitrary distributions and how to integrate the Gentoo packages into the system startup later this month after a few rough spots have been ironed out.

thaJeztah commented 10 years ago

I'll be keeping an eye out on your future blog post. I'm currently testing out various projects in the 'Docker ecosphere' to check out their capabilities and see which ones are useful for our purpose. A lot is going on and a lot of interesting projects exist!

ryao commented 10 years ago

@thaJeztah I have updated the instructions. Here is a changelog:

  1. A new tarball is provided that integrates glibc to allow kernel modules to be rebuilt on production CoreOS images.
  2. The use of LZ4 was eliminated. It turned out that xz -9e has better compression than the lz4c -9 and xz -1 combination that I had previously used. The consequence is that we need 65MB of RAM to decompress, but that should be okay.
  3. The prebuilt modules are built against the 3.16.2+ currently shipping with CoreOS.
  4. PGP signature verification has been added to the instructions for security.
tlvenn commented 9 years ago

Quick question, since coreos is using btrfs now, couldnt flocker be used with zfs or btrfs so that it could works transparently on coreos ? I believe btrfs have all the features you currently leverage from ZFS to make it happen.

exarkun commented 9 years ago

Hi Tlvenn. Thanks for your interest in Flocker. Indeed, btrfs offers capabilities similar to many of those Flocker uses from zfs. I can imagine that at some point in the future Flocker might be able to let you use either btrfs or zfs as the filesystem for the volumes it manages (and the existing implementation of zfs support already takes some steps towards supporting alternate filesystems). At present, development is focused on providing a good experience with zfs. I don't know when a btrfs backend will become a priority. This might be an area where we could collaborate, though. Let me know if you're interested in putting together some code towards this end.

rbucker commented 9 years ago

how will this work with Rocket?

exarkun commented 9 years ago

Hi rbucker,

Rocket support will probably be considered independently of general CoreOS support. As far as I know, there are no plans to make it impossible to use Docker on CoreOS (and no plans to prevent the use of Rocket on other OS).

adamtheturtle commented 9 years ago

We are moving our development planning to JIRA. This issue is now being tracked at https://clusterhq.atlassian.net/browse/FLOC-667. You are welcome to file additional issues in GitHub if that's easier for you.

cybertk commented 9 years ago

Is any update on this topic? https://clusterhq.atlassian.net/browse/FLOC-667 is in BLOCKING state now.

binocarlos commented 9 years ago

@cybertk hey - we are currently working on finding a way to mount a host filesystem from inside a privileged container (i.e. the flocker dataset agent) and then mounting that filesystem into a user initiated, non-privileged container. This has become an active issue that is being worked on - we hope to have some progress within a few weeks. I will update this issue once I have some news.

wallnerryan commented 9 years ago

+1

PhantomYdn commented 9 years ago

@binocarlos , could you please share current status for the issue? We are interested in CoreOS, but absence of support of state-full containers stops us...

binocarlos commented 9 years ago

@PhantomYdn Hey - so we have just published a blog post that demonstrates running Flocker on CoreOS - the main thing being Flocker running inside containers. This is running on AWS using the EBS backend. The next steps are to integrate this repo which builds ZFS binaries for CoreOS and get the ZFS backend working. Can I ask what type of dataset backend you are interested in using on CoreOS (i.e. local storage or something like EBS?)

PhantomYdn commented 9 years ago

Thank for the link! Honestly, we are not interested in EBS especially - we are looking either use our own data-center or digital-ocen like things. The main idea is to provide Container as a Service for solutions built on top of Orienteer. So, ZFS is much more desired:) Just do not be so tied with Amazon...

robhaswell commented 9 years ago

Hi @PhantomYdn, please may I ask if another replicated local storage ("hyperconverged") solution would be appropriate, e.g. Ceph (in a configuration where it works) or EMC ScaleIO.