aquarium: package glass and gravel

[jecluis]

Context Distributing our mighty neat appliance in a ready to consume way is a good path to adoption. Containerizing it will allow users to go through a very simple process to get Aquarium up and running.

Describe the solution you'd like A container containing glass and gravel, with gravel running upon start and serving glass.

Additional context Something that I personally still struggle with is how the bootstrapping will happen. Given we're running on a container, how are we going to start other containers on the host?

[tserong]

Something that I personally still struggle with is how the bootstrapping will happen. Given we're running on a container, how are we going to start other containers on the host?

That would be the trick... Containers aren't really meant to be able to take over the host and tell it to do stuff, right?

I wonder if you could do something ~evil~ ~abominable~ clever with bind mounts inside the container?

Or, as another alternative, we don't put aquarium in a container; instead we build a regular package (RPM, DEB, etc.) then in our MicroOS case, we just install the aquarium RPM (and it can then be updated the same as the rest of MicroOS, as a package). If someone wants to deploy aquarium on, say, Debian, they install our DEB. Of course the downside there is we end up maintaining a handful of different packages for different distros vs. a single container image. But it would fix the bootstrap problem ;-)

[tserong]

...although, it's really only calls to cephadm we care about, right? Everything else will go through ceph orch or librados or whatever. We could make the aquarium container ssh to the host to invoke cephadm boostrap (given all the hosts need ssh running anyway, because that's how cephadm itself works).

[asettle]

@theanalyst any update on this?

[sebastian-philipp]

Something that I personally still struggle with is how the bootstrapping will happen. Given we're running on a container, how are we going to start other containers on the host?

That would be the trick... Containers aren't really meant to be able to take over the host and tell it to do stuff, right?

I wonder if you could do something ~evil~ ~abominable~ clever with bind mounts inside the container?

calling executables on the host (like cephadm) from within a container is a bit awkward. That's why we opted for a non-containerized cephadm daemon.

Or, as another alternative, we don't put aquarium in a container; instead we build a regular package (RPM, DEB, etc.) then in our MicroOS case, we just install the aquarium RPM (and it can then be updated the same as the rest of MicroOS, as a package).

:+1:

If someone wants to deploy aquarium on, say, Debian, they install our DEB. Of course the downside there is we end up maintaining a handful of different packages for different distros vs. a single container image. But it would fix the bootstrap problem ;-)

[theanalyst]

I've tried to containerize gravel but still struggling with getting the dependencies as we depend on things like ssh/system-user etc which can't get inside a docker image yet (per my understanding).

Agreeing with @sebastian-philipp here if we're ending up calling host executables having it inside a container doesn't help.

[jecluis]

I fear I have to agree. Every scenario I can think of will require something to be installed on the host, and having the Aquarium reaching to it for the bootstrap process. And that would mean maintaining a package, or a script, or something, that had to be run on the host with super cow powers anyway. So I wonder if maintaining Aquarium as is, running on the host, is such a bad idea?

[tserong]

OK, sounds like I need to see if I can package this thing :-)

[jecluis]

That is going to be a tough one too, I'm afraid. Especially in opensuse. Things we depend on, like fastapi, do not have packages available in Tumbleweed.

I do wonder if it's particularly horrible that we just supply a vagrant image for people to play with?

[tserong]

We can look at (somehow) embedding difficult dependencies inside the package. Anyway, on that note, let me screw around a bit and see if I can come up with anything useful, or if it just devolves into me swearing lots.

Nothing wrong with the vagrant image for people to play with. My concern is longer term, like, how do we ship a new aquarium in future? How do you upgrade to it?

[jecluis]

Nothing wrong with the vagrant image for people to play with. My concern is longer term, like, how do we ship a new aquarium in future? How do you upgrade to it?

For me, this would be the biggest benefit of MicroOS. Want to upgrade? Reflash the system.

[tserong]

My understanding of MicroOS is that you don't reflash the system, but that you install regular RPM updates when there's new versions of all the OS packages. The difference between this and regular SUSE distros is that the updates happen atomically, thanks to btrfs snapshots.