popper: automate zlog/mantle deploy

[michaelsevilla]

Use Ansible to start these systems on a cluster.

• [ ] start the Docker images so the Ceph service launches
  • this will take some effort to figure out which .sos and /etc/init.d/ scripts Ceph needs
  • might need to do the same thing as ceph-deploy
• [ ] push/get keys and configs for Ceph

[ivotron]

Having all in the dockerfile might be enough. So for example, if a project foo needs an .so then they just install by adding the relevant packages (or compiling) it in the corresponding foo-build image. In the same way you do it for mantle

[michaelsevilla]

Right... but object classes might be dynamic -- so we might want the Docker image to put them in the right place on boot-up.

[dotnwat]

For reference the modes we discussed were:

1. Install Ceph official packages
2. Install Ceph official packages plus one or more custom artifacts
3. Install a modified base Ceph version

Form (1) is easy with existing tools (e.g. ceph-deploy, ceph-ansible). From (2) builds on (1) but copies some modified files into the deployment plus a restart.

From (3) might take as a reference a sha1 or a set of debs. I think in form (3) we want to still build on those tools because there are version-specific things that occur. For instance, in ceph-ansible and ceph-deploy you'll find special cases for a matrix of versions, platforms, file systems, etc...

[dotnwat]

Also, what might make this a lot easier, is to create a git-builder on dexter or some other place (even ec2) where we stuff our own personal build artificats that we want to access. For instance we could do somehting like this and just build locally: (it's basically what redhat/ceph team does)

dexter.soe.ucsc.edu:/debian/jewel/ceph-noah/ jessie stable

[michaelsevilla]

Would a "git-builder" violate Popper -- since we want others to be able to compile/deploy our code? I agree that it's easier.

Also, I got @ivotron's Docker build container to produce a .deb on my own docker-cephdev branch. Using Ceph's make-deb.sh and dpkg-buildpackage -uc -us -b -nc fails miserably. For my solution, I used my own debian/control, debian rules, etc.

[dotnwat]

Good point about violating popper. I was only thinking about how to make the dev cycles easier. My thought was that it would be nice in principal to use the popper convention for everything, including development, but have a popper "mode" in which you could hook up non-popper solutions to building artificats that make the dev cycle faster. In this way you would always be a "configuration step" away from the shippable popperized version... maybe that is too complicated :/

Regarding the debs, do you think that it is worth trying to figure out why your build environment isn't working, to keep the delta to a minimum? For reference, I just did the following on one of the HP proliant (192.168.140.85 if you wanna take a look) nodes upstairs and it worked fine:

mkdir debian-build-example
pushd debian-build-example
git clone --recursive git://github.com/ceph/ceph.git
pushd ceph
time dpkg-buildpackage -j20 -uc -nc
# ... clipped ...
dpkg-buildpackage: full upload; Debian-native package (full source is included)
real    54m58.152s
user    337m48.296s
sys 17m58.940s

Output

issdm@pl3:~/debian-build-example/ceph$ ls -l ..
total 2108072
drwxrwxr-x 24 issdm issdm      4096 Apr 13 09:24 ceph
-rw-rw-r--  1 issdm issdm     16771 Apr 13 10:01 ceph_10.1.1-1_amd64.changes
-rw-r--r--  1 issdm issdm      5528 Apr 13 09:32 ceph_10.1.1-1_amd64.deb
-rw-rw-r--  1 issdm issdm      3735 Apr 13 09:07 ceph_10.1.1-1.dsc
-rw-rw-r--  1 issdm issdm 277099944 Apr 13 09:06 ceph_10.1.1-1.tar.gz
-rw-r--r--  1 issdm issdm  53352334 Apr 13 09:35 ceph-base_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  11990684 Apr 13 09:33 ceph-common_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm 221346190 Apr 13 09:40 ceph-common-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm     26508 Apr 13 09:32 ceph-fs-common_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm     66268 Apr 13 09:32 ceph-fs-common-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   1730260 Apr 13 09:32 ceph-fuse_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  21451112 Apr 13 09:33 ceph-fuse-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   2595206 Apr 13 09:32 ceph-mds_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  45206184 Apr 13 09:34 ceph-mds-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   2588072 Apr 13 09:32 ceph-mon_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  33655660 Apr 13 09:34 ceph-mon-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   9806462 Apr 13 09:33 ceph-osd_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm 131882300 Apr 13 09:38 ceph-osd-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm      8936 Apr 13 09:32 ceph-resource-agents_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  48708198 Apr 13 09:35 ceph-test_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm 808278432 Apr 13 10:00 ceph-test-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   1750266 Apr 13 09:32 libcephfs1_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  24273218 Apr 13 09:34 libcephfs1-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  41299894 Apr 13 09:36 libcephfs-dev_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm     14674 Apr 13 09:26 libcephfs-java_10.1.1-1_all.deb
-rw-r--r--  1 issdm issdm  19357526 Apr 13 09:34 libcephfs-jni_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   1651620 Apr 13 09:32 librados2_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  23022174 Apr 13 09:34 librados2-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  42591512 Apr 13 09:36 librados-dev_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   1725332 Apr 13 09:32 libradosstriper1_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  22388486 Apr 13 09:34 libradosstriper1-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  37747708 Apr 13 09:35 libradosstriper-dev_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   2066456 Apr 13 09:32 librbd1_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  45935878 Apr 13 09:34 librbd1-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  75783550 Apr 13 09:38 librbd-dev_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   2653610 Apr 13 09:33 librgw2_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  36208762 Apr 13 09:34 librgw2-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  61451972 Apr 13 09:37 librgw-dev_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm      5586 Apr 13 09:33 python-ceph_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm    210492 Apr 13 09:33 python-cephfs_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm    577840 Apr 13 09:33 python-rados_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm    220988 Apr 13 09:33 python-rbd_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm    231404 Apr 13 09:33 radosgw_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   2865864 Apr 13 09:33 radosgw-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm    406706 Apr 13 09:32 rbd-fuse_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   3894570 Apr 13 09:33 rbd-fuse-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   1265976 Apr 13 09:32 rbd-mirror_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm  34282026 Apr 13 09:34 rbd-mirror-dbg_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm    442664 Apr 13 09:32 rbd-nbd_10.1.1-1_amd64.deb
-rw-r--r--  1 issdm issdm   4413038 Apr 13 09:33 rbd-nbd-dbg_10.1.1-1_amd64.deb

[ivotron]

If we use only docker images we can solve 1, 2 and 3. For 1, the official docker hub has images for infernalis and (I believe) jewel. For 2, we create a Dockerfile that just does FROM ceph/daemon:infernalis and then apt-get/compile the necessary artifacts. For 3 we can have a builder image (like mike's mantle-build). In all 3 cases, the executing unit is the docker image. These images are then uploaded to the docker hub (or are already there) and all the nodes in an ansible execution pull from the hub (using the official ansible playbooks). Does this work?

[ivotron]

For instance, in ceph-ansible and ceph-deploy you'll find special cases for a matrix of versions, platforms, file systems, etc...

@noahdesu I hadn't read this. I will look at it and rethink what I said above.

[dotnwat]

@ivotron definitely check it out. It's there, but I'm not sure if it is complicated enough to not try to reproduce, or if it is very simple cases.

[ivotron]

@michaelsevilla do you have an up-to-date Dockerfile for mantle that compiles ceph (or the necessary artifacts)? If so, can you please post the link?

[michaelsevilla]

It's on my own branch of docker-cephdev, here .

[michaelsevilla]

@ivotron: where are we with this on our cluster upstairs? Mantle has been fixed (#10) and rebased (#11) . I don't to deploy it or zlog in our own ad-hoc way -- I want to make sure it's set up consistently.