Release/calendar server 9.1 docker

[Pluies]

Hello!

This PR is not meant to be merged straight in, but to server as a discussion base :)

I've set up a Dockerfile for the calendarserver, is that of interest to you, dear maintainers?

If so...

• Could you look through this Dockerfile and tell me whether the way it's built suits you, or whether there's things that should be fixed?
• I had to disable OpenSSL in the script -- we should probably make that configurable, because this possibly breaks the base script
• Should we put this Dockerfile the base folder and make it a bona fide supported way of installing the calendarserver?
• Should we integrate with Docker Hub to provide official images?

Thanks!

• Florent

[dreness]

Hi @Pluies! Your proposal is of interest, and although I don't have answers to all your questions right now, I will review the Dockerfile and we can go from there. Also I can say that we generally prefer to not install OpenSSL ourselves, and prefer to rely on a system-wide installation.

[dreness]

I played around with this a bit tonight, and in general I think it looks good, although I wasn't able to test it as written because in Docker for Mac, "Extended attributes are not yet supported."

Ignoring that problem for a moment, the general flow goes something like this (note: I'm kind of a docker noob):

• install and set up Docker

  mkdir ~/ccs-docker
  cd ~/ccs-docker
  git clone https://github.com/Pluies/ccs-calendarserver -b release/CalendarServer-9.1-docker
  cd ccs-calendarserver
  docker build -t ${USER}/jessie-ccs .

  The end of a successful build looks something like:

  ...
  Step 9/9 : CMD ["./bin/run", "-n"]
  ---> Running in 80f4bdcdebfd
  Removing intermediate container 80f4bdcdebfd
  ---> 502792819e32
  Successfully built 502792819e32
  Successfully tagged dre/jessie-ccs:latest

  Note that an attempt was made to start the container ("run -n"), but if you run docker ps, you don't see it. The reason will be clear soon. Let's manually start a container that maps the 'data' and 'conf' directories from the host to the container:

  docker run --rm --name ccs \
  --volume /path/to/ccs-docker/ccs-calendarserver/data:/opt/ccs/data \
  --volume /path/to/ccs-docker/ccs-calendarserver/conf:/opt/ccs/conf \
  -p 8800:8800 -p 8443:8443 dre/jessie-ccs

  ... and it says:

  
  Missing config file: /opt/ccs/conf/caldavd-dev.plist
  You might want to start by copying the test configuration:
  
  cp conf/caldavd-test.plist conf/caldavd-dev.plist

Would you like to copy the test configuration now? [y/n]

Since the 'conf' directory being used is the one on the host, not in the container, run this (on the host):

cp conf/caldavd-test.plist conf/caldavd-dev.plist

re-run the 'docker run ...' command. This time, the service should start. To check the plumbing, open a web browser and hit https://localhost:8443/ - maybe accepting any self-signed SSL cert warnings. If you see a collection listing, hooray! If you're on a Mac, it doesn't work due to the extended attribute problem mentioned earlier, and you see the following in the container:

IOError: [Errno 95] Operation not supported: '/opt/ccs/data/Data/Documents/.calendarserver_version'

This could be avoided by not using the 'osxfs' docker filesystem to mount the 'data' directory from the host. Don't store 'data' inside the container unless you're very confident that you know how to backup / restore / preserve that data directory (which contains every bit of the data created by users of this service). Of particular note: the 'rm' option to 'docker run' deletes the container after it exits, which is probably not healthy unless all the persistent data lives (or is reliably synced NOT USING FILE UTILS because that's a great way to break a running database) outside the container. It's probably possible to use nfs instead of osxfs for the 'data' directory by nfs exporting it from the host and mounting it in the container at /opt/ccs/data/.

[Pluies]

Ah, I hadn't realised Docker for Mac mounted the folder differently and CCS then relied on extended attributes - I run this particular container inside a Debian box and everything is hunky-dory. After nearly a year of using this setup, everything is still rock solid and the only issues I've ran into were due to my SSL cert needing renewal :)

Regarding volumes, indeed, NFS could work, or any other non-OSX volume mounting system - I'd expect this to work nicely in e.g. Kubernetes' PersistentVolumes.

[stefanoco]

Trying to use this container for test purposes, and it seems to me that mailgateway operations (invites managed by email with IMAP) are missing, no activity and no errors. I configured a working IMAP gateway but no sign of activity...

[dreness]

@stefanoco Hi - did you perhaps find our FAQ item about this? https://www.calendarserver.org/FAQ.html. The mail gateway stuff requires no extra code-level dependencies, so should work as long as the IMAP and SMTP credentials are valid on a real mail server (local or external). It will poll the mailbox, so you don't need any inbound connectivity (other than whatever is needed if you're hosting a mail service also).

[stefanoco]

Yes @dreness sorry for not mentioning, I followed the faq and other examples and configured into conf/caldavd-dev.plist credentials for accessing dedicated email account in a remote SMTP and IMAP service.

[borgoat]

Hi all!

Was working on the same point as @Pluies as I wanted to run CCS on OpenShift. I'm trying to achieve a "productive" image, ie not using bin/develop (as according to the docs bin/package should be used for this). I saw @dreness reply on #488 , if more documentation or at least some hints were available I'd be happy to implement the Dockerfile. The main reasons why I don't like either bin/develop nor bin/package is that they install Postgres and memcached, however, I'd rather run those as separate containers.

Thanks! Giorgio

[borgoat]

Hi

Just a quick update as I made some progress with this. Here you can find the Dockerfile.

I simply used pip install -r requirements-default.txt and everything was installed under /usr/local.
Postgres and Memcached are running as separate containers, what I have to do now though is the setting of passwords and hosts, as right now this is hardcoded in etc/caldavd/caldavd.plist.

I'll do this then also run it on OpenShift to test a cluster deployment (despite the database lots of folders are created, some are for attachments and have to be shared between all nodes, don't know about the others...)
Once I've done all this I'll be happy to contribute to the documentation.

If anyone else is interested in this please let me know!

Cheers Giorgio

[dreness]

Looks cool, @giorgioazzinnaro - I'm interested to hear your experiences with this.

[borgoat]

Hey @dreness , glad to know that there's some interest for this. I actually moved the work here https://github.com/apple/ccs-calendarserver/compare/master...giorgioazzinnaro:master So that if you like it can eventually be merged.

As mentioned earlier, I'm now focusing on an OpenShift deployment (because I have a production use case), however, different images could be created for plain Docker, K8S, or OpenShift. The issue I'm having right now is because OpenShift runs with a random UID, but calendarserver seems to expect the UID to exist in /etc/passwd, I'm still trying to understand better.

If you like, I could recreate the deployment into OpenShift Online so that I can share the progress

[dreness]

@giorgioazzinnaro I'll just keep an eye on what you're doing for now. I think you're the first person to want to run CCS in an OpenShift context (that I've heard about, anyway), but I still wouldn't mind shipping extra docker files under contrib or something. Feel free to open a new PR when you're ready.

Looks like you figured out the stuff with the random UID - you're basically the same thing that the OpenShift docs suggest. I guess they do the random UID thing to help encourage stateless behavior, heh.

Also, you might find some useful info tidbits in the MultiserverDeployment doc. Even if you're only hosting with a single instance of CCS, it sounds like you're using a configuration similar to what could be used in a cluster. In particular, I don't see any shared storage in your config, which would be needed to store calendar event attachments.

[borgoat]

When you mention the shared disk between nodes, isn't it correct that I should share /var/db/caldav (or ServerRoot)? Right now I configured that as a Docker volume. So it will be persisted across restarts, and shared in different pods in Kubernetes.

[dreness]

@giorgioazzinnaro The only hard requirement for shared filesystem space (in a multi-server operation) is to support file attachments. Other stuff can be shared as well, with a few important exceptions:

• RunRoot
• LogRoot
• SocketRoot

Those directories should be kept separate (unless you configure unique paths for all items contained by each). The default relative locations of the various configurable paths are defined in the RELATIVE_PATH array in twistedcaldav/stdconfig.py. Note also that the config parser supports an 'include' directive which is useful to define server-specific configuration.

[mschoengens]

@giorgioazzinnaro Thanks for providing a Dockerfile for CCS. I was actually thinking that it might be interesting to have also a docker-compose file for smaller deployments (with separate Postgres and Memcached containers). Have you had any thoughts about this? Is there a more general interest?

[borgoat]

Hey @mschoengens , yes setting up a Docker Compose or Stack is also an option. Right now I focusing on openshift and plain k8s compatibility, with that in place compose should be a piece of cake.

I can tell you that right now I'm focusing on an openldap image, once I do that I'll write both the k8s/oc template and the docker-compose.yml

I'll keep pushing updates, let me know if you want to collaborate! Cheers!

[mschoengens]

Hi @giorgioazzinnaro, we could definitely collaborate on this. I actually created a Docker Compose file using a modified version of your Dockerfile. I can create a pull request to your repository, so you can have a look at it.