Add Kubernetes helm support

[titilambert]

Hello, I'm currently trying to adapt mailcow-dockerized as kubernetes helm chart. I have already a good result ! I would like to know if you have interest on that stuff ? If yes, how you want I commit that ? (a new folder ?)

Thanks for your answer

[kristiandrucker]

@titilambert where can I find the helm charts? I want to deploy mailcow on my k8s cluster. Also can it work behind traefik and with ingress?

[titilambert]

Hello! I got something working, but I need to rework it to make it more flexible (without my password). BTW, in my own cluster, it works with nginx ingress. But, we need to get https://github.com/andryyy/mailcow-dockerized/pull/190 merged before anything

[titilambert]

@kristiandrucker Where do you have your k8s cluster ? Are you using any cloud provider ? AWS, GCE?

[kristiandrucker]

@titilambert I have a baremetal install with pvc working. Could you give me the helm chart so I may look at it and test it on my cluster?

[kwiesmueller]

Hey there, I would be happy to assist with both reviews and the chart itself. Otherwise I would build one in the next few days.

Oh and @titilambert if you need a quick solution for secrets in helm, we built some small management tool for our own clusters with helm (https://github.com/kwiesmueller/captain-template). It is WIP but i already use it a lot for managing all clusters with helm (aside from plain manifests).

[TBK]

@kwiesmueller Did you implement your own chart?

I am migrating my setup to Kubernetes, so an already made/tested helm chart for mailcow would save me some time.

[kwiesmueller]

@TBK0000 not yet, had no time yet. If you would like, we could try to continue the work that went into this here or take this over together if @titilambert stays inactive on this? Is there a PR maybe, or have yo got a repo?

[TBK]

Sure, I got exams the next couple of weeks after that it is my intention to develop a chart for mailcow.

I have setup a repo for the work https://github.com/JJTC-Helm-Charts/MailCow - PRs are welcome.

[kwiesmueller]

@TBK0000 great, will take a look the some day this week

[Promaethius]

https://github.com/kubernetes/kompose/blob/master/docs/conversion.md As far as I can tell, kompose will work with everything except the volume and dns parameters.

Volumes: Some volumes in docker-compose.yml should be treated as mounted kubernetes configmaps. DNS: I cannot tell if the dns module is actually used for any kind of DKIM signing or letsencrypt requests.The entrypoint.sh script confuses me since it exists as a wrapper for exec commands yet I cannot find exec commands in the code that call it. It might be possible that the unbound dns module is swappable for the default kubernetes dns module.

Past these two issues, mailcow dockerized uses a docker container that directly calls the docker.sock as a client to poll container names, perform a few sieve actions, and restart SoGo when a new domain is added. I'm working on a dropin container based on the kubernetes api over at https://github.com/Promaethius/mailcow-dockerized-kubeapi

[maximbaz]

@titilambert, @Promaethius, @TBK I wonder if you guys had any success with getting helm charts to work? I would really love to use a managed cluster for hosting mailcow (instead of managing my own VM), but I don't think it is possible with Docker compose.

[Promaethius]

@maximbaz there's a couple ways to handle hosting mailcow on kubernetes: docker in docker or refactor. I was looking at a programmatic method to automatically refactor this repository but it'd require more e2e testing than I'm able to do now.

Mailcow-dockerized has two hurdles in creating a kubernetes version: DNS, dockerapi, and the filesystem. All docker dns endpoints for services are hardcoded into mailcow. This isn't a huge problem with kubernetes services. Additionally, it uses a container to access the docker socket. I'd like to either decentralize the container API into kubernetes API basics or just create a kubernetes kubectl drop in container. Recently, mailcow made a switch for rspamd to use an I/o socket. This would need to be changed to a tcp socket to work in a distributed system.

There's a lot of work to do in refactoring the code. For the time being, I'm investigating a dind solution so I can get mailcow at least running in my kubernetes cluster. Then do a refactor.

[maximbaz]

Thanks for the detailed answer! Please post an update if you get it working, I'm very interested to follow your progress.

[Promaethius]

@maximbaz refer to https://github.com/Promaethius/mailcow-dind It quickly threw up some initial scripts, kubernetes examples, and a Dockerfile. Absolutely none of it is tested.

UPDATE: The image seems to work, the stack runs inside of dind. I'm adding e2e tests over these next few days.

[thewilli]

@andryyy can't we left this open while some people seem to work at it? It would definetly be a huge benefit to mailcow, and decoupling the services would - independent from the k8s support - for sure improve the overall quality

[andryyy]

This is not an issue for mailcow. You can work on it and improve the existing scripts, but it will not be used by mailcow.

[Promaethius]

@thewilli unfortunately, the entire stack would have to be changed to fit in with kubernetes standards. Decentralization of mailcow isn't mailcow anymore. You can look at my dind solution that I'm using in production. However, reformatting mailcow into kubernetes isn't practical since they're currently porting mailcow to a stable phone framework.

[zedtux]

Disclaimer: This comment is here to help, not to hurt !

---

Honestly, mailcow people, you're going to the wrong directly from my point of view and I'll try to explain why.

It's super awesome you're using Docker, but you're doing in a weird way. I'm not trying to do some trolling, I'm really honest and it's for the best of this project, I really love mailcow (I'm paying for a single server just to use it)!

The way you made it comes with so many constraints to the users because you made the docker-compose.yml file and mailcow with mailcow runs alone on the node/server in mind which is wrong.

In my case, and I can bet it's the case of the majority, I'm using the nginx-proxy Docker image so that I can deploy my blog, some other apps with the automatic nginx config update and I want to have mailcow too, but it is simply not possible, or very very hard to achieve it. I have already spent dozen of hours trying to make it working all together but I wasn't able.

This should show you that you're not going in the right direct.

Another thing : the fact that mailcow is incompatible with Kubernetes because it would require an adaptation of the docker-compose.yml file shows also that you're going the wrong way with Docker.

I know I'm not gonna change your mind with this comment, but I just want to share my experience with the dockerized mailcow and hopefully start to open a door in your mind helping you to realise that there is an issue about the way you're using Docker.

[andryyy]

Yes, I don’t agree but don’t want to repeat myself over and over again.

There is absolutely no problem running many apps in Docker on the same node alongside mailcow. Incompatibility with Kubernetes seems to worry you and for some reason you seem to think this means we do things wrong. Again, I don’t want to repeat myself. That’s annoying.

The whole concept of a mail server stack/cluster on Kubernetes is not as easy as starting some containers for development purposes.

If you need a mail server stack that is compatible with Kubernetes, you will need to drop a lot of features. We will not do that.

Feel free to enlight us about the correct way of using Docker. And keep in mind to not drop any feature of mailcow.

PS: Using a reverse proxy in a separate image is totally fine. You don’t break any Docker rules by doing it. It seems you are annoyed because things are not implemented as you need them personally for your setup. We offer paid support for special Docker setups and clusters. And no, we won’t do it wrong.

Meeh (or moo), I will not follow that topic anymore.

[zedtux]

Can you point me to the place where you described why and what features you have to drop in order to get a mail server stack that is compatible with Kubernetes please?

[Promaethius]

@zedtux I describe some of the issues above that prevent mailcow from running on kubernetes.

The creators of mailcow have a roadmap for a product that already works. So they're working on code refactoring, security, stability, etc. A port to kubernetes is out of the picture since in this scenario, docker-compose is acting more like a package manager/process daemon rather than a 12 app standard for cloud containerization.

[zedtux]

Thank you @Promaethius for your answer.

I read the following from your last comment :

... a stable phone framework.

What do you mean by that?

As far as I can see, what you described @Promaethius as tasks to be done to migrate to Kubernetes are about volumes and DNS which doesn't seem to cover what @andryyy gently mentioned by

If you need a mail server stack that is compatible with Kubernetes, you will need to drop a lot of features.

I guess I have to look at all the issues until I can find what are the features that would have to be dropped as stated by @andryyy as he ended his comment by "I will not follow that topic anymore." (which I guess is justified from his eyes).

In the case you would know where it is described @Promaethius, please let me know :)

[zedtux]

I tried searching issues with Kubernetes in this repo in order to find why and what a lot of features would be dropped by using Kubernetes ... 🤷‍♂️

[der-eismann]

One year later I would be interested if the maintainer's opinion has changed. To be fair docker and docker-compose are still super easy to install and use, but it's nothing I would run a production-grade mail cluster on. The possibility to easily scale and use features like liveness probes and metrics monitoring is something I would not want to miss. In addition it is more or less agnostic of the underlying container technology (be it docker, containerd or whatever).

With mailu there's a usable alternative (although not as good as mailcow), that's gaining stars quicker than mailcow, if that is a reliable metric :smile: In my opinion docker is becoming less important for actually running containers on servers.

[smueller18]

Hi @der-eismann,

two months ago, I started creating a helm chart for mailcow at https://github.com/smueller18/mailcow-dockerized/tree/helm. Unfortunately, it is very time consuming and I had to interrupt working on it to prepare a lecture for university.

But be careful, it is not working until a lot more effort is spend for migration. Furthermore, I branched the official mailcow master version and did plenty of modifications that it will fit for my needs (currently still running with docker-compose). But I frequently rebase on the current version of the master so it is very transparent (one commit), which parts I customized.

Current challenges are

• Replace sockets with tcp connections (very painful :cry: )
• Sync upstream changes with highly modified mailcow base for helm
• Placing the right config files at the right places with the right permissions in the runtime containers (I tried to do this automatically with helm functions but if files or directories are added in the upstream, they sometimes have to be synced manually)
• Remove redis logging in favor of Loki

There will be also some requirements for the chart to work with Kubernetes:

• cert-manager
• PVC with ReadWriteMany
• Traefik loadbalancer with IPv6 enabled on host and Kubernetes hostPort (to bind IPv6)

Current restrictions:

• Removed Sogo and Solr
• No backup

I don't think I can finish a first version running this summer. But if anyone want's to start from the helm base I provide, you are very welcome.

I also tried mailu with Kubernetes but because of missing features, I currently still prefer running mailcow-dockerized with docker-compose.

[PrivatePuffin]

two months ago, I started creating a helm chart for mailcow at https://github.com/smueller18/mailcow-dockerized/tree/helm. Unfortunately, it is very time consuming and I had to interrupt working on it to prepare a lecture for university.

I'm also looking into this and understand why the mailcow developers came to their conclussion: Mailcow is primarily about "tying things together", The way docker compose and kubernetes allow to tie containers together, is totally different. Which would basically mean a total "rethinking" of the way mailcow works.

To be fair: I think it's technically easier to build something like mailcow on kubernetes than it is to implement the same in docker-compose and Mailcow can be a great inspiration! :)

[antonengelhardt]

Hello everyone,

it's absolutely wild reading this after all those years 😃

I'm just starting out with K8s and try to deploy my first releases with helm. Then i wanted to deploy a mail server and stumbled upon a Video of Christian Lempa.

Has anything new happened here? The last commit in @smueller18's Repo is 2y old. Still interested in this!

[PrivatePuffin]

Hello everyone,

it's absolutely wild reading this after all those years 😃

I'm just starting out with K8s and try to deploy my first releases with helm. Then i wanted to deploy a mail server and stumbled upon a Video of Christian Lempa.

Has anything new happened here? The last commit in @smueller18's Repo is 2y old. Still interested in this!

There is nothing even close to be working here, let alone anything “wild”.

doing this in kube basically means setting up a new project…

[antonengelhardt]

@Ornias1993

Well, I had to change one or two things but It works for me using docker compose...

So, this is very unlikely to happen?

[PrivatePuffin]

@Ornias1993

Well, I had to change one or two things but It works for me using docker compose...

So, this is very unlikely to happen?

This issue is taling about kubernetes, not about docker-compose, which is a completely different deployment method. As made clear previously in this very thread, a compelete rewrite is needed for kube compatibility.

And no, that's not likely to happen, because it would end up being something that doesn't even look like mailcow.

[MohammedNoureldin]

is there any official statement about this?

Hi, @smueller18 is there any news from your side, as the latest commit in your repo is 3 years old. I am just interested to know how far did you get. Or if it usable in production.

[smueller18]

There is no news. I stopped working on that topic 3 years ago. In this time, I switched to a hosted mailbox and therefore had no needs to run a self hosted mailserver.