ARM64 Support

[extremeshok]

I have extensive experience with docker on arm, so before I get to work on this, has anyone got mailcow-dockerized ported or running on arm ?

Did a quick search through the repo and I do not see any references to arm.

Does the repo owner have any issues against arm support being added ?

Thanks

[mkuron]

A Raspberry Pi doesn’t have enough memory. You need around 1GB if you disable ClamAV or 2GB if you don’t. Recent Odroid models should be sufficiently equipped though.

Please give it a try and open a pull request if it works out. You‘ll need to swap out the base images and use arm instead of amd64 repositories where available and compile yourself where not. Just don’t duplicate all the Dockerfiles as that would create a maintenance nightmare.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[MatthiasLohr]

@extremeshok Did anything happen here?

[David91919]

RAM issue on RPI is solved by now, would be great to see MC on RPI

[andryyy]

RAM is/was the smallest problem. Maintenance is another, but the biggest problem is missing software for ARM.

I won't maintain mailcow on ARM, but feel free to port it and create a PR etc. - I think SOGo won't work reliable on ARM though.

[Braintelligence]

Some people even have issues with 4GB RAM if they use ClamAV and SOLR, so I wouldn't say that RPi4 would have none at all.

[Adorfer]

Until ARM-based 1U servers (with decent amount of RAM) become largely availabe at reasonable cost, ARM is not a suitable platform for mailservers on (small) enterprise level.

[bolet]

I have the mailcow stack running on armhf (odroid hc1, 8-core, 2gb), with, as expected, the usual resource hungry services disabled. It consumes less than 1gb, and I plan to run it on a (small) swarm for better resilience (with glusterfs as persistent storage). I'd argue that arm is a perfect platform for small traffic servers, which is my case.

The major part was rebasing debian/strecth-slim and ubuntu/bionic services on debian/buster-slim (the original distros were lacking armhf versions of some required packages). The other part was building mariadb:10.3, which also resulted in rebasing on debian/buster-slim. I believe the rebase on a common distro would benefit the consistency of the project.

Anyway, I'd be glad to share the patch if anyone is still interested.

[Adorfer]

offereing "state of the art" anti-spam and FTS are the key features of mailcow (and to offer it as a real alternative to hosting at gmail/M365 etc). getting rid of that sounds like offering a modding/patch for a semi-truck getting rid of the hook and any lockable doors. Of course it's still a truck, but does not fit in most use cases.

[bolet]

Let's not be dictators! What about letting users decide how they want to configure mailcow? Isn't it one of the freedoms permitted by "free software"? :-)

I'll personally be happy to enable all the features when I switch to a higher end (arm64 based) server, but until then I still prefer using mailcow stripped from a few features than configuring a mail server from scratch (which is what I've been running for about 20 years).

[luckyluc74]

@bolet Would be great, if you could share the patch for mailcow stack for armhf. And I have to agree with you that there are users that are very interested to see mailcow stack running on armhf/aarch64, i am one of those users. I am currently Running 3 raspberry pi 4 (4GB) in swarm and would be nice to run arm version of the mailcow stack on it.

Very recently Ubuntu released version 19.10 with raspberry pi 4 support ( aarch64) . Would be great to also see a aarch64 version of mailcow.

[bolet]

@luckyluc74 Here is the patch : buster+arm.zip

Sogo image from mailcow runs but is broken because it still pulls amd64 code (I keep getting 502 errors).

This project builds sogo from sources and is a good starting point. But then I found out that debian:buster includes pretty recent (4.0.7) Sogo packages, so here is my patch to this last project.

All pieces seem to work now, but I haven't yet taken the time to put them all together. I'll keep this bug posted as I move on, but feel free to help :-)

[luckyluc74]

@bolet thank you for sharing the patch and extra info. That SOHO is not working is ok for me, because I am going to use Roundcube instead of SOHO anyway.

[bolet]

@luckyluc74 I since forgot about this patch. It has to be applied to julienfastre project which results in a working arm sogo build. Then use it with the rest of mailcow.

Since then I also tested mailu which is more lightweight than mailcow, and more fit to rpi like servers.

[luckyluc74]

@bolet thanks for the update and information, appreciated! I have to say I like Mailcow. But mailu is also nice.

[Xeroxxx]

So now (acutally since May 2020) Raspberry Pi 4 with 8GB and VMware ESXi on ARM is available since October I would be happy to see offical support :)

Works patched within a ARM VM like a charm except SoGo (patch above) :)

[bonanza123]

@Xeroxxx I am trying to run mailcow on RPi4 (8GB). Could you link me to the correct docker images to use?

[bolet]

@bonanza123 As far as I know, there are no working mailcow images for any ARM platform. My previous patches still need to be applied and the image built on your device.

The maintainers of mailcow don't seem eager to include ARM support, and though I am happy to participate here and there in various projects, I don't have the resources to maintain an up to date fork of mailcow. This unfortunate situation is common in open source projects...

PS: With Apple's M1 chip showing ARM can outperform Intel/AMD chips (at equal watts), there is hope for a growing consideration for ARM.

[mkuron]

The maintainers of mailcow don't seem eager to include ARM support

The only place you could currently run it on right now is pretty much a Raspberry Pi at home, and we really don‘t encourage running a mail server at home. ARM rackmount or cloud servers are still not common (and not cheaper or higher performance than x86). As long as nobody makes a compelling case beyond "nice to have", I still see no reason to add ARM support to Mailcow. It would take additional effort for testing, and as long as nobody is going to use it productively, that‘s not worth it.

With Apple's M1 chip showing ARM can outperform Intel/AMD chips (at equal watts), there is hope for a growing consideration for ARM.

Not a typical Linux machine, unfortunately. It‘s a very nice chip though and I hope we will at some point see similar chip designs in the server market.

[bolet]

we really don‘t encourage running a mail server at home

That's a sensible opinion, but please have some consideration for ppl with a different one. My experience running a home mail server has been fine for over 20 years (with admittedly a friend running a secondary MX).

It would take additional effort for testing

I can understand that. Maybe a solution would be to discard official support for ARM, and let the community take care of this specific platform? I'd argue the whole project would benefit from this. It's a somewhat similar situation to supporting more compilers, it usually ends up making the code more robust.

In my case, though mailcow was my 1st choice, I ended up installing mailu on my home ARM swarm, and I got quite familiar with it (I even patched most images). Months later, I had to deploy a mail server for my company, and because my knowledge of mailu, I also installed it there even though we had a high end x86 server. Don't you I think it's a pity?

[MatthiasHertel]

The maintainers of mailcow don't seem eager to include ARM support

The only place you could currently run it on right now is pretty much a Raspberry Pi at home, and we really don‘t encourage running a mail server at home. ARM rackmount or cloud servers are still not common (and not cheaper or higher performance than x86). As long as nobody makes a compelling case beyond "nice to have", I still see no reason to add ARM support to Mailcow. It would take additional effort for testing, and as long as nobody is going to use it productively, that‘s not worth it.

With Apple's M1 chip showing ARM can outperform Intel/AMD chips (at equal watts), there is hope for a growing consideration for ARM.

Not a typical Linux machine, unfortunately. It‘s a very nice chip though and I hope we will at some point see similar chip designs in the server market.

@mkuron

i would be very happy about the possibility to deploy mailcow on arm - short - thumbs up and feature request for arm! it is absolutely nice to have for me!

[maroci]

I'm adding my wish to other requests to have armhf image of MailCow. :) I really hope to get one to test soon. Thank you

[alfonsrv]

Any more recent patch @bolet? Possibly maybe even with SOGO recompiled? 🚀

[bolet]

As I said earlier, I can't afford to maintain a fork of mailcow for arm. If one day the official mailcow project wants to support arm, I'll be happy to participate. The main hurdle is the inclusion of pre-compiled amd64 binaries in the build process. Just compiling those binaries at build-time is all it takes to have mailcow working on all architectures.

[tehsunnliu]

+1 for ARM

[maroci]

@bolet I would like to learn to do what you said. I don't have knowledge to do that no though though. If anyone has time and patience to teach me how, I would like to give a try. Even few links to a tutorial.

[alfonsrv]

Unfortunately this is the only book that teaches you what there is to do to make mailcow-dockerized run on ARM.
I think honestly @bolet's patch should still work fine and produce up-to-date containers.

[axkng]

Is there any update on this from the developers?

ARM is gaining more and more momentum in the datacenter. Graviton instances on AWS have high sales and Oracle is offering a free tier that gives you a lot power of on ARM at zero cost. So I would say this will be a major thing in the near future.

[mkuron]

I think we‘ll soon get to the point where ARM cloud servers will make good mail servers. However, we‘re not there yet. I am not aware of ARM offers with significantly better price/performance ratios than x86 (Oracle‘s free tier is pretty cool and I wasn‘t aware of it, but nobody should trust their email to a company they are not paying). There undoubtedly will be such an offer one day, and only once that is the case will it make sense to provide Mailcow images for ARM. It comes at a nonzero maintenance effort, which makes it not worth it until it actually has users. We know that Mailcow can be made to run on ARM since people have tried, so there is no need to provide images already simply out of preparation for a future that hasn‘t arrived. Hopefully by that time SOGo will be providing ARM packages too, saving us the effort of building our own.

Don‘t get me wrong, I am a big fan of ARM chips given the latest developments in Apple computers and high-performance computing (the currently fastest supercomputer runs on ARM). I just don‘t think it‘s a good idea to provide untested ARM images as long as none of the Mailcow maintainers are using ARM servers of their own.

[zoeleu]

Hey, Oracle Cloud has free Ampere A1 (ARM) servers with lots of cores and RAM. Why is mailcow not supported? Why is this issue closed?

[mkuron]

@matteoturini, please read my comment above. You really don't want to run critical infrastructure like your mail server on a free cloud service like Oracle's -- a service you don't pay for could easily vanish on short notice, or you might have a hard time getting support from them when something breaks. As long as people like @andryyy or myself don't have ARM servers, we don't really want to publish ARM images because they would be completely untested. That being said, almost everything will work fine when you build the images on ARM yourself (minor changes will be required to pick up the correct package repositories etc., but nobody has bothered to submit a pull request with these so far). Based on my own experience with cross-architecture software development, my prediction is that there will be some minor issues that will need to be fixed. Software like Postfix should work fine because it's so widespread, but for example I have no idea whether SOGo has been tested much on ARM.

[Adorfer]

it is an upstream issue since many of the used components (Sogo first) are not (yet) mature for armf platform. And mailcow does not have enough dev power to maintain those armf ports on a constant basis.

a) Either we wait for those upstream projects evolve or b) a team of devs joins with PRs and maintains them on a constant base

the the load of b) will reduce by time until a) happens, but i assume that's a project for about 2-3 years from now on.

[dnwk]

Raspberry Pi now has 8G memory version. ARM support should definitely be reconsidered.

[Adorfer]

ARM support should definitely be reconsidered.

Then go ahead, neccesary steps are explained above by mkuron, me and others. Please submit PRs to fix the upstream issues e.g. in SOGO.

[Peddaahh]

@MAGICCC Is now easily possible to run on ARM with the amd64 builds. Docker now supports --platform=linux/amd64 (I look at you.. yes YOU Apple M1...) With this you can specify which architecture the image needs to run with. In the docker-compose file you can specify it with platform: linux/amd64 Example:

version: '2.4'
     services:

    unbound-mailcow:
      image: mailcow/unbound:1.15
      platform: linux/amd64
      [..]

Tested on Ubuntu 21.10 with the newest QEMU sudo apt install qemu binfmt-support qemu-user-static Seems to work pretty fine without any noticeable issues - container logs look fine too - no noticeable performance issue due to emulation

[mkuron]

Seems to work pretty fine without any noticeable issues - container logs look fine too - no noticeable performance issue due to emulation

Since this is software emulation of a completely different CPU, major performance degradation (in the 50-90% range) can be expected in my experience. That's fine for testing purposes and you might even run a mail server for 2-3 users on it, but I certainly wouldn't recommend it.

I look at you.. yes YOU Apple M1

Again, a scenario I wouldn't recommend. You cannot run Docker directly on macOS -- it actually runs in a virtual Linux machine. As such, you'll need to be careful with networking (NAT, open relay, etc.). Again, fine for testing and development purposes, but you don't want to put this out on the open internet.

[Peddaahh]

Yes @mkuron This post was definitely meant for those tinkers out there. It works without noticeable issue, however yes there will be performance degradation. With my private domain and 5 Mails performance looks fine. Also I wouldn't say this is high available and 101% functional as there are still minor errors in the logs which don't affect the service as I assume

[SkanderHelali]

@Peddaahh Just tinkering with the Oracle free plan; I installed the linuxserver docker-compose image for arm. (v1.29.2)

Could you elaborate a little more on which services you set "platform: linux/amd64" to? I tried setting it for all mailcow services, but I'm still unable to get it to run on an arm system. I'm ending up with a 502 gateway error and rspamd restarting.

[unixfox]

@Peddaahh Just tinkering with the Oracle free plan; I installed the linuxserver docker-compose image for arm. (v1.29.2)

Could you elaborate a little more on which services you set "platform: linux/amd64" to? I tried setting it for all mailcow services, but I'm still unable to get it to run on an arm system. I'm ending up with a 502 gateway error and rspamd restarting.

You can't do that, oracle VMs don't support virtualisation.

[Peddaahh]

@Peddaahh Just tinkering with the Oracle free plan; I installed the linuxserver docker-compose image for arm. (v1.29.2)

Could you elaborate a little more on which services you set "platform: linux/amd64" to? I tried setting it for all mailcow services, but I'm still unable to get it to run on an arm system. I'm ending up with a 502 gateway error and rspamd restarting.

It's running on my always free Oracle ARM instance. I am running Ubuntu 21.10 with the newest qemu version. Ubuntu lts seems not to work because it seems to not support the newest qemu version. Set the release branch to latest and do a release upgrade

Qemu: sudo apt-get install qemu binfmt-support qemu-user-static and https://www.smartling.com/resources/product/building-multi-architecture-docker-images-on-arm-64-bit-aws-graviton2/

Those two are the only things i modified on my system. Be sure to have the newest Docker version using Dockers convenient script from their website... not the outdated apt installation. Here is my docker-compose.yaml: https://pastebin.com/kQD9qGwp

Using the newest docker-compose seems to be no issue.

Keep in mind that if you don't have IPv6 to properly disable it: https://mailcow.github.io/mailcow-dockerized-docs/firststeps-disable_ipv6/ solved issues for me

[SkanderHelali]

That worked surprisingly well, thanks. Ported over a backup of a sizeable mail server and it seems to run fine - can send and receive emails just fine, sogo works, and so does nextcloud.

https://mxtoolbox.com/diagnostic.aspx seems to always timeout, despite other services seeing it fine (and receiving emails). Will look into it a bit more, but that's a pretty quick hack to get it to work! Thanks.

Edit: Looks like they block port 25 by default, mystery solved.

[Peddaahh]

Edit: Looks like they block port 25 by default, mystery solved.

https://support.oracle.com/knowledge/Oracle%20Cloud/2787393_1.html

Yes, indeed they do for OCI tenancies created after June 23, 2021. Mine was created before so I had no problem. Solution one would be to force tls for your mailbox Solution two would be to write them a short request for opening port 25 as they stated here https://docs.oracle.com/en-us/iaas/releasenotes/changes/f7e95770-9844-43db-916c-6ccbaf2cfe24/

[gregology]

Set the release branch to latest and do a release upgrade

@Peddaahh, how did you do that? And thanks for sharing your docker-compose.yaml 🙏

[Peddaahh]

Set the release branch to latest and do a release upgrade

@Peddaahh, how did you do that? And thanks for sharing your docker-compose.yaml 🙏

sudo apt install update-manager-core then sudo nano /etc/update-manager/release-upgrades replace Prompt=lts with Prompt=normal then sudo do-release-upgrade

[SkanderHelali]

Set the release branch to latest and do a release upgrade

@Peddaahh, how did you do that? And thanks for sharing your docker-compose.yaml pray

sudo apt install update-manager-core then sudo nano /etc/update-manager/release-upgrades replace Prompt=lts with Prompt=normal then sudo do-release-upgrade

@gregology or you could just move over to 22.04 dev builds since you shouldn't be running this in production anyway :)

[MAGICCC]

Maybe an idea for the docs as community supported platform like we have it for the reverseproxies? Not sure which the best subsection would be for this kind of doc

[SkanderHelali]

@Peddaahh What are you getting when you run docker-compose exec netfilter-mailcow iptables -L on the Oracle ARM VM? I keep getting: iptables v1.8.7 (legacy): can't initialize iptables table `filter': iptables who? (do you need to insmod?) Works outside the docker container obviously. I tried building netfilter for arm64v8 instead of the qemu route, iptable_filter is def loaded in the kernel image. 🤔

Edit: Scratch that, building netfilter for arm64v8/alpine did work, I just had the image set incorrectly in the compose file. iptables works and the container isn't restarting anymore. Zero issues with the setup since this one is fixed, surprisingly! Activesync, sogo, and everything is performing well.

[Peddaahh]

@Peddaahh What are you getting when you run docker-compose exec netfilter-mailcow iptables -L on the Oracle ARM VM? I keep getting: iptables v1.8.7 (legacy): can't initialize iptables table `filter': iptables who? (do you need to insmod?) Works outside the docker container obviously. I tried building netfilter for arm64v8 instead of the qemu route, iptable_filter is def loaded in the kernel image. 🤔

Edit: Scratch that, building netfilter for arm64v8/alpine did work, I just had the image set incorrectly in the compose file. iptables works and the container isn't restarting anymore. Zero issues with the setup since this one is fixed, surprisingly! Activesync, sogo, and everything is performing well.

This issue seems to be happen randomly.. some friends of mine complained about that too, some didn't have the issue like me... but yes building this single Container for arm seems to evade the issue... happens perhaps due to the emulation with the amd64 image..

[zerofoxfive]

@Peddaahh After a couple of failed attempts of cobbling together your advice in this thread, a re-think, and a final attempt I was able to get mailcow up and running on an Oracle Cloud Ampere based arm (free) instance. The only item that I have not completed is asking Oracle Cloud to update the PTR to the instance IP, as I will likely rebuild this again in a new VCN. This isn't badly affecting my spam scores with SPF and DKIM alignment though, so seems good enough for testing.

The weirdest issue I had was that docker-compose would not run when installed with the commands in the instructions. The resulting file was 9K and would throw errors. I ended up with success with an sudo apt get docker-compose and was able to bring the service up.

I'm brand new to mailcow (will likely need to migrate a small number of Google workplace (legacy) accounts, when they get retired this year, so exploring low cost options to keep these mailboxes up and running. I have previously setup mailinabox, however the containerized nature of mailcow and the active development makes this look like a good options.

On the topic of arm support, with the model that you have put forward for emulation it could allow for the containers that do support compilation for arm64 to be done so natively, and the remainder that do not have that upstream support to be emulated until such a point that they do (by setting the platform in the docker-compose.yml. Not withstanding that this model would likely remain unsupported and not recommended for production workloads etc.

However the generous always free tier on Oracle Cloud makes it an attractive hosting option for small workloads at the very least.

Thanks for your input and insight into this thread on this topic.

[klazarev]

@Peddaahh What are you getting when you run docker-compose exec netfilter-mailcow iptables -L on the Oracle ARM VM? I keep getting: iptables v1.8.7 (legacy): can't initialize iptables table `filter': iptables who? (do you need to insmod?) Works outside the docker container obviously. I tried building netfilter for arm64v8 instead of the qemu route, iptable_filter is def loaded in the kernel image. 🤔 Edit: Scratch that, building netfilter for arm64v8/alpine did work, I just had the image set incorrectly in the compose file. iptables works and the container isn't restarting anymore. Zero issues with the setup since this one is fixed, surprisingly! Activesync, sogo, and everything is performing well.

This issue seems to be happen randomly.. some friends of mine complained about that too, some didn't have the issue like me... but yes building this single Container for arm seems to evade the issue... happens perhaps due to the emulation with the amd64 image..

And it is possible in more detail? What to delete? What to create?