Can't start docker after Kernel update (4.9)

[DJIronic]

I have problem with mailcow-dockerized after updating Kernel.

1) I updated it, there was a problem with Docker launching, so I deleted "aufs" folder as was mentioned here in https://github.com/mailcow/mailcow-dockerized/issues/309 (I also used guide how to update the kernel from this page.

2) now whem I go into /home/mailcow-dockerized folder and I try to run docker-compose up -d I get this:

root@MailServer:/home/mailcow-dockerized# docker-compose up -d
Pulling pdns-mailcow (andryyy/mailcow-dockerized:pdns)...
ERROR: repository andryyy/mailcow-dockerized not found: does not exist or no pull access
root@MailServer:/home/mailcow-dockerized#

Server Details: Debian 8: Linux MailServer 4.9.0-0.bpo.4-amd64 #1 SMP Debian 4.9.51-1~bpo8+1 (2017-10-17) x86_64 GNU/Linux RAM: 2GB CPU: 2 (3,39Ghz) cores Gigabit network (OVH)

So now I can't start mailcow-dockerized. I want to run it with actual configuration, I did the update because of many random server freezes (no full memory). So now I want to keep my data at the first place. This going to be a long night for me as this is our company server and they need to access their mails..I know..I should do the snaphshot in VMware before this action..I was tired.

Thank you so much for your help. :)

[DJIronic]

And...I just realized that Docker can't see any of my containers (using docker ps command). 👎

https://i.imgur.com/hceZC9R.png

[mkuron]

It says pdns-mailcow in the output you provided. Are you running some ancient version of Mailcow? We haven‘t been using PowerDNS for a long time... If you want to bring up this old version, you need to build all images manually:

docker-compose build
docker-compose up --no-build -d

[DJIronic]

It was not managed by me for a long time. I was deployed on another projects => it was nto my job to take care about this server...and they called me "something is not working". So...I don't know. :/ But Thank you for your answer!

docker-compose build:

docker-compose up --no-pull -d:

[mkuron]

This seems to be an older docker-compose version. Please remove all image: lines from docker-compose.yml and try docker-compose up -d.

[DJIronic]

I just removed all lines containing image: and I got this:

[mkuron]

Then you need an appropriate dockerfile line to the rspamd-mailcow section in docker-compose.yml. That was peobably missing in the version you are using.

[DJIronic]

OK...and where I can get it? Sorry...I am totally new to docker..Mailcow is my first thing running on it.

So sorry for dumb questions.

[DJIronic]

OK, I tried to change image: link for rspamd-mailcow from actual docker-compose.yml from GitHub. Then it asked for nearly every other app in the same way so I was replacing links the new ones from actual github file...and now it´s asking for image: link for pdns-mailcow:

So I don't know, if I did it right to get "an appropriate dockerfile line" and don't know, what to do now.

[mkuron]

What commit are you on?

[DJIronic]

I am not sure where to get it, but mailcow-dockerized was installed 22.2. 2017 (read from last edit on some folders).

[mkuron]

git log or git describe should tell you.

[DJIronic]

This is the top of history file, seems like last commit from Andryyy is 15bffcd447c669453a521d69d659eeaddad31345 and from André Peters it´s 2ebdb4a54ba69a649e751b56ff9860cdc0cdd240

[mkuron]

I think you didn‘t make the right modifications to the docker-compose.yml file. You need to delete the image: lines and add ones like build: ./data/Dockerfiles/pdns, using the correct directory instead of pdns for every service.

[DJIronic]

OK, I started to removing all image lines like image: andryyy/mailcow-dockerized:dovecot but there is no line with build but I just met something like this:

      depends_on:
        - pdns-mailcow

OK...to make it clear, here is my actual (original docker-compose.yml). and this is what I am doing with it (changed first few servicies)...is that right?

[mkuron]

No, you need to add new build lines to every service. And don’t modify the dependencies.

[DJIronic]

OK...I am so sorry...you must by totally angry at me now for my stupidity...so sorry, I am trying.

Did I get it now?

services:
    pdns-mailcow:
      build: ./data/Dockerfiles/pdns
      depends_on:
        mysql-mailcow:
          condition: service_healthy
      volumes:
        - ./data/conf/pdns/:/etc/powerdns/
      restart: always
      networks:
        mailcow-network:
          ipv4_address: 172.22.1.254
          aliases:
            - pdns

    mysql-mailcow:
      build: ./data/Dockerfiles/mysl
      healthcheck:
        test: ["CMD", "mysqladmin", "ping", "--host", "localhost", "--silent"]
        interval: 10s
        timeout: 30s
        retries: 5
      volumes:
        - mysql-vol-1:/var/lib/mysql/
        - ./data/conf/mysql/:/etc/mysql/conf.d/:ro
      dns:
        - 172.22.1.254
      dns_search: mailcow-network
      environment:
        - MYSQL_ROOT_PASSWORD=${DBROOT}
        - MYSQL_DATABASE=${DBNAME}
        - MYSQL_USER=${DBUSER}
        - MYSQL_PASSWORD=${DBPASS}
      restart: always
      networks:
        mailcow-network:
          aliases:
            - mysql

    redis-mailcow:
      build: ./data/Dockerfiles/redis
      depends_on:
        - pdns-mailcow
      volumes:
        - redis-vol-1:/data/
      restart: always
      dns:
        - 172.22.1.254
      dns_search: mailcow-network
      networks:
        mailcow-network:
          aliases:
            - redis

[mkuron]

Yes, that looks sensible.

[DJIronic]

OK, now it´s like this. Now should I run docker-compose up -d?

[mkuron]

Do docker-compose build first. Then bring up the containers.

If that doesn‘t work or the build fails, you‘ll need to upgrade to the current Mailcow version or reinstall a new Mailcow and restore your data from backup.

[DJIronic]

OK, ending with error Cannot locate specified Dockerfile: Dockerfile and that's interesting. I am in folder /home/mailcow-dockerized and folder in .yml file is ./data/Dockerfiles/mysql.

[stevesbrain]

@DJIronic I wasn't using mailcow-dockerized this long ago, so I don't have experience with the options that were available back then. However, given you've inherited this, and are now expected to get this working + keep it working, I'd suggest backing up, and restoring your email data to the latest mailcow version.

This will mean you'll be getting the latest updates, and updating is much easier (just run ./update.sh - that's literally it). You'll be able to be better supported here, and you'll have a better idea yourself of how it's all running (given you've set it up from scratch).

Also, regarding:

OK...I am so sorry...you must by totally angry at me now for my stupidity...so sorry, I am trying.

I don't think anyone is going to get angry at you :) We're here to help, and you've inherited something and are expected to hit the ground running with zero experience in this project; we can certainly sympathise with the position you're in!

[DJIronic]

OK, updating to newest version and then recovering all the data back is a nice idea, but how to do that.

I updated the system once a loong time ago..then it was mine, running updated and without problems. But now...even the ./update.sh file is missing from the mailcow-dockerized folder. :D This sevrer is in total dead state...so what now? Download update.sh from actual GitHub repository and use it to update my old mailcow?

[DJIronic]

And of course, thank you for understanding my situtation. :) I am trying to do my best o understand you and to don't have stupid questions but sometimes I just don't have any idea what something mean. Now I am creating VMware snapshot to prevent me doing even more damage to all the files.

[DJIronic]

OK...I am probably totally stupid and I probably just killed all the data...

I tought I created snapshot of the server in VMware to be safe trying the update script...but I missclicked and I created an snapshot of another server and I run the update...no mailcow is running, without any of my files and configs... GJ me...I probably just killed all the hope.

[stevesbrain]

@DJIronic How did you update? Normally, you'd do two things:

a) Pull the latest git snapshot with git pull (only necessary this time because you're behind) b) Run the update with ./update.sh

If, after this, it's still dead, you can recover - the docker data containers should still exist. Do you have any other backup aside from snapshot btw? If you wanted to backup from old host, restore to a new host, you could follow this for Mail, and this for MySQL

If you get really stuck, I occasionally do remote work - I might have some time to assist

[DJIronic]

Unfortunately, there is no backups...just the broken server...and I just created a snapshot fo bad server = I have nothing right now...this server was not managed by me...so no backups at all...I think none cared about it...

So what I did is just downloading newest update.sh and running it, that´s all. Am I still able to save my data? (I did the snapshot just now after I realized what I have done..)

[stevesbrain]

@DJIronic As long as you didn't overwrite your server with the snapshot of another server, then it should be fine. Docker keeps the volumes in a specific directory. Can you do this and tell me the output? It should be similar to the following:

[root@stevesbrain mailcow-dockerized]# ls /var/lib/docker/volumes
mailcowdockerized_crypt-vol-1  mailcowdockerized_dkim-vol-1  mailcowdockerized_mysql-vol-1  mailcowdockerized_postfix-vol-1  mailcowdockerized_redis-vol-1  mailcowdockerized_rspamd-vol-1  mailcowdockerized_vmail-vol-1  metadata.db

If you listed the vmail dir, you should see:

[root@stevesbrain mailcow-dockerized]# ls /var/lib/docker/volumes/mailcowdockerized_vmail-vol-1/_data/
domain1.com domain2.com

[DJIronic]

OK...I see hosted domains = data should be still there! Nice, thank you...

so OK, here is the output:

Code version:

root@MailServer:~# ls /var/lib/docker/volumes
mailcowdockerized_crypt-vol-1  mailcowdockerized_postfix-vol-1  mailcowdockerized_vmail-vol-1                                     96caab97240cd60f6ade28e1ac643e08dd16a6885a16d77d0b3acdbbb72e6010
mailcowdockerized_dkim-vol-1   mailcowdockerized_redis-vol-1    metadata.db                                                       99cdc8be7c318966cc3589b71134752a40a3bd3a5c0dad299d53b06986093659
mailcowdockerized_mysql-vol-1  mailcowdockerized_rspamd-vol-1   4232a00103a467358a5d9f8d9b127f160f1edbbfa24e4933fd74b7ac567e4b76

[stevesbrain]

Great, I dare say things shall be fine then. This is what I'd do at this point, given it's not running anyway:

1. Clone the repo to a new directory
2. Copy your config file/relevant parts of your mailcow.conf (or run generate_config.sh and check mailcow.conf against your previous mailcow.conf)
3. Then, run docker-compose pull; docker-compose up -d

Let me know how that goes :) Given it's so far behind, some further manual intervention may be required (I think there was some redis data cleanup a while back, etc)

[DJIronic]

OK, I did it all.. cloned repo to /root and then created new config. I hope that everything is same...

then I run the two commands, end up with this:

ooh...and now I realize that DBPASS adn DBROOT is differenet in the config (and things that don't even exists in the old config). Is that a problem?

[stevesbrain]

Yeah, set your DB stuff to be the same, then recreate: docker-compose down; docker-compose up -d, then check that the VM's are running: docker ps -a and if all is good, make sure you can log in and check your mail, send and receive, etc.

[DJIronic]

You are genius! :) Thank you very much!

Recieving:

Sending:

So yes, nearly everything is working..there is just one small thing..:( My boss hates SOGo so he wants to have RoundCube installed too...I did it in some way on the old installation of Mailcow..but I am not sure if it was the right ways. It was running under https://mydomain.tld/rc. Now I have Nginx 404 there. So is there any normal and right way how to do it.

But if you tired and don't have or you just had enough for me. Don't mind id or answer it later...I can survive some loud words for RC, but not for non-working mail server...so just saved me :) Thank you again.

[stevesbrain]

Glad it's working now :) Give me a few minutes for RoundCube - I'll see what's the best way to go about it!

[stevesbrain]

@DJIronic Actually, I think if you just follow this page that should give you all you need :)

[DJIronic]

OK, I will do another complete backup of server (snapshot) to be sure that I will not break anyting with my changes...and we will see.

Thank you very much again.. :)

[stevesbrain]

That's alright, no worries @DJIronic :) Just a point to note - Snapshots shouldn't be used as backups (at least not long term) for a couple of reasons:

• They can corrupt
• They cause your storage to grow exponentially

It's fine for short term stuff, like making sure you can quickly roll back, but I'd definitely suggest you implement a proper backup solution for the future :) Don't forget to delete the snapshot when you're done!

[DJIronic]

Hi guys,

sorry for re-opening this thread again..after hours and hours of my questions. But I have two more, but less urgent ones...so take your it if you want.

I am running my mailserver with new valid SSL for all domains and all is fine now, so thank you again.

But I am struggling with RoundCube. I went through the configuration and now I have problem to connect it to the database. I am using DB login and password from DBPASS and DBROOT inside mailcow.conf and mailcow as a DB name. But there is still a connection error on the RD page.

So that's it.

And my second question is about backups, I have access to corporate unlimited Google Drive, but I don't have any idea how to backup server based on docker. I want to keep mail messages and configuration safe on the Gdrive..preferably encrypted. I know how to backup files (create encrypted archive and upload into Gdrive using rclone) with autmaticaly created folder structure. But as there is the docker thing, I am lost.

So thank you again for your time. I am so happy that our corporate mailserver is back and working..and I still have my job.

[stevesbrain]

@DJIronic As for roundcube, I probably won't have time to look until tomorrow (my time) sorry - so roughly 12 hours. Maybe someone else will have time before then (though you'll have to actually "reopen" the issue, as it's still closed).

Regarding backups, there's two ways to do it.

1. You can backup by running the specific export commands listed in the mailcow dockerized doco website - this will duplicate your storage requirements for the duration of the backup (more or less). Or,
2. The way I do it - I backup up the Docker volumes folder (as all data is simply kept there as files), and my mailcow conf folder (I actually backup the whole folder - git repo + mailcow.conf, but you could just backup your custom bits).

rclone is good - duplicity might be even better for you (parity bits, point-in-times snapshots, etc), or borg. Check them out anyway :)

[DJIronic]

Unfortunately, I can't reopen this issue.. there is no button on the place where it normally is.

You don't need to apologize :) this is not your work. Take your time and do what you need.

Thank you so much for backup options, I will check them out ASAP.

[stevesbrain]

@DJIronic I tried this out with a fresh install, and everything worked fine. Did you copy the config.inc.php exactly as is from the setup instructions, or did you modify this after copying?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.