search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
757 stars 86 forks source link

Copy opendkim file to container causes crash in build #210

Closed KatsuoRyuu closed 1 year ago

KatsuoRyuu commented 6 years ago

Expected behavior

File copied successfully to the container

Actual behavior

Error ERROR: Service 'mail-server' failed to build: failed to export image: failed to create image: failed to get layer sha256:e1421a08ede1f634d64c374175001be92ae4d72c7d681fcac2a78fe5e92b391f: layer does not exist

Steps to reproduce the behavior

Running docker build that copies file:

-rw-r--r-- 1 root root 1.1K Jan 27 14:10 files/etc/opendkim/opendkim

file contents:

AutoRestart             Yes
AutoRestartRate         10/1h
Syslog                  yes
SyslogSuccess           yes
Canonicalization        relaxed/simple
#ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
#InternalHosts           refile:/etc/opendkim/TrustedHosts
LogWhy                  Yes
Mode                    sv
PidFile                 /var/run/opendkim/opendkim.pid
SignatureAlgorithm      rsa-sha256
KeyTable                dsn:pgsql://{%VIRTUAL_MAIL_USERNAME%}:{%VIRTUAL_MAIL_PASSWORD%}@127.0.0.1/{%VIRTUAL_MAIL_DATABASE%}/table=dkim?keycol=id?datacol=domain_name,selector,private_key
SigningTable            dsn:pgsql://{%VIRTUAL_MAIL_USERNAME%}:{%VIRTUAL_MAIL_PASSWORD%}@127.0.0.1/{%VIRTUAL_MAIL_DATABASE%}/table=dkim?keycol=domain_name?datacol=id
Socket                  local:/var/run/opendkim/opendkim.sock
TemporaryDirectory      /var/tmp
UMask                   007
UserID                  milter:postfix
ReportAddress           {%MAIN_EMAIL%}
Selector                mail
Statistics              /var/lib/opendkim/stats.dat
SendReports             yes

using Dockerfile:

FROM debian:stretch-slim

MAINTAINER e@mail.com

LABEL developer="me"
LABEL version="1.0"
LABEL description="me learning docker"
LABEL maintainer="e@mail.com"
LABEL name="MailServer"

ARG VIRTUAL_MAIL_USERNAME
ARG VIRTUAL_MAIL_PASSWORD
ARG VIRTUAL_MAIL_DATABASE
ARG WEBMAIL_PASSWORD
ARG WEBMAIL_USERNAME
ARG WEBMAIL_DATABASE
ARG DOMAIN

ENV WEBMAIL_PASSWORD=${WEBMAIL_PASSWORD:-"test"}
ENV WEBMAIL_USERNAME=${WEBMAIL_USERNAME:-"webmail"}
ENV WEBMAIL_DATABASE=${WEBMAIL_DATABASE:-"webmail"}

ENV VIRTUAL_MAIL_PASSWORD=${VIRTUAL_MAIL_PASSWORD:-"test"}
ENV VIRTUAL_MAIL_USERNAME=${VIRTUAL_MAIL_USERNAME:-"virtualmail"}
ENV VIRTUAL_MAIL_DATABASE=${VIRTUAL_MAIL_DATABASE:-"virtualmail"}

ENV EMAIL_STORAGE_PATH=/data/mail
ENV APACHE_USER=www-data
ENV DOMAIN=${DOMAIN}

ENV PG_MAJOR=9.6
ENV PATH=/usr/lib/postgresql/$PG_MAJOR/bin:$PATH 
ENV PGDATA=/data/postgresql/9.6/data

# Create users and groups
RUN groupadd -g 2005 postgres
RUN useradd postgres -u 2005 -g postgres -r -m -d /var/lib/postgresql -s /bin/false

RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive \
    apt-get -y install --no-install-recommends \
    wget unzip supervisor strace postgresql-common amavisd-new opendkim gosu \
    opendmarc apache2 php clamav postfix spamassassin dovecot-core libopendbx1-pgsql \
    dovecot-imapd dovecot-pgsql dovecot-antispam ca-certificates; \
    sed -ri 's/#(create_main_cluster) .*$/\1 = false/' /etc/postgresql-common/createcluster.conf; \
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends postgresql-${PG_MAJOR} postgresql-contrib-${PG_MAJOR}; \
    apt-get -y autoclean; apt-get -y clean;exit 0; apt-get -y autoremove; exit 0;

########
# Making sure all the directories are available
####
RUN mkdir -p /etc/apache2/sites-available/ \
        /etc/dovecot/ \
        /etc/opendkim/ \
        /etc/opendmarc/ \
        /etc/postfix/ \
        /etc/postfix/pgsql/ \
        /etc/postgresql/9.6/main/ \
        /opt/mail-server/conf/ \
        /tmp/files/ \
        /opt/mail-server/scripts/ \
        /docker-entrypoint-initdb.d \
        /docker-entrypoint-initdb.d.backup

########
# Copying All the config files that is needed
####
# Install Apache config files
COPY ./files/mail-server/etc/apache2/web.conf                           /etc/apache2/sites-available/

# Install Dovecot configuration files
COPY ./files/mail-server/etc/dovecot/dovecot.conf                       /etc/dovecot/
COPY ./files/mail-server/etc/dovecot/dovecot-sql.conf.ext               /etc/dovecot/
COPY ./files/mail-server/etc/dovecot/conf.d/10-mail.conf                /etc/dovecot/
COPY ./files/mail-server/etc/dovecot/conf.d/10-master.conf              /etc/dovecot/
COPY ./files/mail-server/etc/dovecot/conf.d/15-mailboxes.conf           /etc/dovecot/
COPY ./files/mail-server/etc/dovecot/conf.d/20-imap.conf                /etc/dovecot/
COPY ./files/mail-server/etc/dovecot/conf.d/20-imap.conf                /etc/dovecot/

# Install the OpenDkim config
COPY ./files/etc/opendkim/opendkim                                      /etc/opendkim/opendkim.conf

# Install the OpenDMARC config
COPY ./files/mail-server/etc/opendmarc/opendmarc.conf                   /etc/opendmarc/

# Installing postfix config
COPY ./files/mail-server/etc/postfix/main.cf                            /etc/postfix/
COPY ./files/mail-server/etc/postfix/master.cf                          /etc/postfix/
COPY ./files/mail-server/etc/postfix/pgsql/virtual-alias-maps.cf        /etc/postfix/pgsql/
COPY ./files/mail-server/etc/postfix/pgsql/virtual-mailbox-domains.cf   /etc/postfix/pgsql/
COPY ./files/mail-server/etc/postfix/pgsql/virtual-mailbox-maps.cf      /etc/postfix/pgsql/

# Installing the postgresql config files
COPY ./files/mail-server/etc/postgresql-9.6/postgresql.conf             /etc/postgresql/9.6/main/

# Installing amavisd config
COPY ./files/mail-server/etc/amavisd.conf                               /etc/

# Installing the clam AV service config
COPY ./files/mail-server/etc/clamd.conf                                 /etc/

# Installing Supervisor config
COPY ./files/mail-server/conf/supervisord.conf                          /opt/mail-server/conf/

########
# Copying the SQL files that is needed for Webmail and the virtual mail
####
# Installing database files
COPY ./files/virtualmail.sql                                            /tmp/files/
COPY ./files/webmail.sql                                                /tmp/files/
COPY ./files/0000-add-roles.sql                                         /tmp/files/
COPY ./files/0010-add-database.sql                                      /tmp/files/

########
# Copying executable files
####
# Amavisd
COPY ./files/mail-server/scripts/amavisd.sh                             /opt/mail-server/scripts/

# Apache
COPY ./files/mail-server/scripts/apache.sh                              /opt/mail-server/scripts/

# dovecot
COPY ./files/mail-server/scripts/dovecot.sh                             /opt/mail-server/scripts/

# opendkim
COPY ./files/mail-server/scripts/opendkim.sh                            /opt/mail-server/scripts/

# opendmarc
COPY ./files/mail-server/scripts/opendmarc.sh                           /opt/mail-server/scripts/

# postfix
COPY ./files/mail-server/scripts/postfix.sh                             /opt/mail-server/scripts/

# postgresql
COPY ./files/mail-server/scripts/postgresql.sh                          /opt/mail-server/scripts/

# supervisor
COPY ./files/mail-server/scripts/supervisor.sh                          /opt/mail-server/scripts/

# Entrypoint
COPY ./files/mail-server/bin/entrypoint.sh                              /opt/bin/entrypoint.sh

RUN mkdir -p /var/run/postgresql && \
    chown -R postgres /var/run/postgresql && \
    a2enmod php7.0 && \
    mkdir -p /opt/bin && \
    export PATH=${PATH}:/opt/bin

RUN a2ensite web

RUN chown -R postfix /var/spool/postfix

# Installing databases
RUN mkdir -p /docker-entrypoint-initdb.d/

RUN  ln -s /usr/lib/postgresql/9.6/bin/postgres                 /opt/bin/postgres
RUN  ln -s /usr/lib/postgresql/9.6/bin/pg_ctl                   /opt/bin/pg_ctl

# Fixing the postgresql data dir permissions
RUN mkdir -p ${PGDATA}
RUN chown postgres:postgres -R ${PGDATA}

# initializing postgresql
#RUN gosu postgres initdb

# Create the log and tools directory
RUN mkdir -p /data/log
RUN mkdir -p /opt/mail-server/tools/

# Installing Postfix Admin
RUN wget "https://jaist.dl.sourceforge.net/project/postfixadmin/postfixadmin/postfixadmin-3.1/postfixadmin-3.1.tar.gz" \
        -c -O /tmp/postfixadmin.tar.gz  >> /data/log/install-postfixadmin.log 2>&1 && \
    tar xvf /tmp/postfixadmin.tar.gz -C /opt/mail-server/tools/ >> /data/log/install-postfixadmin.log 2>&1 && \
    mv /opt/mail-server/tools/postfixadmin-3.1 /opt/mail-server/tools/postfixadmin && \
    mkdir -p /opt/mail-server/tools/postfixadmin/templates_c && \
    chown ${APACHE_USER}:${APACHE_USER} -R /opt/mail-server/tools/postfixadmin

RUN sed -i "s#\$CONF\['configured'\] = false;#\$CONF\['configured'\] = true;#g" /opt/mail-server/tools/postfixadmin/config.inc.php; \
    sed -i "s#\$CONF\['database_type'\] = 'mysqli';#\$CONF\['database_type'\] = pgsql;#g" /opt/mail-server/tools/postfixadmin/config.inc.php; \
    sed -i "s#\$CONF\['database_host'\] = 'localhost';#\$CONF\['database_host'\] = localhost;#g" /opt/mail-server/tools/postfixadmin/config.inc.php; \
    sed -i "s#\$CONF\['database_user'\] = 'postfix';#\$CONF\['database_user'\] = {%VIRTUAL_MAIL_USERNAME%};#g" /opt/mail-server/tools/postfixadmin/config.inc.php; \
    sed -i "s#\$CONF\['database_password'\] = 'postfixadmin';#$CONF\['database_password'\] = {%VIRTUAL_MAIL_PASSWORD%};#g" /opt/mail-server/tools/postfixadmin/config.inc.php; \
    sed -i "s#\$CONF\['database_name'\] = 'postfix';#\$CONF\['database_name'\] = {%VIRTUAL_MAIL_DATABASE%};#g" /opt/mail-server/tools/postfixadmin/config.inc.php;

# Installing phppgadmin
RUN wget "https://github.com/phppgadmin/phppgadmin/tarball/master" \
        -c -O /tmp/phppgadmin.tar.gz >> /data/log/install-phppgadmin.log 2>&1 && \
    tar xvf /tmp/phppgadmin.tar.gz -C /opt/mail-server/tools/ >> /data/log/install-phppgadmin.log 2>&1 && \
    mv /opt/mail-server/tools/phppgadmin* /opt/mail-server/tools/phppgadmin && \
    chown ${APACHE_USER}:${APACHE_USER} -R /opt/mail-server/tools/phppgadmin

# Installing RoundCube Webmail
RUN wget "https://github.com/roundcube/roundcubemail/archive/master.zip" \
        -c -O /tmp/roundcube.zip >> /data/log/install-roundcube.log 2>&1 && \
    unzip /tmp/roundcube.zip -d /opt/mail-server/tools/ >> /data/log/install-roundcube.log 2>&1 && \
    mv /opt/mail-server/tools/roundcubemail-master /opt/mail-server/tools/roundcubemail && \
    chown ${APACHE_USER}:${APACHE_USER} -R /opt/mail-server/tools/roundcubemail
RUN cp /opt/mail-server/tools/roundcubemail/config/config.inc.php.sample /opt/mail-server/tools/roundcubemail/config/config.inc.php
RUN sed -i "s#\$config\['db_dsnw'\] = 'mysql://roundcube:pass@localhost/roundcubemail';#\$config\['db_dsnw'\] = 'mysql://{%WEBMAIL_USERNAME%}:{%WEBMAIL_PASSWORD%}@localhost/{%WEBMAIL_DATABASE%}';#g" /opt/mail-server/tools/roundcubemail/config/config.inc.php;

# Installing scripts for execution
## Amavisd
RUN  ln -s /opt/mail-server/scripts/amavisd.sh      /opt/bin/amavisd
RUN  chmod +x /opt/bin/amavisd
## Apache
RUN  ln -s /opt/mail-server/scripts/apache.sh       /opt/bin/apache
RUN  chmod +x /opt/bin/apache
## dovecot
RUN  ln -s /opt/mail-server/scripts/dovecot.sh      /opt/bin/dovecot
RUN  chmod +x /opt/bin/dovecot
## opendkim
RUN  ln -s /opt/mail-server/scripts/opendkim.sh     /opt/bin/opendkim
RUN  chmod +x /opt/bin/opendkim
## opendmarc
RUN  ln -s /opt/mail-server/scripts/opendmarc.sh    /opt/bin/opendmarc
RUN  chmod +x /opt/bin/opendmarc
## postfix
RUN  ln -s /opt/mail-server/scripts/postfix.sh      /opt/bin/postfix
RUN  chmod +x /opt/bin/postfix
## postgresql
RUN  ln -s /opt/mail-server/scripts/postgresql.sh   /opt/bin/postgresql
RUN  chmod +x /opt/bin/postgresql
## supervisor
RUN  ln -s /opt/mail-server/scripts/supervisor.sh   /opt/bin/supervisor
RUN  chmod +x /opt/bin/supervisor
## entrypoint
RUN  chmod +x /opt/bin/entrypoint.sh

RUN useradd milter  -r -m -d /var/lib/milter -s /sbin/nologin
RUN mkdir -p /var/spool/postfix
RUN chown root:root -R /var/spool/postfix
RUN chmod og-w /var/spool/postfix

EXPOSE 9080
EXPOSE 9081
EXPOSE 9082

VOLUME /data

ENTRYPOINT [ "/opt/bin/entrypoint.sh" ]
CMD "/usr/bin/supervisord" "-n" "-c" "/opt/mail-server/conf/supervisord.conf"

Running docker-compose build, docker-compose up -d or docker build ./ will start building the container, a clean build will stop at opendkim file with the error:

ERROR: Service 'mail-server' failed to build: failed to export image: failed to create image: failed to get layer sha256:e1421a08ede1f634d64c374175001be92ae4d72c7d681fcac2a78fe5e92b391f: layer does not exist

Running it a second time will continue the build, there will be no problems after and the files is copied correctly. If I uncommenting the opendkim file will make it build from clean with no problems.

I have tried renaming the file moving the file and change permission, with no result.

Output of docker version:

Docker version 18.01.0-ce-dev, build 486a48d

Output of docker info:

Containers: 11
 Running: 11
 Paused: 0
 Stopped: 0
Images: 1931
Server Version: dev
Storage Driver: btrfs
 Build Version: Btrfs v4.14
 Library Version: 102
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f2 (expected: 89623f28b87a6004d4b785663257362d1658a729)
runc version: b2567b3 (expected: b2567b37d7b75eb4cf325b77297b140ea686ce8f)
init version: v0.16.1 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.14.11-gentoo
Operating System: Gentoo/Linux
OSType: linux
Architecture: x86_64
CPUs: 20
Total Memory: 196.6GiB
Name: localhost
ID: BD7A:YHDE:WBNJ:RPY2:2J72:R73P:NLUS:2JC3:NHTF:LYUM:GAAO:QAXS
Docker Root Dir: /srv/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 nexus.admin.new.ryuu.technology:8082
 registry.admin.new.ryuu.technology
 registry.admin.new.ryuu.technology:5000
 172.17.0.2:8082
 172.29.0.2:8082
 173.212.244.183:5000
 new.ryuu.technology:5000
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.) Proxmox virtual machine on qEmu

Keyboard            Layout Default
Memory              200.00 GiB
Processors          20 (1 sockets, 20 cores) [numa=1]
Display             Default
CD/DVD Drive (ide2) Images:iso/install-amd64-minimal-20171228T214501Z.iso,media=cdrom,size=268M
Hard Disk (scsi0)   local:100/vm-100-disk-1.raw,size=80G
Hard Disk(scsi1)    local:100/vm-100-disk-2.raw,size=2000G
Network (net0)      virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0
troublete commented 6 years ago

Were you able to solve this problem somehow?

KatsuoRyuu commented 6 years ago

Unfortunately not. I have instead set my CI/CD system to run that build twice for it to work. This is not a good way to do it but its a fix.

troublete commented 6 years ago

I had the same build error last week... It seems to be related to COPY. I've updated my docker version to 18.05-ce what fixed it. It seems really weird.

KatsuoRyuu commented 6 years ago

Thanks, ill try to update and see if that helps

thaJeztah commented 1 year ago

Let me close this ticket for now, as it looks like it went stale.