Replication issues

[SowAbdoul]

I've got the issues with replication When I create a zone on the master, the zones do not replicate on the slave. With php admin, I see my zones in powerdns, powernds-admin but in powerdns-slave > Domains, there is nothing. Is this normal? I would like it to replicate?

[pschiffe]

Try to see if there's an error in the master or slave logs. Do you have set supermaster correctly? https://github.com/pschiffe/docker-pdns/blob/master/docker-compose.yml#L88

[SowAbdoul]

I respected what you did! I use swarm cluster :

version: '3.8'

services:

  pdns-recursor:
    image: pschiffe/pdns-recursor:${RECURSOR_TAG:-latest}
    networks:
      - pdns
    volumes:
      - /etc/localtime:/etc/localtime:ro

  mariadb:
    image: mariadb:10.10
    networks:
      pdns:
        aliases:
          - db
          - mysql
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /mnt/dns/mariadb:/var/lib/mysql:z
    environment:
      - MYSQL_ROOT_PASSWORD=my-secret-pw
    healthcheck:
      test: ['CMD', 'mysqladmin', 'ping', '-h', 'localhost', '-pmy-secret-pw']
      timeout: 10s
      retries: 5

  phpmyadmin:
    image: phpmyadmin:5
    networks:
      - pdns
    ports:
      - '8988:80'
    volumes:
      - /etc/localtime:/etc/localtime:ro

  pdns-master:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns1.devops.com
    networks:
      pdns:
        ipv4_address: 192.168.100.185
        aliases:
          - pdns
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_master=yes
      - PDNS_api=yes
      - PDNS_api_key=secret
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_default_ttl=1500
      - PDNS_allow_axfr_ips=192.168.100.186,192.168.100.187
      - PDNS_only_notify=192.168.100.186,192.168.100.187
    depends_on:
      - mariadb

  pdns-slave1:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns2.devops.com
    networks:
      pdns:
        ipv4_address: 192.168.100.186
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave1
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=192.168.100.185
      - SUPERMASTER_IPS=192.168.100.185
    depends_on:
      - mariadb
      - pdns-master

  pdns-slave2:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns3.devops.com
    networks:
      pdns:
        ipv4_address: 192.168.100.187
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave2
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=192.168.100.185
      - SUPERMASTER_IPS=192.168.100.185
    depends_on:
      - mariadb
      - pdns-master

  pdns-admin-uwsgi:
    image: pschiffe/pdns-admin-uwsgi
    networks:
      pdns:
        aliases:
          - pdns-admin-uwsgi
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_ADMIN_SQLA_DB_PASSWORD='my-secret-pw'
      - PDNS_VERSION=4.7
      - PDNS_API_KEY=secret
    depends_on:
      - mariadb
      - pdns-master

  pdns-admin-static:
    image: pschiffe/pdns-admin-static
    networks:
      - pdns
    ports:
      - '80:80'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    depends_on:
      - pdns-admin-uwsgi

networks:
  pdns:
    ipam:
      config:
        - subnet: 192.168.100.0/24

[pschiffe]

Thank you; could you also share logs from master and slave containers, from the start of the containers to after you create the zone? If there's an error in communication, it should be visible there.

[SowAbdoul]

Thank you for your help :

# pdns-master from the start
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Created local state directory '/var/run/pdns/'
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Listening on controlsocket in '/var/run/pdns/pdns.controlsocket'
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Guardian is launching an instance
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Loading '/usr/lib64/pdns/libgmysqlbackend.so'
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 This is a guarded instance of pdns
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 UDP server bound to 0.0.0.0:53
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 UDP server bound to [::]:53
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 TCP server bound to 0.0.0.0:53
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 TCP server bound to [::]:53
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 PowerDNS Authoritative Server 4.7.2 (C) 2001-2022 PowerDNS.COM BV
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Using 64-bits mode. Built using gcc 12.2.1 20220819 (Red Hat 12.2.1-2) on Nov  1 2022 00:00:00 by mockbuild@localhost.
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 [webserver] Listening for HTTP requests on 0.0.0.0:8081

dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Polled security status of version 4.7.2 at startup, no known issues reported: OK
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Creating backend connection for TCP
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Primary/secondary communicator launching
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 About to create 3 backend threads for UDP

dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Done launching threads, ready to distribute questions

# pdns-slave1 from the starting
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Created local state directory '/var/run/pdns/'
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Listening on controlsocket in '/var/run/pdns/pdns.controlsocket'
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Guardian is launching an instance
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Loading '/usr/lib64/pdns/libgmysqlbackend.so'
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 This is a guarded instance of pdns
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 UDP server bound to 0.0.0.0:53
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 UDP server bound to [::]:53
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 TCP server bound to 0.0.0.0:53
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 TCP server bound to [::]:53
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 PowerDNS Authoritative Server 4.7.2 (C) 2001-2022 PowerDNS.COM BV
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Using 64-bits mode. Built using gcc 12.2.1 20220819 (Red Hat 12.2.1-2) on Nov  1 2022 00:00:00 by mockbuild@localhost.
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 [webserver] Listening for HTTP requests on 0.0.0.0:8081
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Polled security status of version 4.7.2 at startup, no known issues reported: OK
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Creating backend connection for TCP
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Primary/secondary communicator launching
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 About to create 3 backend threads for UDP
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Done launching threads, ready to distribute questions

[SowAbdoul]

I think this is the mistake

# pdns-master from creating zone
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:30:30 Request to queue notification for domain 'new-domain.com' was processed, but no valid nameservers or ALSO-NOTIFYs found. Not notifying!

# pdns-slave1 from zone zone : nothing

[pschiffe]

After you create a new zone, you also need to add NS records with all ns servers. Also the domain type must be set as PRIMARY (not native):

Once you add the NS records, wait 1 or 2 minutes and check the master and slave logs, if it's still not working, post them here.

[SowAbdoul]

It's all the same, I have done as you said but my zone has not spread to the slaves and there are no new logs yet.

# pdns-master
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 15:09:31 Request to queue notification for domain 'new-domain.com' was processed, but no valid nameservers or ALSO-NOTIFYs found. Not notifying!

[SowAbdoul]

[pschiffe]

That's weird, I'm not sure here. Try to create a new random record in the new-domain.com. The master should try to notify the change (there may be 2 minutes delay), and some log should appear in the master container.

[SowAbdoul]

dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 16:35:14 Request to queue notification for domain 'new-domain.com' was processed, but no valid nameservers or ALSO-NOTIFYs found. Not notifying!

[pschiffe]

Can you verify connectivity between the containers? ie exec to ns1, and try curl -v ns2.devops.com:53 (this is stupid, but curl is installed and it will connect to that port if it's possible).

Also you can try to provide the master env var PDNS_also_notify=192.168.100.186,192.168.100.187 to force the notification to these IPs (if the NS records don't work).

[SowAbdoul]

I added this line PDNS_also_notify=192.168.100.186,192.168.100.187 The database restarts several times. Then it recovers on another node of my cluster.

# pdns-master
dns-server_pdns-master.1.rtvrzplmy1jl@srv-manager-pdns-3    | Apr 20 07:59:35 Notification for devops.ci to 192.168.100.186:53 failed after retries
dns-server_pdns-master.1.rtvrzplmy1jl@srv-manager-pdns-3    | Apr 20 07:59:35 Notification for devops.ci to 192.168.100.187:53 failed after retries
dns-server_pdns-master.1.rtvrzplmy1jl@srv-manager-pdns-3    | Apr 20 08:00:28 gmysql Connection failed: Unable to connect to database: ERROR 2005 (HY000): Unknown server host 'mysql' (-2)
dns-server_pdns-master.1.rtvrzplmy1jl@srv-manager-pdns-3    | Apr 20 08:00:28 Caught an exception instantiating a backend: Unable to launch gmysql connection: Unable to connect to database: ERROR 2005 (HY000): Unknown server host 'mysql' (-2)
dns-server_pdns-master.1.rtvrzplmy1jl@srv-manager-pdns-3    | Apr 20 08:00:28 Cleaning up
dns-server_pdns-master.1.rtvrzplmy1jl@srv-manager-pdns-3    | Apr 20 08:00:28 PDNSException while updating zone cache: Unable to launch gmysql connection: Unable to connect to database: ERROR 2005 (HY000): Unknown server host 'mysql' (-2)

# pdns-slave
dns-server_pdns-slave1.1.so7bvttrag7g@srv-manager-pdns-2    | Apr 20 08:04:49 gmysql Connection failed: Unable to connect to database: ERROR 2005 (HY000): Unknown server host 'mysql' (-2)
dns-server_pdns-slave1.1.so7bvttrag7g@srv-manager-pdns-2    | Apr 20 08:04:49 Caught an exception instantiating a backend: Unable to launch gmysql connection: Unable to connect to database: ERROR 2005 (HY000): Unknown server host 'mysql' (-2)
dns-server_pdns-slave1.1.so7bvttrag7g@srv-manager-pdns-2    | Apr 20 08:04:49 Cleaning up
dns-server_pdns-slave1.1.so7bvttrag7g@srv-manager-pdns-2    | Apr 20 08:04:49 PDNSException while filling the zone cache: Unable to launch gmysql connection: Unable to connect to database: ERROR 2005 (HY000): Unknown server host 'mysql' (-2)
dns-server_pdns-slave1.1.so7bvttrag7g@srv-manager-pdns-2    | Apr 20 08:04:50 Our pdns instance exited with code 1, respawning
dns-server_pdns-slave1.1.so7bvttrag7g@srv-manager-pdns-2    | Apr 20 08:04:51 Guardian is launching an instance

[SowAbdoul]

The stdout output of the docker service ls command.

ID             NAME                           MODE         REPLICAS   IMAGE                               PORTS
pwceb33wior4   dns-server_mariadb             replicated   0/1        mariadb:10.10                       
9nclkfa32tk0   dns-server_pdns-admin-static   replicated   1/1        pschiffe/pdns-admin-static:latest   *:80->80/tcp
qb0nyyx72jje   dns-server_pdns-admin-uwsgi    replicated   1/1        pschiffe/pdns-admin-uwsgi:latest    
fxjrpso0j3v3   dns-server_pdns-master         replicated   1/1        pschiffe/pdns-mysql:latest          
l2lw4mx1ogqe   dns-server_pdns-recursor       replicated   1/1        pschiffe/pdns-recursor:latest       
u3ktxk6wxew3   dns-server_pdns-slave1         replicated   1/1        pschiffe/pdns-mysql:latest          
xjukho1ofrp7   dns-server_pdns-slave2         replicated   1/1        pschiffe/pdns-mysql:latest          
tyhffm1ei653   dns-server_phpmyadmin          replicated   1/1        phpmyadmin:5                        *:8988->80/tcp
5y6kqp0en4n0   portainer                      replicated   1/1        portainer/portainer-ce:latest       *:9000->9000/tcp

[pschiffe]

Notification for devops.ci to 192.168.100.186:53 failed after retries seems like the containers cannot reach each other. I'm sorry, I'm not that familiar with docker swarm. You can try that curl from the master container (or install dig - dnf install bind-utils) and try to reach the slaves. If that doesn't work, there's something either with the docker swarm network configuration, or the docker-compose file must be adjusted for docker swarm, I'm not sure.

[SowAbdoul]

You were right, it is the network that is the problem.

# pdns-master
curl -v ns2.devops.com:53
* Trying 192.168.100.186:53...
* connect to 192.168.100.186 port 53 failed : No route to host
* Failed to connect to ns2.devops.com port 53 after 3068 ms : No route to host
* Closing of the connection 0
curl: (7) Connection to ns2.devops.com port 53 failed after 3068 ms: No route to host

# pdns-slave
curl -v ns1.devops.com:53
* Trying 192.168.100.185:53...
* Connection to 192.168.100.185 port 53 failed: No route to host
* Failed to connect to ns1.devops.com port 53 after 3064 ms: No route to host
* Connection closure 0
curl: (7) Connection to ns1.devops.com port 53 failed after 3064 ms: No route to host

[SowAbdoul]

# pdns-master
cat /etc/hosts 
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.100.185 ns1.devops.com
192.168.100.186 ns2.devops.com
192.168.100.187 ns3.devops.com
192.168.100.24 ns2.devops.com ns2

# pdn-slave
cat /etc/hosts 
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.100.185 ns1.devops.com
192.168.100.186 ns2.devops.com
192.168.100.187 ns3.devops.com
192.168.100.21 ns1.devops.com ns1

I actually notice that my services are not getting the ip I assigned in the YAML, there is 192.168.100.24 ns2.devops.com ns2 and 192.168.100.21 ns1.devops.com ns1 And when I redo the curl but with these ip, it works

# pdns-master
curl -v 192.168.100.24:53
* Trying 192.168.100.24:53...
* Connected to 192.168.100.24 (192.168.100.24) port 53 (#0)
> GET / HTTP/1.1
> Host: 192.168.100.24:53
> User-Agent: curl/7.85.0
> Accept: */*
> 
* Empty server response
* Closing the connection 0
curl : (52) Empty server response

# pdns-slave
curl -v 192.168.100.21:53
* Trying 192.168.100.21:53...
* Connected to 192.168.100.21 (192.168.100.21) port 53 (#0)
> GET / HTTP/1.1
> Host: 192.168.100.21:53
> User-Agent: curl/7.85.0
> Accept: */*
> 
* Empty server response
* Closing the connection 0
curl: (52) Empty server response

[pschiffe]

Now I see that it's not possible to use static IPs for containers in docker swarm: https://github.com/moby/moby/issues/24170

[pschiffe]

At least supermaster feature requires static IP, and axfr transfers probably as well; I don't see right now how this could be work-arounded for swarm :/

[SowAbdoul]

How are you? Thanks for your help, I've been looking for how to define static ip's to services with swarm, but I couldn't find anything.

I tried to modify extra_hosts by removing the ip's and I kept the hostnames:

    extra_hosts:
      - 'ns1.devops.com'
      - 'ns2.devops.com'
      - 'ns3.devops.com'

I can do curl inside the containers. My problem is that I have to fill in the fields PDNS_also_notify PDNS_allow_axfr_ips SUPERMASTER_IPS.

Is it possible to proceed in this way:

  pdns-master:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns1.devops.com
    networks:
      pdns:
        ipv4_address: 192.168.100.185
        aliases:
          - pdns
    extra_hosts:
      - 'ns1.devops.com'
      - 'ns2.devops.com'
      - 'ns3.devops.com'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_master=yes
      - PDNS_api=yes
      - PDNS_api_key=secret
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_default_ttl=1500
      - PDNS_also_notify=ns2.devops.com,ns3.devops.com
      - PDNS_allow_axfr_ips=ns2.devops.com,ns3.devops.com
      - PDNS_only_notify=ns2.devops.com,ns3.devops.com
    depends_on:
      - mariadb

  pdns-slave1:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns2.devops.com
    networks:
      pdns:
        ipv4_address: 192.168.100.186
    extra_hosts:
      - 'ns1.devops.com'
      - 'ns2.devops.com'
      - 'ns3.devops.com'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave1
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=ns1.devops.com
      - SUPERMASTER_IPS=ns1.devops.com
    depends_on:
      - mariadb
      - pdns-master

  pdns-slave2:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns3.devops.com
    networks:
      pdns:
        ipv4_address: 192.168.100.187
    extra_hosts:
      - 'ns1.devops.com'
      - 'ns2.devops.com'
      - 'ns3.devops.com'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave2
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=ns1.devops.com
      - SUPERMASTER_IPS=ns1.devops.com
    depends_on:
      - mariadb
      - pdns-master

All services start except pdns-admin-static

dns_pdns-admin-static.1.goqydtm2elwt@srv-manager-pdns-1 | nginx : [emerg] host not found in upstream "pdns-admin-uwsgi" in /etc/nginx/nginx.conf:48
dns_pdns-admin-static.1.obr031430xkr@srv-manager-pdns-1 | nginx : [emerg] host not found in upstream "pdns-admin-uwsgi" in /etc/nginx/nginx.conf:48
dns_pdns-admin-static.1.jl85lkhrbdo2@srv-manager-pdns-1 | nginx : [emerg] host not found in upstream "pdns-admin-uwsgi" in /etc/nginx/nginx.conf:48
dns_pdns-admin-static.1.v0j7dggdmyw4@srv-manager-pdns-3 | nginx : [emerg] host not found in upstream "pdns-admin-uwsgi" in /etc/nginx/nginx.conf:48
dns_pdns-admin-static.1.yk8rrmofmfr4@srv-manager-pdns-1 | nginx : [emerg] host not found in upstream "pdns-admin-uwsgi" in /etc/nginx/nginx.conf:48

[pschiffe]

hi @asow25, I'm good, thank you. The error from static container means that pdns-admin-uwsgi doesn't resolve to the IP of uwsgi container. In the original docker-compose, this works thanks to the alias: https://github.com/pschiffe/docker-pdns/blob/master/docker-compose.yml#L98

Reg. the options, the PDNS_also_notify shouldn't be needed, that was a try to force the updates to slaves, but the original issue was the connection problems. PDNS_allow_axfr_ips must be IPs or subnets, but this can be set to the whole subnet. It's a security feature to not broadcast transfers to the internet. So PDNS_allow_axfr_ips=192.168.100.0/24 should work in your case. PDNS_only_notify is also a security feature, and can be set the same way as allow axfr ips: PDNS_only_notify=192.168.100.0/24 The same applies to PDNS_allow_notify_from as well: PDNS_allow_notify_from=192.168.100.0/24 Documentation to these options is here: https://doc.powerdns.com/authoritative/settings.html

Problem is with the SUPERMASTER IPS, there must be an IP: https://doc.powerdns.com/authoritative/modes-of-operation.html#autoprimary-automatic-provisioning-of-secondaries

The slave replication can work without SUPERMASTER feature, but you have to provision the domain on slaves manually (with NS records). pdnsutil cli command is available in the containers for this: https://doc.powerdns.com/authoritative/manpages/pdnsutil.1.html#zone-manipulation-commands https://makarainen.net/PowerDNS-pdnsutil-cheat-sheet You could also create the mysql records directly in DB too. This could work if you don't have too many zones (domains) I guess.

The third option can be to don't use pdns replication, but use mysql replication instead. That's a supported mode of operation: https://doc.powerdns.com/authoritative/modes-of-operation.html#native-replication Also, if you are using only the one mysql server, all pdns masters and slaves can share the single mysql database, and if the zone type of your zones will be Native, I think you are good to go as well.

Otherwise I'm not sure, as the static IPs are not supported for docker swarm: https://github.com/moby/moby/issues/24170 and we are dealing with DNS here :sweat_smile:

[SowAbdoul]

I managed to set static ip to pdns-master and pdns-slave1 and pdns-slave2.

# create mcvlan network on each node pdns-master, pdns-slave1 and pdns-slave2

docker network create --config-only --subnet 192.168.100.0/24 --gateway 192.168.100.1 --ip-range 192.168.100.185/32 pdns-config

docker network create --config-only --subnet 192.168.100.0/24 --gateway 192.168.100.1 --ip-range 192.168.100.186/32 pdns-config

docker network create --config-only --subnet 192.168.100.0/24 --gateway 192.168.100.1 --ip-range 192.168.100.187/32 pdns-config

docker network create -d macvlan --scope swarm --config-from pdns-config --attachable pdns

and a second network for the rest of the services

docker network create -d overlay --subnet 192.168.100.0/24 --gateway 192.168.100.2 --ip-range 192.168.100.0/24 --attachable pdns_net

At first, I had errors that it was the same gateway so I changed it to 192.168.100.2

When I run the stack, it's Ok but when I connect to php-admin the databases of pdns-master, pdns-slave1 and pdns-slave2 were not created, so I looked at the logs of the master and :

dns-server_pdns-master.1.ym0l0yov8kkw@ans1 | MySQL is unavailable - sleeping
dns-server_pdns-master.1.ym0l0yov8kkw@ans1 | ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (-3)
dns-server_pdns-master.1.ym0l0yov8kkw@ans1 | MySQL is unavailable - sleeping
dns-server_pdns-master.1.ym0l0yov8kkw@ans1 | ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (-3)

this means my two networks are not talking to each other but overlay is the swarm network that allows multi-data communication. Anyway, I added an alias

  pdns-master:
  ...
    networks:
      pdns:
        aliases:
          - pdns

  pdns-slave1:
  ...
    networks:
      pdns:
        aliases:
          - pdns

  pdns-slave2:
  ...
    networks:
      pdns:
        aliases:
          - pdns

networks:
  pdns:
    external: true
  pdns_net:
    external: true      

I still have the same error, my two networks do not communicate so the databases of these three services were not created when I deploy the stack. Can you help me please ?

[pschiffe]

I think 2 networks should be fine, but use different subnets, as this will most likely clash. You can attach container to multiple networks. Also alias must be different for every container.

  pdns-master:
  ...
    networks:
      pdns:
        aliases:
          - pdns-master
      pdns_net:

  pdns-slave1:
  ...
    networks:
      pdns:
        aliases:
          - pdns-slave1
      pdns_net:

  pdns-slave2:
  ...
    networks:
      pdns:
        aliases:
          - pdns-slave2
      pdns_net:

networks:
  pdns:
    external: true
  pdns_net:
    external: true    

[SowAbdoul]

It's all the same, Please take a look!

version: '3.8'

services:

  pdns-recursor:
    image: pschiffe/pdns-recursor:${RECURSOR_TAG:-latest}
    networks:
      - pdns_net
    volumes:
      - /etc/localtime:/etc/localtime:ro

  mariadb:
    image: mariadb:10.10
    networks:
      pdns_net:
        aliases:
          - db
          - mysql
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - mariadb:/var/lib/mysql:z
    environment:
      - MYSQL_ROOT_PASSWORD=my-secret-pw
    healthcheck:
      test: ['CMD', 'mysqladmin', 'ping', '-h', 'localhost', '-pmy-secret-pw']
      timeout: 10s
      retries: 5

  phpmyadmin:
    image: phpmyadmin:5
    networks:
      - pdns_net
    ports:
      - '8988:80'
    volumes:
      - /etc/localtime:/etc/localtime:ro

  pdns-master:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns1.devops.com
    networks:
      pdns:
        aliases:
          - pdns-master
      pdns_net:
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_master=yes
      - PDNS_api=yes
      - PDNS_api_key=secret
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_default_ttl=1500
      - PDNS_also_notify=192.168.100.186,192.168.100.187
      - PDNS_allow_axfr_ips=192.168.100.186,192.168.100.187
      - PDNS_only_notify=192.168.100.186,192.168.100.187
    depends_on:
      - mariadb
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == 94ymd6l9cg9njxwqqq71i4rvn
      restart_policy:
        condition: on-failure

  pdns-slave1:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns2.devops.com
    networks:
      pdns:
        aliases:
          - pdns-slave1
      pdns_net:
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave1
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=192.168.100.185
      - SUPERMASTER_IPS=192.168.100.185
    depends_on:
      - mariadb
      - pdns-master
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == re89fo9grvr9zx6pi3prxdypp
      restart_policy:
        condition: on-failure

  pdns-slave2:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns3.devops.com
    networks:
      pdns:
        aliases:
          - pdns-slave2
      pdns_net:
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave2
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=192.168.100.185
      - SUPERMASTER_IPS=192.168.100.185
    depends_on:
      - mariadb
      - pdns-master
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == 49v6r3dk26i1oteub2vf9buy8
      restart_policy:
        condition: on-failure

  pdns-admin-uwsgi:
    image: pschiffe/pdns-admin-uwsgi
    networks:
      pdns_net:
        aliases:
          - pdns-admin-uwsgi
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_ADMIN_SQLA_DB_PASSWORD='my-secret-pw'
      - PDNS_VERSION=4.7
      - PDNS_API_KEY=secret
    depends_on:
      - mariadb
      - pdns-master

  pdns-admin-static:
    image: pschiffe/pdns-admin-static
    networks:
      - pdns_net
    ports:
      - '80:80'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    depends_on:
      - pdns-admin-uwsgi

networks:
  pdns:
    external: true
  pdns_net:
    external: true

volumes:
  mariadb:

[pschiffe]

hi @asow25, I'm sorry I cannot tell what could be the problem here. You can try to exec to the containers and try some network debugging to check if the networks are routing correctly. Alternatively I see that you have node.id constraints for all pdns master and slaves... If they are running on different nodes, maybe you could just expose ports 53 tcp and udp to host and use connection between the nodes directly? Like outside of the docker virtual network, and use IPs of the hosts..

[SowAbdoul]

Hi @pschiffe all my services are up and running, my 3 powerdns have static ip's and a second ip that allows them to communicate with other services. I checked the container logs, everything seems to be working fine except pdns-admin-uwsgi.

|pdns-server_pdns_uwsgi`` has been updated with the addition of a new module for managing access to the Internet and a new module for managing access to the Internet and a new module for managing access to the Internet and a new module for managing access to the Internet.
dns-server_pdns-admin-uwsgi.1 .o7m9lzjqqb43@srv-manager-pdns-2 | [pid: 133|app: 0|req: 43/43] 10.0.0.2 () {46 vars in 1127 bytes} [Mon Apr 24 07:13:05 2023] GET /user/image?username=pdns => generated 0 bytes in 15 msec (HTTP/1.1 304) 4 headers in 284 bytes (0 switches on core 0)
dns-server_pdns-admin-uwsgi.1 .o7m9lzjqqb43@srv-manager-pdns-2 | [pid: 133|app: 0|req: 44/44] 10.0.0.2 () {46 vars in 1081 bytes} [Mon Apr 24 07:14:30 2023] GET /ping?_=1682320384994 => generated 2 bytes in 48 msec (HTTP/1.1 200) 3 headers in 193 bytes (1 switches on core 0)
dns-server_pdns-admin-uwsgi.1.o7m9lzjqqb43@srv-manager-pdns-2 | [pid: 133|app: 0|req: 45/45] 10.0.0.2 () {46 vars in 1140 bytes} [Mon Apr 24 07:14:32 2023] GET /domain/add => generated 31581 bytes in 98 msec (HTTP/1.1 200) 5 headers in 378 bytes (1 switches on core 0)
dns-server_pdns-admin-uwsgi.1 .o7m9lzjqqb43@srv-manager-pdns-2 | [pid: 133|app: 0|req: 46/46] 10.0.0.2 () {44 vars in 1072 bytes} [Mon Apr 24 07:14:32 2023] GET /user/image?username=pdns => generated 2718 bytes in 14 msec via sendfile() (HTTP/1.1 200) 7 headers in 367 bytes (0 switches on core 0)
dns-server_pdns-admin-uwsgi.1.o7m9lzjqqb43@srv-manager-pdns-2 | [2023-04-24 07:14:49,498] [domain.py:281] ERROR - Cannot add zone asow.ci. HTTPConnectionPool(host='pdns', port=8081) : Max retries exceeded with url : /api/v1/servers/localhost/zones (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcf14be8f90> : Failed to establish a new connection : [Errno -2] Name or service not known')

When I try to create a zone, I get an error : HTTP 404 Error

[pschiffe]

If the pdns-master is not available under pdns alias anymore, you can configure pdns-admin-uwsgi with PDNS_API_URL="http://pdns-master:8081/" or PDNS_API_URL="http://ns1.devops.com:8081/" env var (whichever will work). Also IP should work there too.

[SowAbdoul]

dns-server_pdns-admin-uwsgi.1.0l5mggn5odp5@srv-manager-pdns-2    | [2023-04-24 16:37:50,832] [server.py:39] ERROR - Can not get server configuration. DETAIL: Invalid URL '/api/v1/servers/localhost/config': No scheme supplied. Perhaps you meant https:///api/v1/servers/localhost/config?
dns-server_pdns-admin-uwsgi.1.0l5mggn5odp5@srv-manager-pdns-2    | [pid: 55|app: 0|req: 16/16] 10.0.0.2 () {46 vars in 1164 bytes} [Mon Apr 24 16:37:50 2023] GET /admin/server/configuration => generated 19278 bytes in 109 msecs (HTTP/1.1 200) 3 headers in 197 bytes (1 switches on core 0)
dns-server_pdns-admin-uwsgi.1.0l5mggn5odp5@srv-manager-pdns-2    | [pid: 55|app: 0|req: 17/17] 10.0.0.2 () {46 vars in 1135 bytes} [Mon Apr 24 16:37:51 2023] GET /user/image?username=pdns => generated 0 bytes in 58 msecs (HTTP/1.1 304) 4 headers in 284 bytes (0 switches on core 0)

[pschiffe]

The value must include full URL, with http://, it's an address for pdns API: PDNS_API_URL="http://ns1.devops.com:8081/" You can check this value also in pdns admin web interface, in settings, to see if it's set correctly.

[SowAbdoul]

It's the same, honestly, I don't understand, when I fix a problem, another one is created. Before everything worked, but the ip were not fixed so impossible that a zone is propagated to other servers powernds since swarm manages itself the allocation of ip addresses. I found this method:

# node master
docker network create --config-only --subnet 192.168.100.0/24 --gateway 192.168.100.1 --ip-range 192.168.100.185/32 pdns-config

# node slave1
docker network create --config-only --subnet 192.168.100.0/24 --gateway 192.168.100.1 --ip-range 192.168.100.186/32 pdns-config

# node slave2
docker network create --config-only --subnet 192.168.100.0/24 --gateway 192.168.100.1 --ip-range 192.168.100.187/32 pdns-config

# node master
docker network create -d macvlan --scope swarm --config-from pdns-config --attachable pdns

Now the ip's are fixed and I have added another common network for all the services to communicate.

docker network create -d overlay --attachable pdns_net

Then, database problem, it's fixed, now it's the turn of pdns-admin-uwsgi please take a last look at my YAML and give me your opinion.

version: '3.8'

services:

  pdns-recursor:
    image: pschiffe/pdns-recursor:${RECURSOR_TAG:-latest}
    networks:
      - pdns_net
    volumes:
      - /etc/localtime:/etc/localtime:ro

  mariadb:
    image: mariadb:10.10
    networks:
      pdns_net:
        aliases:
          - db
          - mysql
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /mnt/gfs/mariadb:/var/lib/mysql:z
    environment:
      - MYSQL_ROOT_PASSWORD=my-secret-pw
    healthcheck:
      test: ['CMD', 'mysqladmin', 'ping', '-h', 'localhost', '-pmy-secret-pw']
      timeout: 10s
      retries: 5

  phpmyadmin:
    image: phpmyadmin:5
    networks:
      - pdns_net
    ports:
      - '8988:80'
    volumes:
      - /etc/localtime:/etc/localtime:ro

  pdns-master:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns1.devops.com
    networks:
      pdns_net:
      pdns:
        aliases:
          - pdns-master
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_master=yes
      - PDNS_api=yes
      - PDNS_api_key=secret
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_default_ttl=1500
      - PDNS_also_notify=192.168.100.186,192.168.100.187
      - PDNS_allow_axfr_ips=192.168.100.186,192.168.100.187
      - PDNS_only_notify=192.168.100.186,192.168.100.187
    depends_on:
      - mariadb
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == 2x07fshor56uthrnj20ew7y3m
      restart_policy:
        condition: on-failure
    ports:
      - '53:53'
      - '53:53/udp'
      - '8081:8081'

  pdns-slave1:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns2.devops.com
    networks:
      pdns_net:
      pdns:
        aliases:
          - pdns-slave1
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave1
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=192.168.100.185
      - SUPERMASTER_IPS=192.168.100.185
    depends_on:
      - mariadb
      - pdns-master
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == sp4oa4ev5kuj0a6efqx1by0w9
      restart_policy:
        condition: on-failure

  pdns-slave2:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns3.devops.com
    networks:
      pdns_net:
      pdns:
        aliases:
          - pdns-slave2
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave2
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=192.168.100.185
      - SUPERMASTER_IPS=192.168.100.185
    depends_on:
      - mariadb
      - pdns-master
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == lvjwq59t0gd9hs13b4ytol7uw
      restart_policy:
        condition: on-failure

  pdns-admin-uwsgi:
    image: pschiffe/pdns-admin-uwsgi
    networks:
      pdns_net:
        aliases:
          - pdns-admin-uwsgi
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_ADMIN_SQLA_DB_PASSWORD='my-secret-pw'
      - PDNS_API_URL="http://pdns-master:8081/"
      - PDNS_VERSION=4.7
      - PDNS_API_KEY=secret
    depends_on:
      - mariadb
      - pdns-master

  pdns-admin-static:
    image: pschiffe/pdns-admin-static
    networks:
      - pdns_net
    ports:
      - '80:80'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    depends_on:
      - pdns-admin-uwsgi

networks:
  pdns:
    external: true
  pdns_net:
    external: true

[pschiffe]

I'm so sorry to hear that :( I do see one issue in the yaml - pdns-admin-uwsgi container is connected to the pdns_net, but the pdns-master container has alias on the pdns network. Try to move the alias to pdns_net or add it to both (not sure if that will work):

  pdns-master:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns1.devops.com
    networks:
      pdns_net:
        aliases:
          - pdns-master
      pdns:
        aliases:
          - pdns-master

It's true that docker swarm doesn't have very good reputation, maybe you could take a look at some simple kubernetes distro, such as https://k3s.io/

[SowAbdoul]

 | *** Starting uWSGI 2.0.21 (64bit) on [Mon Apr 24 19:49:19 2023] ***
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | compiled with version: 13.0.1 20230117 (Red Hat 13.0.1-0) on 24 January 2023 00:00:00
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | os: Linux-5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | nodename: bf2664ff8867
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | machine: x86_64
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | clock source: unix
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | pcre jit disabled
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | detected number of CPU cores: 2
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | current working directory: /opt/powerdns-admin
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | writing pidfile to /run/uwsgi/uwsgi.pid
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | detected binary path: /usr/sbin/uwsgi
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | setgid() to 998
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | setuid() to 998
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | chdir() to /opt/powerdns-admin
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | your memory page size is 4096 bytes
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | detected max file descriptor number: 1048576
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | lock engine: pthread robust mutexes
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | thunder lock: disabled (you can enable it with --thunder-lock)
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | uwsgi socket 0 bound to TCP address 0.0.0.0:9494 fd 3
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | Python version: 3.11.2 (main, Feb  8 2023, 00:00:00) [GCC 13.0.1 20230208 (Red Hat 13.0.1-0)]
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | Python main interpreter initialized at 0x7f1b8f4aabf8
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | python threads support enabled
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | your server socket listen backlog is limited to 100 connections
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | your mercy for graceful operations on workers is 60 seconds
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | mapped 219568 bytes (214 KB) for 1 cores
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | *** Operational MODE: single process ***
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | added /opt/powerdns-admin/ to pythonpath.
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | mounting run.py on /
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | WSGI app 0 (mountpoint='/') ready in 1 seconds on interpreter 0x7f1b8f4aabf8 pid: 1 (default app)
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | *** uWSGI is running in multiple interpreter mode ***
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | spawned uWSGI master process (pid: 1)
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | spawned uWSGI worker 1 (pid: 148, cores: 1)

The service started, then when I try to click on the Server Configuration menu it says No data available in table and here are the server errors at the same time.

dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | [2023-04-24 19:51:42,848] [server.py:60] ERROR - Can not get server statistics. DETAIL: Invalid URL '/api/v1/servers/localhost/statistics': No scheme supplied. Perhaps you meant https:///api/v1/servers/localhost/statistics?
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | [pid: 148|app: 0|req: 14/14] 10.0.0.2 () {46 vars in 1176 bytes} [Mon Apr 24 19:51:42 2023] GET /admin/server/statistics => generated 19256 bytes in 234 msecs (HTTP/1.1 200) 3 headers in 197 bytes (1 switches on core 0)
dns-server_pdns-admin-uwsgi.1.d8zgx1hw1dzy@srv-manager-pdns-2    | [pid: 148|app: 0|req: 15/15] 10.0.0.2 () {46 vars in 1150 bytes} [Mon Apr 24 19:51:43 2023] GET /user/image?username=pdns => generated 0 bytes in 145 msecs (HTTP/1.1 304) 4 headers in 284 bytes (0 switches on core 0)

[pschiffe]

The Invalid URL error is weird, could you check the pdns admin mysql database, table setting. There should be the pdns_api_url configuration, please check it's value.

[SowAbdoul]

powerdns > settings

[pschiffe]

Hm, the URL should be there without the quotes, remove them and try again.

[SowAbdoul]

I noticed in your YAML, you have not defined PDNS_API_URL instead, there is the alias of the powerdns-master network which is pdns when I checked powerdns > settings I see it says pdns_api_url http://pdns:8081/ it automatically retrieves the network alias of powerdns-master network This way it works when I do the docker-compose up -d so I tried to do the same in my Swarm environment: Once my stack was deployed, I tried to bring up the Server Configuration section to see if it picked up the environment variables I provided and this is the error I get:

dns-server_pdns-admin-uwsgi.1.jdifwv2pd353@srv-manager-pdns-1 | [2023-04-25 07:06:57,037] [server.py:39] ERROR - Can not get server configuration. DETAIL: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
dns-server_pdns-admin-uwsgi.1.jdifwv2pd353@srv-manager-pdns-1 | [pid: 145|app: 0|req: 12/12] 10.0.0.2 () {46 vars in 1178 bytes} [Tue Apr 25 07:06:57 2023] GET /admin/server/configuration => generated 19278 bytes in 104 msec (HTTP/1.1 200) 3 headers in 197 bytes (1 switches on core 0)
dns-server_pdns-admin-uwsgi.1 .jdifwv2pd353@srv-manager-pdns-1 | [pid: 145|app: 0|req: 13/13] 10.0.0.2 () {46 vars in 1149 bytes} [Tue Apr 25 07:06:57 2023] GET /user/image?username=pdns => generated 0 bytes in 14 msec (HTTP/1.1 304) 4 headers in 284 bytes (0 switches on core 0)
dns-server_pdns-admin-uwsgi.1 .jdifwv2pd353@srv-manager-pdns-1 | [pid: 145|app: 0|req: 14/14] 10.0.0.2 () {46 vars in 1103 bytes} [Tue Apr 25 07:07:58 2023] GET /ping?_=1682406417168 => generated 2 bytes in 60 msec (HTTP/1.1 200) 3 headers in 193 bytes (1 switches on core 0)

I noticed that the error was different from what I had yesterday.

[SowAbdoul]

  pdns-master:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns1.devops.com
    networks:
      pdns_net:
        aliases:
          - pdns
      pdns:
        aliases:
          - pdns
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_master=yes
      - PDNS_api=yes
      - PDNS_api_key=secret
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_default_ttl=1500
      - PDNS_also_notify=192.168.100.186,192.168.100.187
      - PDNS_allow_axfr_ips=192.168.100.186,192.168.100.187
      - PDNS_only_notify=192.168.100.186,192.168.100.187
    depends_on:
      - mariadb
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == 2x07fshor56uthrnj20ew7y3m
      restart_policy:
        condition: on-failure

  pdns-admin-uwsgi:
    image: pschiffe/pdns-admin-uwsgi
    networks:
      pdns_net:
        aliases:
          - pdns-admin-uwsgi
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_ADMIN_SQLA_DB_PASSWORD='my-secret-pw'
      - PDNS_VERSION=4.7
      - PDNS_API_KEY=secret
    depends_on:
      - mariadb
      - pdns-master

[pschiffe]

Yep, this should work, configuration wise. So now only the network issue persists :D Try to exec to pdns-admin-uwsgi and run curl -v http://pdns:8081/. The network error should be visible there.

[SowAbdoul]

curl -v http://pdns:8081/
*   Trying 10.0.3.19:8081...
* Connected to pdns (10.0.3.19) port 8081 (#0)
> GET / HTTP/1.1
> Host: pdns:8081
> User-Agent: curl/7.87.0
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

[SowAbdoul]

The Server Configuration tab is empty, it's like powerdns-admin doesn't get the information from powerdns-master. When I want to create a zone or perform an action on powrdns-admin, I get this error:

[domain.py:281] ERROR - Cannot add zone asow.ci. ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
dns-server_pdns-admin-uwsgi.1.jdifwv2pd353@srv-manager-pdns-1    | [pid: 145|app: 0|req: 30/30] 10.0.0.2 () {52 vars in 1314 bytes} [Tue Apr 25 07:31:13 2023] POST /domain/add => generated 18182 bytes in 149 msecs (HTTP/1.1 400) 3 headers in 206 bytes (1 switches on core 0)
dns-server_pdns-admin-uwsgi.1.jdifwv2pd353@srv-manager-pdns-1    | [pid: 145|app: 0|req: 31/31] 10.0.0.2 () {46 vars in 1147 bytes} [Tue Apr 25 07:31:13 2023] GET /user/image?username=pdns => generated 0 bytes in 14 msecs (HTTP/1.1 304) 4 headers in 284 bytes (0 switches on core 0)

[pschiffe]

So the connection works, try to set the API URL manually on the configuration page of pdns admin, so it reads http://pdns:8081/ or edit the mysql table directly.

[SowAbdoul]

It was already done

[pschiffe]

So the configuration seems OK, the connection seems to work as well.. Question, is this the correct IP of the pdns master in the pdns_net?

curl -v http://pdns:8081/
*   Trying 10.0.3.19:8081...
* Connected to pdns (10.0.3.19) port 8081 (#0)
...

[SowAbdoul]

[root@34af705b2d62 powerdns-admin]# curl -v http://pdns:8081/
*   Trying 10.0.3.19:8081...
* Connected to pdns (10.0.3.19) port 8081 (#0)
> GET / HTTP/1.1
> Host: pdns:8081
> User-Agent: curl/7.87.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
[root@34af705b2d62 powerdns-admin]# 

[pschiffe]

If the IP address 10.0.3.19 is correct, you need to update this value for pdns master: PDNS_webserver_allow_from=192.168.100.0/24 to PDNS_webserver_allow_from=10.0.3.0/24, or the subnet you used when creating the pdns_net.

[SowAbdoul]

Ok but I have another network between the 3 powerdns, it will not create any problem at the replication.

[pschiffe]

The webserver is only for the API, which is used only by the pdns admin. Slaves are using axfr and the notifies, which is different config.

[SowAbdoul]

It works, I can consult the information of powerdns-master. I'm inside the containers and I have displayed /etc/hosts here are the two ip of my two networks for each powerdns

# pdns-master
cat /etc/hosts

10.0.3.35 ns1.devops.com ns1
192.168.100.185 ns1.devops.com ns1

# pdns-slave1
cat /etc/hosts

192.168.100.186 ns2.devops.com ns2
10.0.3.37 ns2.devops.com ns2

# pdns-slave2
cat /etc/hosts

192.168.100.187 ns3.devops.com ns3
10.0.3.39 ns3.devops.com ns3

Also, in /etc/hosts on all 3, we have the variable extra_hosts

192.168.100.185 ns1.devops.com
192.168.100.186 ns2.devops.com
192.168.100.187 ns3.devops.com

I get errors when I run the command curl -v ns2.devops.com

[root@ns1 /]# curl -v ns2.devops.com:53
* Trying 192.168.100.186:53...
* Connection to port 53 of 192.168.100.186 failed: No route to host
* Failed to connect to port 53 of ns2.devops.com after 3068 ms: Unable to connect to server
* Connection closure 0
curl: (7) Failed to connect to ns2.devops.com port 53 after 3068 ms: Unable to connect to server
[root@ns1 /]# 

[pschiffe]

Great! The error No route to host means it's not able to route the traffic inside of pdns network, which I'm really not sure how to help with, as I've never worked with the macvlan network.

[SowAbdoul]

Thank you very much, I will take a moment and think more calmly, if I find the solution, I will share with you!

[SowAbdoul]

Excuse me, before there was Dashboard Info with graphs that can be easily interpreted. Now, in the Dashboard section, we have nothing but the list of the areas we create.