Tpot.service continually restarting (docker-compose non-overlapping address pool)

[ProducerMatt]

Issues

While the web admin page at 64294 is accessible and so is SSH at 64295, the Kibana dashboard at 64297 is not. Upon closer inspection I realized that tpot.service is constantly failing and restarting. This happens on brand-new installs on VMs AND real hardware.

I've looked through logs and the one line that stands out to me and looks like it causes the cascade of failure:

Dec 06 00:03:07 deliciousrocketship docker-compose[1170519]: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

I'm initially inclined to think this is an issue with the docker-compose config, and therefore an issue with tpot -- but if that were so, why haven't I seen any Issues posted that match my problems? If it's not an issue with tpot, then it must be some kind of problem with my network -- but I have no idea what kind of problem it would be, or how to troubleshoot it. I'm stumped.

More details

In between restarts the service looks like this:

● tpot.service - tpot
     Loaded: loaded (/etc/systemd/system/tpot.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2019-12-06 01:06:47 UTC; 1s ago
    Process: 229989 ExecStartPre=/opt/tpot/bin/updateip.sh (code=exited, status=0/SUCCESS)
    Process: 230009 ExecStartPre=/bin/bash -c /opt/tpot/bin/clean.sh on (code=exited, status=0/SUCCESS)
    Process: 230054 ExecStartPre=/usr/bin/docker-compose -f /opt/tpot/etc/tpot.yml down -v (code=exited, status=0/SUCCESS)
    Process: 230469 ExecStartPre=/usr/bin/docker-compose -f /opt/tpot/etc/tpot.yml rm -v (code=exited, status=0/SUCCESS)
    Process: 230472 ExecStartPre=/bin/bash -c docker network rm $(docker network ls -q) (code=exited, status=1/FAILURE)
    Process: 230491 ExecStartPre=/bin/bash -c docker volume rm $(docker volume ls -q) (code=exited, status=1/FAILURE)
    Process: 230510 ExecStartPre=/bin/bash -c docker rm -v $(docker ps -aq) (code=exited, status=1/FAILURE)
    Process: 230529 ExecStartPre=/bin/bash -c docker rmi $(docker images | grep "<none>" | awk '{print $3}') (code=exited, status=1/FAILURE)
    Process: 230550 ExecStartPre=/bin/bash -c /sbin/ethtool --offload $(/sbin/ip address | grep "^2: " | awk '{ print $2 }' | tr -d [:punct:]) rx off tx off (code=exited, status=0/SUCCESS)
    Process: 230556 ExecStartPre=/bin/bash -c /sbin/ethtool -K $(/sbin/ip address | grep "^2: " | awk '{ print $2 }' | tr -d [:punct:]) gso off gro off (code=exited, status=0/SUCCESS)
    Process: 230562 ExecStartPre=/bin/bash -c /sbin/ip link set $(/sbin/ip address | grep "^2: " | awk '{ print $2 }' | tr -d [:punct:]) promisc on (code=exited, status=0/SUCCESS)
    Process: 230568 ExecStartPre=/opt/tpot/bin/rules.sh /opt/tpot/etc/tpot.yml set (code=exited, status=0/SUCCESS)
    Process: 230643 ExecStart=/usr/bin/docker-compose -f /opt/tpot/etc/tpot.yml up --no-color (code=exited, status=1/FAILURE)
    Process: 231425 ExecStopPost=/opt/tpot/bin/rules.sh /opt/tpot/etc/tpot.yml unset (code=exited, status=0/SUCCESS)
   Main PID: 230643 (code=exited, status=1/FAILURE)

Here's the loop it goes on:

Dec 06 00:02:53 deliciousrocketship systemd[1]: tpot.service: Scheduled restart job, restart counter is at 3488.
Dec 06 00:02:53 deliciousrocketship systemd[1]: Stopped tpot.
Dec 06 00:02:53 deliciousrocketship systemd[1]: Starting tpot...
Dec 06 00:02:53 deliciousrocketship updateip.sh[1169838]: Trying: dig +short myip.opendns.com @resolver1.opendns.com
Dec 06 00:02:53 deliciousrocketship updateip.sh[1169850]: [MAIN]
Dec 06 00:02:53 deliciousrocketship updateip.sh[1169850]: ip = [NOPE]
Dec 06 00:02:53 deliciousrocketship updateip.sh[1169851]: MY_EXTIP=[NOPE]
Dec 06 00:02:53 deliciousrocketship updateip.sh[1169851]: MY_INTIP=192.168.1.231
Dec 06 00:02:53 deliciousrocketship updateip.sh[1169851]: MY_HOSTNAME=deliciousrocketship
Dec 06 00:02:53 deliciousrocketship bash[1169854]: Cleaning up and preparing data folders.
Dec 06 00:02:54 deliciousrocketship docker-compose[1169935]: Removing network etc_adbhoney_local
Dec 06 00:02:54 deliciousrocketship docker-compose[1169935]: Removing network etc_conpot_local_IEC104
Dec 06 00:02:55 deliciousrocketship docker-compose[1169935]: Removing network etc_conpot_local_guardian_ast
Dec 06 00:02:55 deliciousrocketship docker-compose[1169935]: Removing network etc_conpot_local_ipmi
Dec 06 00:02:56 deliciousrocketship docker-compose[1169935]: Removing network etc_conpot_local_kamstrup_382
Dec 06 00:02:56 deliciousrocketship docker-compose[1169935]: Removing network etc_cowrie_local
Dec 06 00:02:56 deliciousrocketship docker-compose[1169935]: Removing network etc_heralding_local
Dec 06 00:02:57 deliciousrocketship docker-compose[1169935]: Removing network etc_honeypy_local
Dec 06 00:02:57 deliciousrocketship docker-compose[1169935]: Removing network etc_mailoney_local
Dec 06 00:02:58 deliciousrocketship docker-compose[1169935]: Removing network etc_medpot_local
Dec 06 00:02:58 deliciousrocketship docker-compose[1169935]: Removing network etc_rdpy_local
Dec 06 00:02:58 deliciousrocketship docker-compose[1169935]: Removing network etc_tanner_local
Dec 06 00:02:59 deliciousrocketship docker-compose[1169935]: Removing network etc_cyberchef_local
Dec 06 00:02:59 deliciousrocketship docker-compose[1169935]: Removing network etc_default
Dec 06 00:02:59 deliciousrocketship docker-compose[1169935]: Removing network etc_ewsposter_local
Dec 06 00:02:59 deliciousrocketship docker-compose[1169935]: Network etc_ewsposter_local not found.
Dec 06 00:02:59 deliciousrocketship docker-compose[1169935]: Removing network etc_spiderfoot_local
Dec 06 00:02:59 deliciousrocketship docker-compose[1169935]: Network etc_spiderfoot_local not found.
Dec 06 00:03:00 deliciousrocketship docker-compose[1170347]: No stopped containers
Dec 06 00:03:01 deliciousrocketship bash[1170350]: Error response from daemon: bridge is a pre-defined network and cannot be removed
Dec 06 00:03:01 deliciousrocketship bash[1170350]: Error response from daemon: host is a pre-defined network and cannot be removed
Dec 06 00:03:01 deliciousrocketship bash[1170350]: Error response from daemon: none is a pre-defined network and cannot be removed
Dec 06 00:03:01 deliciousrocketship bash[1170368]: "docker volume rm" requires at least 1 argument.
Dec 06 00:03:01 deliciousrocketship bash[1170368]: See 'docker volume rm --help'.
Dec 06 00:03:01 deliciousrocketship bash[1170368]: Usage:  docker volume rm [OPTIONS] VOLUME [VOLUME...]
Dec 06 00:03:01 deliciousrocketship bash[1170368]: Remove one or more volumes
Dec 06 00:03:01 deliciousrocketship bash[1170386]: "docker rm" requires at least 1 argument.
Dec 06 00:03:01 deliciousrocketship bash[1170386]: See 'docker rm --help'.
Dec 06 00:03:01 deliciousrocketship bash[1170386]: Usage:  docker rm [OPTIONS] CONTAINER [CONTAINER...]
Dec 06 00:03:01 deliciousrocketship bash[1170386]: Remove one or more containers
Dec 06 00:03:01 deliciousrocketship bash[1170407]: "docker rmi" requires at least 1 argument.
Dec 06 00:03:01 deliciousrocketship bash[1170407]: See 'docker rmi --help'.
Dec 06 00:03:01 deliciousrocketship bash[1170407]: Usage:  docker rmi [OPTIONS] IMAGE [IMAGE...]
Dec 06 00:03:01 deliciousrocketship bash[1170407]: Remove one or more images
Dec 06 00:03:01 deliciousrocketship rules.sh[1170446]: All arguments met. Continuing.
Dec 06 00:03:01 deliciousrocketship rules.sh[1170446]: Detected glutton as NFQ based honeypot, iptables-legacy rules needed. Continuing.
Dec 06 00:03:02 deliciousrocketship rules.sh[1170446]: Setting up / removing these ports:
Dec 06 00:03:02 deliciousrocketship rules.sh[1170446]: 7

[...]

Dec 06 00:03:02 deliciousrocketship rules.sh[1170446]: 64303
Dec 06 00:03:02 deliciousrocketship systemd[1]: Started tpot.
Dec 06 00:03:02 deliciousrocketship docker-compose[1170519]: Creating network "etc_adbhoney_local" with the default driver
Dec 06 00:03:03 deliciousrocketship docker-compose[1170519]: Creating network "etc_conpot_local_IEC104" with the default driver
Dec 06 00:03:03 deliciousrocketship docker-compose[1170519]: Creating network "etc_conpot_local_guardian_ast" with the default driver
Dec 06 00:03:03 deliciousrocketship docker-compose[1170519]: Creating network "etc_conpot_local_ipmi" with the default driver
Dec 06 00:03:04 deliciousrocketship docker-compose[1170519]: Creating network "etc_conpot_local_kamstrup_382" with the default driver
Dec 06 00:03:04 deliciousrocketship docker-compose[1170519]: Creating network "etc_cowrie_local" with the default driver
Dec 06 00:03:04 deliciousrocketship docker-compose[1170519]: Creating network "etc_heralding_local" with the default driver
Dec 06 00:03:04 deliciousrocketship docker-compose[1170519]: Creating network "etc_honeypy_local" with the default driver
Dec 06 00:03:05 deliciousrocketship docker-compose[1170519]: Creating network "etc_mailoney_local" with the default driver
Dec 06 00:03:05 deliciousrocketship docker-compose[1170519]: Creating network "etc_medpot_local" with the default driver
Dec 06 00:03:05 deliciousrocketship docker-compose[1170519]: Creating network "etc_rdpy_local" with the default driver
Dec 06 00:03:06 deliciousrocketship docker-compose[1170519]: Creating network "etc_tanner_local" with the default driver
Dec 06 00:03:06 deliciousrocketship docker-compose[1170519]: Creating network "etc_cyberchef_local" with the default driver
Dec 06 00:03:06 deliciousrocketship docker-compose[1170519]: Creating network "etc_default" with the default driver
Dec 06 00:03:07 deliciousrocketship docker-compose[1170519]: Creating network "etc_ewsposter_local" with the default driver
Dec 06 00:03:07 deliciousrocketship docker-compose[1170519]: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network
Dec 06 00:03:07 deliciousrocketship systemd[1]: tpot.service: Main process exited, code=exited, status=1/FAILURE
Dec 06 00:03:07 deliciousrocketship rules.sh[1171301]: All arguments met. Continuing.
Dec 06 00:03:07 deliciousrocketship rules.sh[1171301]: Detected glutton as NFQ based honeypot, iptables-legacy rules needed. Continuing.
Dec 06 00:03:07 deliciousrocketship rules.sh[1171301]: Setting up / removing these ports:
Dec 06 00:03:07 deliciousrocketship rules.sh[1171301]: 7

[...]

Dec 06 00:03:07 deliciousrocketship systemd[1]: tpot.service: Failed with result 'exit-code'.
Dec 06 00:03:12 deliciousrocketship systemd[1]: tpot.service: Scheduled restart job, restart counter is at 3489.
Dec 06 00:03:12 deliciousrocketship systemd[1]: Stopped tpot.
Dec 06 00:03:12 deliciousrocketship systemd[1]: Starting tpot...
Dec 06 00:03:12 deliciousrocketship updateip.sh[1171378]: Trying: dig +short myip.opendns.com @resolver1.opendns.com

... And back to the beginning.

From dps.sh at any point in this cycle:

=======| System |========
    Date:  Fri 06 Dec 2019 12:10:40 AM UTC
  Uptime:  00:10:40 up 18:30,  1 user,  load average: 1.05, 1.23, 1.26

NAME                  STATUS                       PORTS
adbhoney              DOWN                  
ciscoasa              DOWN                  
conpot_guardian_ast   DOWN                  
conpot_iec104         DOWN                  
conpot_ipmi           DOWN                  
conpot_kamstrup_382   DOWN                  
cowrie                DOWN                  
cyberchef             DOWN                  
dionaea               DOWN                  
elasticsearch         DOWN                  
ewsposter             DOWN                  
fatt                  DOWN                  
glutton               DOWN                  
head                  DOWN                  
heralding             DOWN                  
honeypy               DOWN                  
kibana                DOWN                  
logstash              DOWN                  
mailoney              DOWN                  
medpot                DOWN                  
nginx                 DOWN                  
p0f                   DOWN                  
rdpy                  DOWN                  
snare                 DOWN                  
spiderfoot            DOWN                  
suricata              DOWN                  
tanner                DOWN                  
tanner_api            DOWN                  
tanner_phpox          DOWN                  
tanner_redis          DOWN                  
tanner_web            DOWN

When running docker-compose manually I see this:

[root@deliciousrocketship:~]# /usr/bin/docker-compose -f /opt/tpot/etc/tpot.yml up
Creating network "etc_adbhoney_local" with the default driver
Creating network "etc_conpot_local_IEC104" with the default driver
Creating network "etc_conpot_local_guardian_ast" with the default driver
Creating network "etc_conpot_local_ipmi" with the default driver
Creating network "etc_conpot_local_kamstrup_382" with the default driver
Creating network "etc_cowrie_local" with the default driver
Creating network "etc_heralding_local" with the default driver
Creating network "etc_honeypy_local" with the default driver
Creating network "etc_mailoney_local" with the default driver
Creating network "etc_medpot_local" with the default driver
Creating network "etc_rdpy_local" with the default driver
Creating network "etc_tanner_local" with the default driver
Creating network "etc_cyberchef_local" with the default driver
Creating network "etc_default" with the default driver
Creating network "etc_ewsposter_local" with the default driver
ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network
[root@deliciousrocketship:~]#

Then when bringing it down:

[root@deliciousrocketship:~]# /usr/bin/docker-compose -f /opt/tpot/etc/tpot.yml down -v
Removing network etc_adbhoney_local
Removing network etc_conpot_local_IEC104
Removing network etc_conpot_local_guardian_ast
Removing network etc_conpot_local_ipmi
Removing network etc_conpot_local_kamstrup_382
Removing network etc_cowrie_local
Removing network etc_heralding_local
Removing network etc_honeypy_local
Removing network etc_mailoney_local
Removing network etc_medpot_local
Removing network etc_rdpy_local
Removing network etc_tanner_local
Removing network etc_cyberchef_local
Removing network etc_default
Removing network etc_ewsposter_local
WARNING: Network etc_ewsposter_local not found.
Removing network etc_spiderfoot_local
WARNING: Network etc_spiderfoot_local not found.

Checks

Before you post your issue make sure it has not been answered yet and provide basic support information if you come to the conclusion it is a new issue:

• [x] 🔍 Use the search function first
• [x] 🧐 Check our WIKI
• [x] 📚 Consult the documentation of 💻 Debian, 🐳 Docker, the 🦌 ELK stack and the 🍯 T-Pot Readme.
• [x] ⚠️ Provide basic support information or similiar information with regard to your issue or we can not help you and will close the issue without further notice

⚠️ Basic support information (commands are expected to run as root)

• [x] What version of the OS are you currently using lsb_release -a and uname -a?

  [root@deliciousrocketship:~]# lsb_release -a
  No LSB modules are available.
  Distributor ID: Debian
  Description:    Debian GNU/Linux bullseye/sid
  Release:        unstable
  Codename:       sid

  Linux deliciousrocketship 5.3.0-3-amd64 #1 SMP Debian 5.3.15-1 (2019-12-07) x86_64 GNU/Linux

• [x] What T-Pot version are you currently using? Changelog.md says 20191016
• [x] What edition (Standard, Nextgen, etc.) of T-Pot are you running? Nextgen, but I tried Standard as well.
• [x] What architecture are you running on (i.e. hardware, cloud, VM, etc.)? I've tried VM and hardware.
• [x] Did you have any problems during the install? If yes, please attach /install.log /install.err.
• [x] How long has your installation been running? I just rebooted it ¯_(ᐛ)_/¯
• [x] Did you install upgrades, packages or use the update script? I used the upgrade script several times.
• [x] Did you modify any scripts or configs? If yes, please attach the changes.
• [x] Please provide a screenshot of glances and htop.
• [x] How much free disk space is available (df -h)? Over a terabyte :)
• [x] What is the current container status (dps.sh)? Mentioned previously
• [x] What is the status of the T-Pot service (systemctl status tpot)? Mentioned previously
• [x] What ports are being occupied? Stop T-Pot systemctl stop tpot and run netstat -tulpen

  Active Internet connections (only servers)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name    
  tcp        0      0 0.0.0.0:64295           0.0.0.0:*               LISTEN      0          17396      615/sshd            
  tcp        0      0 0.0.0.0:61209           0.0.0.0:*               LISTEN      0          18295      595/python3         
  tcp6       0      0 :::64294                :::*                    LISTEN      0          14773      1/init              
  tcp6       0      0 :::64295                :::*                    LISTEN      0          17398      615/sshd            
  udp        0      0 0.0.0.0:68              0.0.0.0:*                           0          16877      491/dhclient        
  udp        0      0 192.168.1.231:123       0.0.0.0:*                           0          19187      617/ntpd            
  udp        0      0 127.0.0.1:123           0.0.0.0:*                           0          19185      617/ntpd            
  udp        0      0 0.0.0.0:123             0.0.0.0:*                           0          19181      617/ntpd            
  udp6       0      0 :::123                  :::*                                0          19178      617/ntpd

• [x] If a single container shows as DOWN you can run docker logs <container-name> for the latest log entries

[t3chn0m4g3]

I am AFK, but you are running services ntpd and dhclient which are bound to any and collide with Dionaea.

[ProducerMatt]

I've disabled dhclient and replaced ntpd with openntpd which doesn't bind any ports. Netstat now looks like this:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name    
tcp        0      0 0.0.0.0:64295           0.0.0.0:*               LISTEN      0          20839      645/sshd            
tcp        0      0 0.0.0.0:61209           0.0.0.0:*               LISTEN      0          21727      609/python3         
tcp6       0      0 :::64294                :::*                    LISTEN      0          14726      1/init              
tcp6       0      0 :::64295                :::*                    LISTEN      0          20841      645/sshd

But docker-compose still throws the non-overlapping IP address error and tpot restarts.

[t3chn0m4g3]

By any chance, do you have any other netmask configured than a /24 (255.255.255.0)?

[ProducerMatt]

I have a /16, making my network 192.168.x.x

What are you thinking is the issue?

[t3chn0m4g3]

Since docker-compose will try to allocate multiple /24 upon start which is by default from the same RFC1918 range as configured on your NIC (192.168.x.x), it is possible you will get the overlapping error as a result.

Switch to a /24 and it should work.

[ProducerMatt]

You're right, it is indeed working now!

I'll stick with this for the moment as working is better than not working, but it does present an issue -- some of my network isn't in the 192.168.1 range. Doesn't this mean tpot now can't access it? Is this something I could make docker-compose get to grips with, or have I just designed my network poorly?

[t3chn0m4g3]

Check out the docker documentation, you can tweak the default range, I think there was already an issue for that.

[Raul1718]

@ProducerMatt hi. I hava same errors with you. But I don't understand how to solve.Can you help me? Should I need to modify my network's netmask to 255.255.255.0? But my network's netmask is just 255.255.255.0, the error is still appear.

[nicostubi]

Hello, I'm having similar problems here. What was the solution?

[nicostubi]

When running system status tpot:

[nicostubi]

cups was using the port 631 so it was failing due to cups. When running docker-compose

I had the error: tpot error starting userland proxy: listen tcp4 0.0.0.:631: bind: address already in use

I uninstalled cups....