Unable to start servers when Docker Swarm is active

[YoshiWalsh]

Background:

• Panel or Daemon: Daemon
• Version of Panel/Daemon: v0.6.12
• Server's OS: Ubuntu Server 19.04
• Your Computer's OS & Browser: Windows 10, Firefox Developer Edition 68.0b3 (64-bit)

Describe the bug I'm trying to set up Pterodactyl on my homelab server. Docker is configured to use Swarm Mode as I intend to add a new physical server to my setup soon, I'm experimenting with a little bit of HA stuff.

The Daemon appears to be working, so I created a server using this egg. The install completed successfully, but when I try to actually start the server I get an error.

[Pterodactyl Daemon] Server marked as STARTING
[Pterodactyl Daemon] Checking size of server data directory...
[Pterodactyl Daemon] Disk Usage: 259M / 20480M
[Pterodactyl Daemon] Ensuring file permissions.
[Pterodactyl Daemon] Running server preflight.
[Pterodactyl Daemon] Starting server container.
[Pterodactyl Daemon] Server marked as OFF
[Pterodactyl Daemon] A fatal error was encountered while starting this server.

I was able to find additional details within wings.log. (Am I missing something, or is there no GUI way to view these logs?)

/srv/daemon/logs # cat wings.log
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"+ ------------------------------------ +","time":"2019-05-24T10:28:31.368Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"| Running Pterodactyl Daemon v0.6.12    |","time":"2019-05-24T10:28:31.370Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"|        https://pterodactyl.io        |","time":"2019-05-24T10:28:31.370Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"|  Copyright 2015 - 2019 Dane Everitt  |","time":"2019-05-24T10:28:31.370Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"+ ------------------------------------ +","time":"2019-05-24T10:28:31.370Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Loading modules, this could take a few seconds.","time":"2019-05-24T10:28:31.370Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Modules loaded, starting Pterodactyl Daemon...","time":"2019-05-24T10:28:31.788Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Configuring user pterodactyl (id: 500) as the owner of all server files.","time":"2019-05-24T10:28:31.855Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Configuring timezone file location...","time":"2019-05-24T10:28:31.858Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Checking container networking environment...","time":"2019-05-24T10:28:31.866Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Found network interface for daemon: docker_gwbridge","time":"2019-05-24T10:28:31.875Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Ensuring correct network interface for containers...","time":"2019-05-24T10:28:31.875Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Networking gateway detected as 172.18.0.1 for interface: pterodactyl0.","time":"2019-05-24T10:28:31.887Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Contacting panel to retrieve a list of currrent Eggs available to the node.","time":"2019-05-24T10:28:31.888Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Checking existing eggs against Panel response...","time":"2019-05-24T10:28:32.033Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Beginning server initialization process.","time":"2019-05-24T10:28:32.600Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Configuring websocket for daemon stats...","time":"2019-05-24T10:28:32.629Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Configuring internal SFTP server...","time":"2019-05-24T10:28:32.630Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Pterodactyl Daemon is up-to-date!","time":"2019-05-24T10:28:32.649Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"level":30,"msg":"Pterodactyl Daemon is now listening for insecure connections on 0.0.0.0:8080","time":"2019-05-24T10:28:32.650Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"server":"938aa123-b8d8-4e71-a67c-9dbe7dd8077e","level":30,"msg":"Server status has been changed to STARTING","time":"2019-05-24T10:29:01.807Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"server":"938aa123-b8d8-4e71-a67c-9dbe7dd8077e","level":30,"msg":"Server status has been changed to OFF","time":"2019-05-24T10:29:02.111Z","v":0}
{"name":"wings","hostname":"afa11b8b6c8b","pid":28,"server":"938aa123-b8d8-4e71-a67c-9dbe7dd8077e","level":50,"err":{"message":"(HTTP code 500) server error - Could not attach to network ingress: rpc error: code = PermissionDenied desc = network ingress not manually attachable ","name":"Error","stack":"Error: (HTTP code 500) server error - Could not attach to network ingress: rpc error: code = PermissionDenied desc = network ingress not manually attachable \n    at /srv/daemon/node_modules/docker-modem/lib/modem.js:254:17\n    at getCause (/srv/daemon/node_modules/docker-modem/lib/modem.js:284:7)\n    at Modem.buildPayload (/srv/daemon/node_modules/docker-modem/lib/modem.js:253:5)\n    at IncomingMessage.<anonymous> (/srv/daemon/node_modules/docker-modem/lib/modem.js:229:14)\n    at emitNone (events.js:111:20)\n    at IncomingMessage.emit (events.js:208:7)\n    at endReadableNT (_stream_readable.js:1064:12)\n    at _combinedTickCallback (internal/process/next_tick.js:139:11)\n    at process._tickDomainCallback (internal/process/next_tick.js:219:9)"},"msg":"(HTTP code 500) server error - Could not attach to network ingress: rpc error: code = PermissionDenied desc = network ingress not manually attachable ","time":"2019-05-24T10:29:02.113Z","v":0}

As you can see, the issue is that the container is trying to attach to ingress, but ingress is not an attachable network. I've done a little bit of Googling about this and the only thing I can find is this which suggests that when Docker is configured with Swarm mode active it's impossible for standalone containers to attach to the ingress network. But that issue also says it was fixed in Docker 1.13. My Docker version is 19.03.0-beta4 (API version 1.40) so I assume that issue can't be the cause.

yoshie@compute01:~$ uname -a
Linux compute01 5.0.0-13-generic #14-Ubuntu SMP Mon Apr 15 14:59:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
yoshie@compute01:~$ docker info
Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0-beta2)

Server:
 Containers: 12
  Running: 7
  Paused: 0
  Stopped: 5
 Images: 12
 Server Version: 19.03.0-beta4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: rg2d0t26h2lzxmtkp8cmuuy8t
  Is Manager: true
  ClusterID: v8x97hewiwnxoh7djh49rnl0r
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.10.211
  Manager Addresses:
   192.168.10.211:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
 runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.0.0-13-generic
 Operating System: Ubuntu 19.04
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 82.55GiB
 Name: compute01
 ID: 12a19a71-0b07-42d2-9fc0-849d7669157e
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

To Reproduce Steps to reproduce the behavior:

1. Install Docker
2. Enable Swarm
3. Install pterodactyl daemon and set it up as a Node in pterodactyl panel
4. Change the network name in the generated core.json from the default pterodactyl_nw to docker_gwbridge, otherwise the Daemon will probably fail to start. Place the core.json file within the config directory of the daemon installation.
5. Create a new Server in the panel on the node. Observe that it should install successfully.
6. Attempt to start the Server. Observe that it fails.

Expected behavior I expect the created Server to start successfully.

[parkervcp]

Instead of changing the network why don't you leave it default and find out if it fails... instead of guessing that it will fail?

[YoshiWalsh]

Sorry for the lack of clarity there. It fails for me unless I change it, I said 'probably' because I'm not sure if the same issue happens for others or if it's specific to my setup.

[YoshiWalsh]

@parkervcp if I leave the network as default I get this error:

14:24:25.593Z  INFO wings: + ------------------------------------ +
14:24:25.595Z  INFO wings: | Running Pterodactyl Daemon v0.6.12    |
14:24:25.595Z  INFO wings: |        https://pterodactyl.io        |
14:24:25.595Z  INFO wings: |  Copyright 2015 - 2019 Dane Everitt  |
14:24:25.595Z  INFO wings: + ------------------------------------ +
14:24:25.595Z  INFO wings: Loading modules, this could take a few seconds.
14:24:26.016Z  INFO wings: Modules loaded, starting Pterodactyl Daemon...
14:24:26.086Z  INFO wings: Configuring user pterodactyl (id: 500) as the owner of all server files.
14:24:26.089Z  INFO wings: Configuring timezone file location...
14:24:26.098Z  INFO wings: Checking container networking environment...
14:24:26.106Z  WARN wings: No isolated network interface for containers was detected, creating one now.
14:24:26.111Z FATAL wings: A fatal error caused the daemon to abort the startup.
    Error: (HTTP code 403) unexpected - Pool overlaps with other one on this address space 
        at /srv/daemon/node_modules/docker-modem/lib/modem.js:254:17
        at getCause (/srv/daemon/node_modules/docker-modem/lib/modem.js:284:7)
        at Modem.buildPayload (/srv/daemon/node_modules/docker-modem/lib/modem.js:253:5)
        at IncomingMessage.<anonymous> (/srv/daemon/node_modules/docker-modem/lib/modem.js:229:14)
        at emitNone (events.js:111:20)
        at IncomingMessage.emit (events.js:208:7)
        at endReadableNT (_stream_readable.js:1064:12)
        at _combinedTickCallback (internal/process/next_tick.js:139:11)
        at process._tickDomainCallback (internal/process/next_tick.js:219:9)
    --
    additional: {
      "statusCode": 403,
      "json": {
        "message": "Pool overlaps with other one on this address space"
      }
    }
14:24:26.111Z ERROR wings: You should forcibly quit this process (CTRL+C) and attempt to fix the issue.

[DaneEveritt]

@JoshuaWalsh check out this documentation for how to change the interface used when creating the pterodactyl network.

[YoshiWalsh]

Hi @DaneEveritt , that worked (after deleting my Server and recreating it). Thanks!

EDIT: I didn't want to use the host network so I changed back to pterodactyl_nw, but I specified a different subnet in interfaces and it avoided the pool overlap error. Now everything is working just how I want it. Thanks!

[magnus919]

@JoshuaWalsh check out this documentation for how to change the interface used when creating the pterodactyl network.

For those googling ye olde "Pool overlaps with other one on this address space" error on Docker Swarm, this documentation link is no longer around. But the error remains.

[parkervcp]

It moved from daemon to wings https://pterodactyl.io/wings/1.0/configuration.html#custom-network-interfaces

[magnus919]

It moved from daemon to wings https://pterodactyl.io/wings/1.0/configuration.html#custom-network-interfaces

That fixed it! Doc site behaves weirdly in brave browser so subsections didn't come up in RTFM, but direct link to it worked.

[mzramna]

i'm having a related problem, but not the same, i cannot set the pterodactyl when using a nginx reverse proxy in swarm mode, i have a swag reverse proxy and if i redirect the pterodactyl to be accessed trough there i cannot use mostly of the system, if i uses https on pterodactyl i cannot access it trough the reverse proxy anyway, but if i set the http to access pterodactyl i cannot access some features as create a new server or register a new node. ps: i'm using all the machines into the swarm to recive wings using the global distribution and each new machine i put i register a new reverse proxy to the swag access the wings host in a new subdomain