when trying to install CapRover on a grid 3 following CapRover installation instructions here we weren't able to access the dashboard on port 3000.
using full VM on DigitalOcean the Ingress/Overlay network is working as expected.
we also found that issue that refer to use the edge version as it has this new commit that possibly fix it for OS versions that are having this problem.
this fixed the port issue but introduced another Ingress/Overlay network issue after connecting the root domain to the server IP address. apparently Nginx container can't reach the VirtualIp of the ingress service captain-captain that already attached to the same overlay network and gives 502 when we try to access the dashboard through captain.roverapps.grid.tf.
here is the the details from a debugging session
root@vm1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1abd7687d06c nginx:1 "/docker-entrypoint.…" About a minute ago Up 59 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp captain-nginx.1.pahiv797drldfowk9yg3ot8k6
48e70b886f7a caprover/certbot-sleeping:v1.6.0 "/bin/sh -c 'sleep 9…" About a minute ago Up 51 seconds 80/tcp, 443/tcp captain-certbot.1.sngugbmfc3k7p1ewc1d7w6i7h
270d7b503c7f caprover/caprover-edge:0.0.1 "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp captain-captain.1.n3d91pvuyomed2zzxkf5zr44p
root@vm1:~# docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
0932qc4oesrj captain-captain replicated 1/1 caprover/caprover-edge:0.0.1
qrgggv5klzme captain-certbot replicated 1/1 caprover/certbot-sleeping:v1.6.0
hz76taorz11k captain-nginx replicated 0/1 nginx:1
root@vm1:~# docker service ps captain-captain --no-trunc
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
ra6kuu0vfacw9gtz5ah2jq7r7 captain-captain.1 caprover/caprover-edge:0.0.1 vm1 Running Running 47 seconds ago *:3000->3000/tcp,*:3000->3000/tcp
zodmsrsd5be43nyernzfmmdaj \_ captain-captain.1 caprover/caprover-edge:0.0.1 vm1 Shutdown Failed 56 seconds ago "task: non-zero exit (1)"
n3d91pvuyomed2zzxkf5zr44p \_ captain-captain.1 caprover/caprover-edge:0.0.1 vm1 Shutdown Failed 2 minutes ago "task: non-zero exit (1)"
qnjco477mnf952zs4ladowtjz \_ captain-captain.1 caprover/caprover-edge:0.0.1 vm1 Shutdown Failed 4 minutes ago "task: non-zero exit (1)"
nz21md0kieqgwzavx3h7681mh \_ captain-captain.1 caprover/caprover-edge:0.0.1 vm1 Shutdown Failed 6 minutes ago "task: non-zero exit (1)"
root@vm1:~# docker service logs captain-captain --since 60m
captain-captain.1.l8zio3vwtmbr@vm1 | Captain Starting ...
captain-captain.1.l8zio3vwtmbr@vm1 | Overriding publishedNameOnDockerHub from /usr/src/app/config-override.json
captain-captain.1.l8zio3vwtmbr@vm1 | Overriding version from /usr/src/app/config-override.json
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:14.878 am Emptying generated and temp folders.
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:14.926 am Ensuring directories are available on host. Started.
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:14.955 am Ensuring directories are available on host. Finished.
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:14.972 am Network captain-overlay-network is already attached to service: captain-captain
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:14.989 am captain-captain (service) has already been connected to secret: captain-salt
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:14.995 am Migration not needed, skipping.
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:15.023 am Copying fake certificates...
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:15.045 am Updating Load Balancer - Setting up NGINX conf file...
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:15.046 am Locking NGINX configuration reloading...
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:15.098 am SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:15.106 am Captain Nginx is already running..
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:15.115 am Updating NGINX service...
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:15.145 am Waiting for 5 seconds for nginx reload to take into effect
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:20.151 am Pruning containers...
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:20.154 am NGINX is fully set up and working...
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:20.174 am Captain Certbot is already running..
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:20.186 am Updating Certbot service...
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:23.246 am Retrying to get containerId for captain-certbot retry count:0
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:25.226 am Pruning containers...
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:25.233 am Skipping prune due to a minor error: Error: (HTTP code 409) unexpected - a prune operation is already running
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:26.253 am Retrying to get containerId for captain-certbot retry count:1
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:29.264 am Retrying to get containerId for captain-certbot retry count:2
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:32.271 am Retrying to get containerId for captain-certbot retry count:3
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:35.284 am Retrying to get containerId for captain-certbot retry count:4
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:35.296 am executeCommand Container: captain-certbot certbot certificates --non-interactive
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:36.937 am **** Captain is initialized and ready to serve you! ****
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:39.999 am Captain health check failed: #0 at captain.roverapps.grid.tf
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:46.934 am executeCommand Container: captain-certbot certbot certificates --non-interactive
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:48.406 am executeCommand Container: captain-certbot certbot renew --non-interactive
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:49.855 am Updating Load Balancer - renewAllCerts
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:49.855 am Locking NGINX configuration reloading...
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:49.886 am SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:49.886 am sendReloadSignal...
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:04:49.892 am Kill HUP Container: 96eb8b4621f0b40b000ef4b9564136b8a0dc2423f5673fbca766f1317a81734d
captain-captain.1.l8zio3vwtmbr@vm1 | October 5th 2021, 11:05:03.068 am Captain health check failed: #1 at captain.roverapps.grid.tf
root@vm1:~# docker logs captain-nginx.1.vvle7yrisscxehrwh3e3ngqqf
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2021/10/05 11:06:58 [error] 24#24: *1 connect() failed (113: No route to host) while connecting to upstream, client: 172.18.0.1, server: captain.roverapps.grid.tf, request: "GET /checkhealth HTTP/1.1", upstream: "http://10.0.1.2:3000/checkhealth", host: "captain.roverapps.grid.tf"
root@vm1:~# docker service inspect captain-captain
[
{
"ID": "0932qc4oesrjqzs2ti2ec03mu",
"Version": {
"Index": 47
},
"CreatedAt": "2021-10-05T09:51:15.225659442Z",
"UpdatedAt": "2021-10-05T09:51:50.434003339Z",
"Spec": {
"Name": "captain-captain",
"Labels": {},
"TaskTemplate": {
"ContainerSpec": {
"Image": "caprover/caprover-edge:0.0.1",
"Labels": {
"randomLabelForceUpdate": "bb16e586-70f0-4e5e-a9ba-8dfc50dd3e9e"
},
"Env": [
"IS_CAPTAIN_INSTANCE=1"
],
"Mounts": [
{
"Type": "bind",
"Source": "/captain",
"Target": "/captain"
},
{
"Type": "bind",
"Source": "/var/run/docker.sock",
"Target": "/var/run/docker.sock"
}
],
"StopGracePeriod": 10000000000,
"DNSConfig": {},
"Secrets": [
{
"File": {
"Name": "captain-salt",
"UID": "0",
"GID": "0",
"Mode": 292
},
"SecretID": "bx5o66qcifb51fn0ugsnowx0w",
"SecretName": "captain-salt"
}
],
"Isolation": "default"
},
"Resources": {},
"RestartPolicy": {
"Condition": "any",
"Delay": 5000000000,
"MaxAttempts": 0
},
"Placement": {
"Constraints": [
"node.id == ri03jp8udscfsddrwljj773e5"
]
},
"Networks": [
{
"Target": "nuqyuts03ithqasl7aol5nyqq"
}
],
"LogDriver": {
"Name": "json-file",
"Options": {
"max-size": "512m"
}
},
"ForceUpdate": 0,
"Runtime": "container"
},
"Mode": {
"Replicated": {
"Replicas": 1
}
},
"UpdateConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"RollbackConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"EndpointSpec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 3000,
"PublishedPort": 3000,
"PublishMode": "host"
}
]
}
},
"PreviousSpec": {
"Name": "captain-captain",
"Labels": {},
"TaskTemplate": {
"ContainerSpec": {
"Image": "caprover/caprover-edge:0.0.1",
"Labels": {
"randomLabelForceUpdate": "ee2c56b0-a2ee-4c12-a506-f407aac9916b"
},
"Env": [
"IS_CAPTAIN_INSTANCE=1"
],
"Mounts": [
{
"Type": "bind",
"Source": "/captain",
"Target": "/captain"
},
{
"Type": "bind",
"Source": "/var/run/docker.sock",
"Target": "/var/run/docker.sock"
}
],
"Isolation": "default"
},
"Resources": {},
"Placement": {
"Constraints": [
"node.id == ri03jp8udscfsddrwljj773e5"
]
},
"Networks": [
{
"Target": "nuqyuts03ithqasl7aol5nyqq"
}
],
"LogDriver": {
"Name": "json-file",
"Options": {
"max-size": "512m"
}
},
"ForceUpdate": 0,
"Runtime": "container"
},
"Mode": {
"Replicated": {
"Replicas": 1
}
},
"EndpointSpec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 3000,
"PublishedPort": 3000,
"PublishMode": "host"
}
]
}
},
"Endpoint": {
"Spec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 3000,
"PublishedPort": 3000,
"PublishMode": "host"
}
]
},
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 3000,
"PublishedPort": 3000,
"PublishMode": "host"
}
],
"VirtualIPs": [
{
"NetworkID": "nuqyuts03ithqasl7aol5nyqq",
"Addr": "10.0.1.2/24"
}
]
},
"UpdateStatus": {
"State": "completed",
"StartedAt": "2021-10-05T09:51:22.836601272Z",
"CompletedAt": "2021-10-05T09:51:50.433952543Z",
"Message": "update completed"
}
}
]
root@vm1:~# docker network ls
NETWORK ID NAME DRIVER SCOPE
f5ba312731e4 bridge bridge local
nuqyuts03ith captain-overlay-network overlay swarm
20ea1054c510 docker_gwbridge bridge local
879bc5ee478f host host local
32w01r0pp1f6 ingress overlay swarm
e89154989237 none null local
root@vm1:~# docker network inspect captain-overlay-network
[
{
"Name": "captain-overlay-network",
"Id": "nuqyuts03ithqasl7aol5nyqq",
"Created": "2021-10-05T09:51:35.368028878Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.1.0/24",
"Gateway": "10.0.1.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"3d4bbed9df26a639276421cdfb392d375d12558f4b7ece214efaf666b30a78d0": {
"Name": "captain-nginx.1.lee0kahw8ds9a0m7eefr6kcro",
"EndpointID": "2662c9be5c3253f5ee7196f0f2a4b5f336cd23e91f106dd2e3b2b4ea56c1331d",
"MacAddress": "02:42:0a:00:01:49",
"IPv4Address": "10.0.1.73/24",
"IPv6Address": ""
},
"50c0ebc305bae8252e0cd03a93b6dda470ea2de5d5c3d0cfc555ffd4f666d075": {
"Name": "captain-captain.1.9ktrn4limz0kmhc1w79m06rpa",
"EndpointID": "f7313b3094249b6016792eb8ad470291927145e9cb5df67fd2d844d6f474deee",
"MacAddress": "02:42:0a:00:01:48",
"IPv4Address": "10.0.1.72/24",
"IPv6Address": ""
},
"lb-captain-overlay-network": {
"Name": "captain-overlay-network-endpoint",
"EndpointID": "d34dbad56634e8cd89038d6f1e9156dda50611fe9bc270f3c0a75b19f420463f",
"MacAddress": "02:42:0a:00:01:05",
"IPv4Address": "10.0.1.5/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": {},
"Peers": [
{
"Name": "c8fe65020ff6",
"IP": "127.0.0.1"
}
]
}
]
root@vm1:~# docker service update --network-add captain-overlay-network captain-captain
service is already attached to network captain-overlay-network
# disabling the health check so containers stop restarting
root@vm1:~# echo "{\"skipVerifyingDomains\":\"true\"}" > /captain/data/config-override.json
root@vm1:~# docker service update captain-captain --force
root@vm1:~# docker network inspect captain-overlay-network
[
{
"Name": "captain-overlay-network",
"Id": "nuqyuts03ithqasl7aol5nyqq",
"Created": "2021-10-05T09:51:35.368028878Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.1.0/24",
"Gateway": "10.0.1.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"62d4710e2b36a9b705d9549d871e8c695e7b5739f6dd8b8677e95626d12c8566": {
"Name": "captain-captain.1.lvixdsiml4pdomp11gsboftki",
"EndpointID": "d84da8e34cafe9459f2f2d3d7f67203b5432ff879de60d186e5abf19344ad543",
"MacAddress": "02:42:0a:00:01:4e",
"IPv4Address": "10.0.1.78/24",
"IPv6Address": ""
},
"73c10958da4856f811392d24c6f262218042a0a4c19ad9c424bdf78bb64d0703": {
"Name": "captain-nginx.1.r9pgfn2of0p0k3nzk0zzaud5f",
"EndpointID": "6b12ff57223894071963c33d7c04a8913c25bac1be0e23c20e2d711a7cb332c2",
"MacAddress": "02:42:0a:00:01:4f",
"IPv4Address": "10.0.1.79/24",
"IPv6Address": ""
},
"lb-captain-overlay-network": {
"Name": "captain-overlay-network-endpoint",
"EndpointID": "d34dbad56634e8cd89038d6f1e9156dda50611fe9bc270f3c0a75b19f420463f",
"MacAddress": "02:42:0a:00:01:05",
"IPv4Address": "10.0.1.5/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": {},
"Peers": [
{
"Name": "c8fe65020ff6",
"IP": "127.0.0.1"
}
]
}
]
root@vm1:~# docker exec -it captain-nginx.1.r9pgfn2of0p0k3nzk0zzaud5f /bin/bash
# curl the docker swarm service captain-captain VirtualIp from the Nginx container (both are in same network)
root@73c10958da48:/# curl http://10.0.1.2:3000 -v
* Expire in 0 ms for 6 (transfer 0x55a72a082ee0)
* Trying 10.0.1.2...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55a72a082ee0)
* connect to 10.0.1.2 port 3000 failed: No route to host
* Failed to connect to 10.0.1.2 port 3000: No route to host
* Closing connection 0
curl: (7) Failed to connect to 10.0.1.2 port 3000: No route to host
# curl caprover regular container from the Nginx container
root@73c10958da48:/# curl http://10.0.1.79 -v
* Expire in 0 ms for 6 (transfer 0x55bfe1e79ee0)
* Trying 10.0.1.79...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55bfe1e79ee0)
* Connected to 10.0.1.79 (10.0.1.79) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.0.1.79
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 05 Oct 2021 11:29:43 GMT
< Content-Type: text/html
< Content-Length: 2401
< Last-Modified: Tue, 05 Oct 2021 11:21:41 GMT
< Connection: keep-alive
< ETag: "615c3545-961"
< Accept-Ranges: bytes
<
<!doctype html>
<html>
<head>
<meta charset=utf-8>
<meta content="width=device-width,initial-scale=1.0,minimum-scale=1.0,maximum-scale=1.0,user-scalable=no" name=viewport>
<title>Powered by CapRover</title>
<style>
html,
body {
font-family: sans-serif;
-ms-text-size-adjust: 100%;
-webkit-text-size-adjust: 100%;
background-color: #F7F8FB;
height: 100%;
-webkit-font-smoothing: antialiased;
}
body {
margin: 0;
padding: 0;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
}
.message {
text-align: center;
align-self: center;
display: flex;
flex-direction: column;
align-items: center;
padding: 20px;
max-width: 550px;
}
.message__title {
font-size: 32px;
font-weight: 100;
margin-top: 15px;
color: #47494E;
margin-bottom: 8px;
}
.btn {
text-decoration: none;
padding: 11px 25px;
border-radius: 11px;
margin-top: 110px;
font-size: 19px;
color: #7F828B;
}
body.defaultpagebody {
background: -webkit-linear-gradient(-45deg, #1d5b85 0%, #3295c7 100%);
background: linear-gradient(135deg, #1d5b85 0%, #3295c7 100%);
}
.message__title {
color: #fff;
}
.message {
padding: 25px;
color: #fff;
}
body.defaultpagebody p {
color: rgba(255, 255, 255, 0.6);
}
body.defaultpagebody .info {
fill: rgba(255, 255, 255, 0.9);
}
body.defaultpagebody .btn {
color: #fff;
border: 2px solid rgba(255, 255, 255, 0.7);
}
</style>
<base target=_parent />
</head>
<body class=defaultpagebody>
<div class=message>
<div class=message__title>
Nothing here yet :/
</div>
<div class=message>
</div>
<a href="https://caprover.com/" target="_blank" rel="noopener noreferrer" class="btn">
Read Docs</a>
</div>
</body>
* Connection #0 to host 10.0.1.79 left intact
when trying to install CapRover on a grid 3 following CapRover installation instructions here we weren't able to access the dashboard on port 3000.
using full VM on DigitalOcean the Ingress/Overlay network is working as expected.
we also found that issue that refer to use the edge version as it has this new commit that possibly fix it for OS versions that are having this problem.
this fixed the port issue but introduced another Ingress/Overlay network issue after connecting the root domain to the server IP address. apparently
Nginx
container can't reach theVirtualIp
of the ingress servicecaptain-captain
that already attached to the same overlay network and gives 502 when we try to access the dashboard through captain.roverapps.grid.tf.here is the the details from a debugging session
finally here is the full docker log which contains some errors https://gist.github.com/sameh-farouk/89a62aeca9c27607a50bb5de8e39bd22