search

threefoldtech / zos

Autonomous operating system
https://threefold.io/host/
Apache License 2.0
84 stars 14 forks source link

cant't run CapRover on grid 3 - Docker's ingress network issue #1410

Closed sameh-farouk closed 3 years ago

sameh-farouk commented 3 years ago

when trying to install CapRover on a grid 3 following CapRover installation instructions here we weren't able to access the dashboard on port 3000.

using full VM on DigitalOcean the Ingress/Overlay network is working as expected.

we also found that issue that refer to use the edge version as it has this new commit that possibly fix it for OS versions that are having this problem.

this fixed the port issue but introduced another Ingress/Overlay network issue after connecting the root domain to the server IP address. apparently Nginx container can't reach the VirtualIp of the ingress service captain-captain that already attached to the same overlay network and gives 502 when we try to access the dashboard through captain.roverapps.grid.tf.

here is the the details from a debugging session 

root@vm1:~# docker ps
CONTAINER ID   IMAGE                              COMMAND                  CREATED              STATUS              PORTS                                                                      NAMES
1abd7687d06c   nginx:1                            "/docker-entrypoint.…"   About a minute ago   Up 59 seconds       0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   captain-nginx.1.pahiv797drldfowk9yg3ot8k6
48e70b886f7a   caprover/certbot-sleeping:v1.6.0   "/bin/sh -c 'sleep 9…"   About a minute ago   Up 51 seconds       80/tcp, 443/tcp                                                            captain-certbot.1.sngugbmfc3k7p1ewc1d7w6i7h
270d7b503c7f   caprover/caprover-edge:0.0.1       "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                                  captain-captain.1.n3d91pvuyomed2zzxkf5zr44p

root@vm1:~# docker service ls
ID             NAME              MODE         REPLICAS   IMAGE                              PORTS
0932qc4oesrj   captain-captain   replicated   1/1        caprover/caprover-edge:0.0.1       
qrgggv5klzme   captain-certbot   replicated   1/1        caprover/certbot-sleeping:v1.6.0   
hz76taorz11k   captain-nginx     replicated   0/1        nginx:1               

root@vm1:~# docker service ps captain-captain --no-trunc
ID                          NAME                    IMAGE                          NODE      DESIRED STATE   CURRENT STATE            ERROR                       PORTS
ra6kuu0vfacw9gtz5ah2jq7r7   captain-captain.1       caprover/caprover-edge:0.0.1   vm1       Running         Running 47 seconds ago                               *:3000->3000/tcp,*:3000->3000/tcp
zodmsrsd5be43nyernzfmmdaj    \_ captain-captain.1   caprover/caprover-edge:0.0.1   vm1       Shutdown        Failed 56 seconds ago    "task: non-zero exit (1)"   
n3d91pvuyomed2zzxkf5zr44p    \_ captain-captain.1   caprover/caprover-edge:0.0.1   vm1       Shutdown        Failed 2 minutes ago     "task: non-zero exit (1)"   
qnjco477mnf952zs4ladowtjz    \_ captain-captain.1   caprover/caprover-edge:0.0.1   vm1       Shutdown        Failed 4 minutes ago     "task: non-zero exit (1)"   
nz21md0kieqgwzavx3h7681mh    \_ captain-captain.1   caprover/caprover-edge:0.0.1   vm1       Shutdown        Failed 6 minutes ago     "task: non-zero exit (1)"   

root@vm1:~# docker service logs captain-captain --since 60m
captain-captain.1.l8zio3vwtmbr@vm1    | Captain Starting ...
captain-captain.1.l8zio3vwtmbr@vm1    | Overriding publishedNameOnDockerHub from /usr/src/app/config-override.json
captain-captain.1.l8zio3vwtmbr@vm1    | Overriding version from /usr/src/app/config-override.json
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:14.878 am    Emptying generated and temp folders.
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:14.926 am    Ensuring directories are available on host. Started.
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:14.955 am    Ensuring directories are available on host. Finished.
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:14.972 am    Network captain-overlay-network is already attached to service: captain-captain
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:14.989 am    captain-captain (service) has already been connected to secret: captain-salt
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:14.995 am    Migration not needed, skipping.
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:15.023 am    Copying fake certificates...
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:15.045 am    Updating Load Balancer - Setting up NGINX conf file...
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:15.046 am    Locking NGINX configuration reloading...
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:15.098 am    SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:15.106 am    Captain Nginx is already running.. 
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:15.115 am    Updating NGINX service...
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:15.145 am    Waiting for 5 seconds for nginx reload to take into effect
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:20.151 am    Pruning containers...
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:20.154 am    NGINX is fully set up and working...
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:20.174 am    Captain Certbot is already running.. 
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:20.186 am    Updating Certbot service...
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:23.246 am    Retrying to get containerId for captain-certbot retry count:0
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:25.226 am    Pruning containers...
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:25.233 am    Skipping prune due to a minor error: Error: (HTTP code 409) unexpected - a prune operation is already running 
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:26.253 am    Retrying to get containerId for captain-certbot retry count:1
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:29.264 am    Retrying to get containerId for captain-certbot retry count:2
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:32.271 am    Retrying to get containerId for captain-certbot retry count:3
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:35.284 am    Retrying to get containerId for captain-certbot retry count:4
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:35.296 am    executeCommand Container: captain-certbot certbot certificates --non-interactive 
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:36.937 am    **** Captain is initialized and ready to serve you! ****
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:39.999 am    Captain health check failed: #0 at captain.roverapps.grid.tf
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:46.934 am    executeCommand Container: captain-certbot certbot certificates --non-interactive 
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:48.406 am    executeCommand Container: captain-certbot certbot renew --non-interactive 
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:49.855 am    Updating Load Balancer - renewAllCerts
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:49.855 am    Locking NGINX configuration reloading...
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:49.886 am    SUCCESS: UNLocking NGINX configuration reloading...
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:49.886 am    sendReloadSignal...
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:04:49.892 am    Kill HUP Container: 96eb8b4621f0b40b000ef4b9564136b8a0dc2423f5673fbca766f1317a81734d
captain-captain.1.l8zio3vwtmbr@vm1    | October 5th 2021, 11:05:03.068 am    Captain health check failed: #1 at captain.roverapps.grid.tf

root@vm1:~# docker logs captain-nginx.1.vvle7yrisscxehrwh3e3ngqqf
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2021/10/05 11:06:58 [error] 24#24: *1 connect() failed (113: No route to host) while connecting to upstream, client: 172.18.0.1, server: captain.roverapps.grid.tf, request: "GET /checkhealth HTTP/1.1", upstream: "http://10.0.1.2:3000/checkhealth", host: "captain.roverapps.grid.tf"

root@vm1:~# docker service inspect captain-captain
[
    {
        "ID": "0932qc4oesrjqzs2ti2ec03mu",
        "Version": {
            "Index": 47
        },
        "CreatedAt": "2021-10-05T09:51:15.225659442Z",
        "UpdatedAt": "2021-10-05T09:51:50.434003339Z",
        "Spec": {
            "Name": "captain-captain",
            "Labels": {},
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "caprover/caprover-edge:0.0.1",
                    "Labels": {
                        "randomLabelForceUpdate": "bb16e586-70f0-4e5e-a9ba-8dfc50dd3e9e"
                    },
                    "Env": [
                        "IS_CAPTAIN_INSTANCE=1"
                    ],
                    "Mounts": [
                        {
                            "Type": "bind",
                            "Source": "/captain",
                            "Target": "/captain"
                        },
                        {
                            "Type": "bind",
                            "Source": "/var/run/docker.sock",
                            "Target": "/var/run/docker.sock"
                        }
                    ],
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {},
                    "Secrets": [
                        {
                            "File": {
                                "Name": "captain-salt",
                                "UID": "0",
                                "GID": "0",
                                "Mode": 292
                            },
                            "SecretID": "bx5o66qcifb51fn0ugsnowx0w",
                            "SecretName": "captain-salt"
                        }
                    ],
                    "Isolation": "default"
                },
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Constraints": [
                        "node.id == ri03jp8udscfsddrwljj773e5"
                    ]
                },
                "Networks": [
                    {
                        "Target": "nuqyuts03ithqasl7aol5nyqq"
                    }
                ],
                "LogDriver": {
                    "Name": "json-file",
                    "Options": {
                        "max-size": "512m"
                    }
                },
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 3000,
                        "PublishedPort": 3000,
                        "PublishMode": "host"
                    }
                ]
            }
        },
        "PreviousSpec": {
            "Name": "captain-captain",
            "Labels": {},
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "caprover/caprover-edge:0.0.1",
                    "Labels": {
                        "randomLabelForceUpdate": "ee2c56b0-a2ee-4c12-a506-f407aac9916b"
                    },
                    "Env": [
                        "IS_CAPTAIN_INSTANCE=1"
                    ],
                    "Mounts": [
                        {
                            "Type": "bind",
                            "Source": "/captain",
                            "Target": "/captain"
                        },
                        {
                            "Type": "bind",
                            "Source": "/var/run/docker.sock",
                            "Target": "/var/run/docker.sock"
                        }
                    ],
                    "Isolation": "default"
                },
                "Resources": {},
                "Placement": {
                    "Constraints": [
                        "node.id == ri03jp8udscfsddrwljj773e5"
                    ]
                },
                "Networks": [
                    {
                        "Target": "nuqyuts03ithqasl7aol5nyqq"
                    }
                ],
                "LogDriver": {
                    "Name": "json-file",
                    "Options": {
                        "max-size": "512m"
                    }
                },
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 3000,
                        "PublishedPort": 3000,
                        "PublishMode": "host"
                    }
                ]
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 3000,
                        "PublishedPort": 3000,
                        "PublishMode": "host"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 3000,
                    "PublishedPort": 3000,
                    "PublishMode": "host"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "nuqyuts03ithqasl7aol5nyqq",
                    "Addr": "10.0.1.2/24"
                }
            ]
        },
        "UpdateStatus": {
            "State": "completed",
            "StartedAt": "2021-10-05T09:51:22.836601272Z",
            "CompletedAt": "2021-10-05T09:51:50.433952543Z",
            "Message": "update completed"
        }
    }
]

root@vm1:~# docker network ls
NETWORK ID     NAME                      DRIVER    SCOPE
f5ba312731e4   bridge                    bridge    local
nuqyuts03ith   captain-overlay-network   overlay   swarm
20ea1054c510   docker_gwbridge           bridge    local
879bc5ee478f   host                      host      local
32w01r0pp1f6   ingress                   overlay   swarm
e89154989237   none                      null      local

root@vm1:~# docker network inspect captain-overlay-network   
[
    {
        "Name": "captain-overlay-network",
        "Id": "nuqyuts03ithqasl7aol5nyqq",
        "Created": "2021-10-05T09:51:35.368028878Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "3d4bbed9df26a639276421cdfb392d375d12558f4b7ece214efaf666b30a78d0": {
                "Name": "captain-nginx.1.lee0kahw8ds9a0m7eefr6kcro",
                "EndpointID": "2662c9be5c3253f5ee7196f0f2a4b5f336cd23e91f106dd2e3b2b4ea56c1331d",
                "MacAddress": "02:42:0a:00:01:49",
                "IPv4Address": "10.0.1.73/24",
                "IPv6Address": ""
            },
            "50c0ebc305bae8252e0cd03a93b6dda470ea2de5d5c3d0cfc555ffd4f666d075": {
                "Name": "captain-captain.1.9ktrn4limz0kmhc1w79m06rpa",
                "EndpointID": "f7313b3094249b6016792eb8ad470291927145e9cb5df67fd2d844d6f474deee",
                "MacAddress": "02:42:0a:00:01:48",
                "IPv4Address": "10.0.1.72/24",
                "IPv6Address": ""
            },
            "lb-captain-overlay-network": {
                "Name": "captain-overlay-network-endpoint",
                "EndpointID": "d34dbad56634e8cd89038d6f1e9156dda50611fe9bc270f3c0a75b19f420463f",
                "MacAddress": "02:42:0a:00:01:05",
                "IPv4Address": "10.0.1.5/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "c8fe65020ff6",
                "IP": "127.0.0.1"
            }
        ]
    }
]

root@vm1:~# docker service update --network-add captain-overlay-network captain-captain
service is already attached to network captain-overlay-network

# disabling the health check so containers stop restarting
root@vm1:~# echo  "{\"skipVerifyingDomains\":\"true\"}" >  /captain/data/config-override.json
root@vm1:~# docker service update captain-captain --force

root@vm1:~# docker network inspect captain-overlay-network
[
    {
        "Name": "captain-overlay-network",
        "Id": "nuqyuts03ithqasl7aol5nyqq",
        "Created": "2021-10-05T09:51:35.368028878Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "62d4710e2b36a9b705d9549d871e8c695e7b5739f6dd8b8677e95626d12c8566": {
                "Name": "captain-captain.1.lvixdsiml4pdomp11gsboftki",
                "EndpointID": "d84da8e34cafe9459f2f2d3d7f67203b5432ff879de60d186e5abf19344ad543",
                "MacAddress": "02:42:0a:00:01:4e",
                "IPv4Address": "10.0.1.78/24",
                "IPv6Address": ""
            },
            "73c10958da4856f811392d24c6f262218042a0a4c19ad9c424bdf78bb64d0703": {
                "Name": "captain-nginx.1.r9pgfn2of0p0k3nzk0zzaud5f",
                "EndpointID": "6b12ff57223894071963c33d7c04a8913c25bac1be0e23c20e2d711a7cb332c2",
                "MacAddress": "02:42:0a:00:01:4f",
                "IPv4Address": "10.0.1.79/24",
                "IPv6Address": ""
            },
            "lb-captain-overlay-network": {
                "Name": "captain-overlay-network-endpoint",
                "EndpointID": "d34dbad56634e8cd89038d6f1e9156dda50611fe9bc270f3c0a75b19f420463f",
                "MacAddress": "02:42:0a:00:01:05",
                "IPv4Address": "10.0.1.5/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "c8fe65020ff6",
                "IP": "127.0.0.1"
            }
        ]
    }
]

root@vm1:~# docker exec -it  captain-nginx.1.r9pgfn2of0p0k3nzk0zzaud5f /bin/bash
# curl the docker swarm service captain-captain VirtualIp from the Nginx container (both are in same network)
root@73c10958da48:/# curl http://10.0.1.2:3000 -v
* Expire in 0 ms for 6 (transfer 0x55a72a082ee0)
*   Trying 10.0.1.2...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55a72a082ee0)
* connect to 10.0.1.2 port 3000 failed: No route to host
* Failed to connect to 10.0.1.2 port 3000: No route to host
* Closing connection 0
curl: (7) Failed to connect to 10.0.1.2 port 3000: No route to host

# curl caprover regular container from the Nginx container 
root@73c10958da48:/# curl http://10.0.1.79 -v
* Expire in 0 ms for 6 (transfer 0x55bfe1e79ee0)
*   Trying 10.0.1.79...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55bfe1e79ee0)
* Connected to 10.0.1.79 (10.0.1.79) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.0.1.79
> User-Agent: curl/7.64.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 05 Oct 2021 11:29:43 GMT
< Content-Type: text/html
< Content-Length: 2401
< Last-Modified: Tue, 05 Oct 2021 11:21:41 GMT
< Connection: keep-alive
< ETag: "615c3545-961"
< Accept-Ranges: bytes
< 
<!doctype html>
<html>

<head>
    <meta charset=utf-8>
    <meta content="width=device-width,initial-scale=1.0,minimum-scale=1.0,maximum-scale=1.0,user-scalable=no" name=viewport>
    <title>Powered by CapRover</title>
    <style>
        html,
        body {
            font-family: sans-serif;
            -ms-text-size-adjust: 100%;
            -webkit-text-size-adjust: 100%;
            background-color: #F7F8FB;
            height: 100%;
            -webkit-font-smoothing: antialiased;
        }

        body {
            margin: 0;
            padding: 0;
            display: flex;
            flex-direction: column;
            align-items: center;
            justify-content: center;
        }

        .message {
            text-align: center;
            align-self: center;
            display: flex;
            flex-direction: column;
            align-items: center;
            padding: 20px;
            max-width: 550px;
        }

        .message__title {
            font-size: 32px;
            font-weight: 100;
            margin-top: 15px;
            color: #47494E;
            margin-bottom: 8px;
        }

        .btn {
            text-decoration: none;
            padding: 11px 25px;
            border-radius: 11px;
            margin-top: 110px;
            font-size: 19px;
            color: #7F828B;
        }

        body.defaultpagebody {
            background: -webkit-linear-gradient(-45deg, #1d5b85 0%, #3295c7 100%);
            background: linear-gradient(135deg, #1d5b85 0%, #3295c7 100%);
        }

        .message__title {
            color: #fff;
        }

        .message {
            padding: 25px;
            color: #fff;
        }

        body.defaultpagebody p {
            color: rgba(255, 255, 255, 0.6);
        }

        body.defaultpagebody .info {
            fill: rgba(255, 255, 255, 0.9);
        }

        body.defaultpagebody .btn {
            color: #fff;
            border: 2px solid rgba(255, 255, 255, 0.7);
        }
    </style>
    <base target=_parent />
</head>

<body class=defaultpagebody>
    <div class=message>
        <div class=message__title>
            Nothing here yet :/
        </div>
        <div class=message>

        </div>
        <a href="https://caprover.com/" target="_blank" rel="noopener noreferrer" class="btn">
            Read Docs</a>
    </div>
</body>

* Connection #0 to host 10.0.1.79 left intact

finally here is the full docker log which contains some errors https://gist.github.com/sameh-farouk/89a62aeca9c27607a50bb5de8e39bd22

sameh-farouk commented 3 years ago

Apparently fixed in https://github.com/threefoldtech/cloud-container/pull/2