[Bug]: Installation Breaks after reboot

[ds141414]

Contact Details

ds1414@outlook.com

What happened?

A bug happened!

Hello, I must have run the installation procedure five times and I always get the same error. Everything runs smoothly during the installation process and I am able to log in the web UI. At this stage I am able to create my networks and connect using Netclient as well as Wireguard. Once the server is rebooted I have the following errors in the UI and the system does not route traffic. Especially when I try to access are network behind an egress gateway. This is what I get on the UI

This is a brand new server on Digital Ocean. I tried installing on different providers but the issue is still the same. I also changed the wild card domain every time to keep the certificates fresh.

Your insights would be much appreciated.

Version

v0.20.0

What OS are you using?

Linux

Relevant log output

I looked at the docker log and I get this: 

[netmaker] 2023-05-27 10:25:01 connecting to sqlite
[netmaker] 2023-05-27 10:25:01 database successfully connected
[netmaker] 2023-05-27 10:25:02 no OAuth provider found or not configured, continuing without OAuth
[netmaker] 2023-05-27 10:25:02 netmaker-stun listening on 0.0.0.0:3478 via udp
[netmaker] 2023-05-27 10:25:02 REST Server successfully started on port  8081  (REST)
[netmaker] 2023-05-27 10:25:02 connecting to mq broker at ws://mq:1883
[netmaker] 2023-05-27 10:25:04 connected to MQ Broker
[netmaker] 2023-05-27 10:25:13 masteradministrator failed to create network:  could not find any records
[netmaker] 2023-05-27 10:25:13 processed request error: could not find any records
[netmaker] 2023-05-27 10:25:42 received registration attempt with token eyJzZXJ2ZXIiOiJhcGkudm5ldC5kbW1pLnh5eiIsInZhbHVlIjoiUFRONVRFNUk3RkZTVk9NVlBCTFBNUVFGTE5KSTVOVjMifQ==
[netmaker] 2023-05-27 10:25:45 failed to register host with turn server:  httpclient: json error
[netmaker] 2023-05-27 10:25:48 failed to register host with turn server:  httpclient: json error
2023/05/27 10:25:48 checking default proxy {true false}
2023/05/27 10:25:49 set proxy enabled to  false
[netmaker] 2023-05-27 10:25:49 netmaker 5743d879-d20a-4f2f-a75b-a168914dc203 registered with Netmaker
[netmaker] 2023-05-27 10:25:49 added new node 1903d093-a292-4dd4-8a05-228a6489b859 to host netmaker
[netmaker] 2023-05-27 10:25:49 peer update for host 5743d879-d20a-4f2f-a75b-a168914dc203
[netmaker] 2023-05-27 10:25:50 peer update for host 5743d879-d20a-4f2f-a75b-a168914dc203
[netmaker] 2023-05-27 10:25:50 error retrieving extclients could not find any records
[netmaker] 2023-05-27 10:25:50 error retrieving custom dns entries could not find any records
[netmaker] 2023-05-27 10:25:50 peer update for host 5743d879-d20a-4f2f-a75b-a168914dc203
[netmaker] 2023-05-27 10:25:53 failed to register host with turn server:  httpclient: json error
[netmaker] 2023-05-27 10:25:55 masteradministrator created ingress gateway on node 1903d093-a292-4dd4-8a05-228a6489b859 on network netmaker
[netmaker] 2023-05-27 10:26:17 processed request error: unauthorized
[netmaker] 2023-05-27 10:26:23 failed to register host with turn server:  httpclient: json error
[netmaker] 2023-05-27 10:26:23 peer update for host 5743d879-d20a-4f2f-a75b-a168914dc203
[netmaker] 2023-05-27 10:26:23 sent peer updates after signal received from 1903d093-a292-4dd4-8a05-228a6489b859
[netmaker] 2023-05-27 10:26:35 superuser was made a new admin
[netmaker] 2023-05-27 10:27:31 shut down STUN server
[netmaker] 2023-05-27 10:27:31 Message Queue shutting down
[netmaker] 2023-05-27 10:27:31 Stopping the REST server...
[netmaker] 2023-05-27 10:27:31 http: Server closed
[netmaker] 2023-05-27 10:27:31 REST Server closed.

Contributing guidelines

• [X] Yes, I did.

[christian-schlichtherle]

I've encountered the same error. The workaround is to run sudo -i docker-compose up -d after reboot. This will start the netmaker and turn services again. I believe the root cause is that in /root/docker-compose.yaml these services are missing restart: always.

[ds141414]

Hello! I was able to start by installing and running Portainer. However this is not a solution because a restart might happen for other reasons than specifically intended.

I was also having issues with routing. I can ping all the nodes that are created but when I define an egress gateway to access let's say my home network, the gateway node does not route to the egress node ip and hence cannot reach the internal network.

[pete1019]

I've encountered the same error. The workaround is to run sudo -i docker-compose up -d after reboot. This will start the netmaker and turn services again. I believe the root cause is that in /root/docker-compose.yaml these services are missing restart: always.

True, turn and netmaker need "restart: always" in docker-compose.yml Don't know if this will be fixed in next release.

This should help: https://github.com/gravitl/netmaker/pull/2394