[Bug]: wait.sh stuck in installation.

PeterWang-dev commented 1 year ago

Contact Details

PeterWang030908@hotmail.com

What happened?

Seems there is something wrong when trying to creating ingress during installation.

After testing and digging into log, I found:

The curl inquiry response always got a null.
Logs of Caddy revealed an 502 error of api and broker access.
Mq logs said it was waiting for netmaker sever set up.

So it seems like something wrong in mq initialize.

Then I found error occured in server availability check, that is, the wait.sh. _curl --output /dev/null --silent --fail --head --location "${NETMAKER_SERVER_HOST}/api/server/health could not get response so mq did not start up!_

I checked logs of Caddy again and found the 502 Error said connection refused and had the message "err_trace":"reverseproxy.statusError (reverseproxy.go:1272).

Details of log are pasted in output section.

Version

v0.17.1

What OS are you using?

Linux

Relevant log output

caddy          | {"level":"info","ts":1676526319.114452,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"api.nm.38-55-96-217.nip.io"}
caddy          | {"level":"info","ts":1676526319.114867,"logger":"tls.obtain","msg":"releasing lock","identifier":"api.nm.38-55-96-217.nip.io"}
caddy          | {"level":"error","ts":1676526344.3769405,"logger":"http.log.error","msg":"dial tcp 172.27.0.3:8081: connect: connection refused","request":{"remote_ip":"38.55.96.217","remote_port":"36832","proto":"HTTP/2.0","method":"HEAD","host":"api.nm.38-55-96-217.nip.io","uri":"/","headers":{"User-Agent":["curl/7.81.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.nm.38-55-96-217.nip.io"}},"duration":0.55579573,"status":502,"err_id":"h8tkjzg5z","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
caddy          | {"level":"error","ts":1676526385.9281342,"logger":"http.log.error","msg":"dial tcp 172.27.0.6:8883: connect: connection refused","request":{"remote_ip":"54.242.107.231","remote_port":"35052","proto":"HTTP/1.1","method":"GET","host":"broker.nm.38-55-96-217.nip.io","uri":"/.git/config","headers":{"User-Agent":["'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.39 (KHTML, like Gecko) Chrome/89.0.4389.111 Safari/537.39'"],"Accept-Charset":["utf-8"],"Accept-Encoding":["gzip"],"Connection":["close"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"broker.nm.38-55-96-217.nip.io"}},"duration":0.002217655,"status":502,"err_id":"9y9xkbwjm","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
mq             | fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
mq             | (1/4) Installing brotli-libs (1.0.9-r6)
mq             | (2/4) Installing nghttp2-libs (1.47.0-r0)
mq             | (3/4) Installing libcurl (7.83.1-r6)
mq             | (4/4) Installing curl (7.83.1-r6)
mq             | Executing busybox-1.35.0-r17.trigger
mq             | OK: 8 MiB in 20 packages
mq             | SERVER: https://api.nm.38-55-96-217.nip.io
mq             | Waiting for netmaker server to startup
netmaker       |               
netmaker       |  __   __     ______     ______   __    __     ______     __  __     ______     ______    
netmaker       | /\ "-.\ \   /\  ___\   /\__  _\ /\ "-./  \   /\  __ \   /\ \/ /    /\  ___\   /\  == \   
netmaker       | \ \ \-.  \  \ \  __\   \/_/\ \/ \ \ \-./\ \  \ \  __ \  \ \  _"-.  \ \  __\   \ \  __<   
netmaker       |  \ \_\\"\_\  \ \_____\    \ \_\  \ \_\ \ \_\  \ \_\ \_\  \ \_\ \_\  \ \_____\  \ \_\ \_\ 
netmaker       |   \/_/ \/_/   \/_____/     \/_/   \/_/  \/_/   \/_/\/_/   \/_/\/_/   \/_____/   \/_/ /_/ 
netmaker       |                                                                                                                                                                             
netmaker       | 
netmaker       | [netmaker] 2023-02-16 05:44:35 connecting to sqlite 
netmaker       | [netmaker] 2023-02-16 05:44:36 database successfully connected 
netmaker       | [netmaker] 2023-02-16 05:44:37 no OAuth provider found or not configured, continuing without OAuth 
netmaker       | [netmaker] 2023-02-16 05:44:37 Configuring MQ... 
netmaker       | [netmaker] 2023-02-16 05:44:37 REST Server successfully started on port  8081  (REST) 
netmaker       | [netmaker] 2023-02-16 05:44:38 connecting to mq broker at ws://mq:1883 with TLS? false 
netmaker       | [netmaker] Fatal: Admin: could not connect to broker, token timeout, exiting ...

Contributing guidelines

[X] Yes, I did.

PeterWang-dev commented 1 year ago

I am not familiar with details about computer network and netmaker inner work. However I still have some guess about the bug:

I tested curl command separately on my machine out of docker and worked well. But it failed in docker container. That my related to some network (port forwarding) error between docker container?

I used curl intending to get from dashboard but nothing response while domain ping worked well.

PeterWang-dev commented 1 year ago

Solved by reappend firewall rules.

Mydreamandreality commented 1 year ago

通过重新追加防火墙规则解决。

Hello, I also encountered the same problem. What firewall rules did you add? Can you explain it in detail? Thank you

gravitl / netmaker