search

gravitl / netmaker

Netmaker makes networks with WireGuard. Netmaker automates fast, secure, and distributed virtual networks.
https://netmaker.io
Other
9.56k stars 553 forks source link

Neclient (host machine) losed conectivity in all networks (This has happened multiple times) #2686

Open sicutdeux opened 1 year ago

sicutdeux commented 1 year ago

Contact Details

axzelmarin@gmail.com

What happened?

I've seen this hapening a multiple times in the past, the only fix for me was reinstall everything but I've a buch of network with almost 30 clients machines and don't want to go to the same process again, let me describe the issue to see if anyone here can help me since I've issues using Discord does not allow me to post questions and my axiety is through the roof with this:

I use CheckMK using the network that netmaker creates, 9h ago the checkmk host could not connect to the netmaker server, after rebooting both vm's the connection did not came back, on netclient on the nemaker server i get this error:

Nov 22 18:16:01 netmaker netclient[918998]: {"time":"2023-11-22T18:16:01.50537199Z","level":"ERROR","source":"mqhandlers.go 199}","msg":"error decrypting message","error":"could not decrypt message, [139 199 75 75 156 222 13 97 14 169 >

On my laptop I use wireguard client using a Client config that is made by the dashboard, from this host I cant reach any client machine in the network.

2023-11-22 13:24:23.092 [NET] peer(o9jA…vISs) - Sending handshake initiation

I keep getting this error on the wireguard

and in the dashboard and all the networks I've this:

image

which is the netmaker server that is not reachable on every of those networks.

I've attached last lines on the docker containers.

Version

v0.21.2

What OS are you using?

Linux

Relevant log output

Attaching to mq, coredns, netmaker-ui, turn, netmaker, caddy
caddy          | {"level":"error","ts":1700674945.1099956,"logger":"http.log.error","msg":"dial tcp 172.18.0.4:8883: connect: connection refused","request":{"remote_ip":"142.132.214.92","remote_port":"34796","proto":"HTTP/1.1","method":"GET","host":"broker.netmaker.axenxio.net","uri":"/","headers":{"User-Agent":["Go-http-client/1.1"],"Connection":["Upgrade"],"Sec-Websocket-Key":["pGJkgbB7hlrO7MWtaFcgtw=="],"Sec-Websocket-Protocol":["mqtt"],"Sec-Websocket-Version":["13"],"Upgrade":["websocket"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"broker.netmaker.axenxio.net"}},"duration":0.001156557,"status":502,"err_id":"fz85e0px6","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
caddy          | {"level":"error","ts":1700674946.390914,"logger":"http.log.error","msg":"dial tcp 172.18.0.4:8883: i/o timeout","request":{"remote_ip":"135.148.54.78","remote_port":"59756","proto":"HTTP/1.1","method":"GET","host":"broker.netmaker.axenxio.net","uri":"/","headers":{"Connection":["Upgrade"],"Sec-Websocket-Key":["4hGzfCzx1L0yF+du0zO+Og=="],"Sec-Websocket-Protocol":["mqtt"],"Sec-Websocket-Version":["13"],"Upgrade":["websocket"],"User-Agent":["Go-http-client/1.1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"broker.netmaker.axenxio.net"}},"duration":3.00144749,"status":502,"err_id":"4854tg02i","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
caddy          | {"level":"error","ts":1700674946.427677,"logger":"http.log.error","msg":"dial tcp 172.18.0.4:8883: i/o timeout","request":{"remote_ip":"147.135.68.126","remote_port":"40682","proto":"HTTP/1.1","method":"GET","host":"broker.netmaker.axenxio.net","uri":"/","headers":{"User-Agent":["Go-http-client/1.1"],"Connection":["Upgrade"],"Sec-Websocket-Key":["pnlthoBZG5nFbf24FnM19g=="],"Sec-Websocket-Protocol":["mqtt"],"Sec-Websocket-Version":["13"],"Upgrade":["websocket"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"broker.netmaker.axenxio.net"}},"duration":3.000626867,"status":502,"err_id":"s5yygue44","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
coredns        | [INFO] plugin/reload: Running configuration SHA512 = 7756a7dc8ca1f3bff79c30438b6f47e299df3244c7e5d9978507f7380aeee497c25205a60a8a1ee18ece118df5ef351e6365e14acf9cf2120fa4ffd783ff5ae2
coredns        | CoreDNS-1.10.1
coredns        | linux/amd64, go1.20, 055b2c3
netmaker       | {"time":"2023-11-22T18:29:57.208399091Z","level":"ERROR","source":"handlers.go 85}","msg":"failed to decrypt message for host","id":"d7efad04-7c7c-435d-8559-5235ca98ad7f","error":"could not decrypt message, [21 110 226 27 234 48 225 53 95 9 155 229 157 209 12 230 200 62 180 29 175 180 72 209 114 145 33 145 110 208 43 96 220 216 111 28 40 69 135 75 17 232 25 122 68 62 201 220 62 7 225 201 198 174 171 152 145 214 92 20 247 31 172 18 151 29 156 157 193 112 3 165 230 71 156 218 98 239 95 9 139 98 152 119 225 246 243 133 156 212 78 76 230 215 70 18 60 88 153 10 140 157 70 141 40 147 213 241 109 59 114 206 168 212 144 222 42 119 180 168 113 109 242 96 121 214 224 1 180 166 131 104 55 249 248 183 221 84 114 162 78 116 43 180 147 145 184 70 122 106 62 135 253 35 133 198 70 65 120 161 21 163 235 234 201 61 211 9 245 24 185 117 47 61 114 243 41 70 114 241 122 102 115 228 196 229 29 220 105 85 114 246 213 65 178 202 39 134 170 185 108 78 160 118 236 164 126 22 83 183 143 205 67 186 157 168 164 160 51 20 213 138 160 45 185 20 103 207 2 20 43 134 32 147 247 252 243 66 10 155 154 132 92 142 142 149 244 123 180 0 144 19 201 216 66 222 60 186 86 205 228 10 230 236 220 95 78 129 161 62 69 135 203 199 13 108 62 126 167 161 153 72 135 208 198 238 23 215 24 63 211 127 63 60 172 54 239 127 193 189 42 235 122 30 6 246 45 164 53 95 51 128 217 4 217 139 74 116 137 106 37 98 200 84 86 82 166 162 208 16 10 44 28 41 71 4 92 150 205 233 253 169 5 216 91 71 136 100 42 206 101 188 149 244 57 136 245 53 238 42 160 38 61 215 39 119 12 117 115 19 126 218 41 250 77 155 94 134 211 203 95 54 116 206 65 115 247 216 2 203 93 8 176 17 101 182 59 240 11 248 104 161 202 199 135 226 240 28 10 243 70 136 244 240 58 94 168 53 143 182 74 220 240 125 188 244 55 1 124 142 64 189 40 230 219 15 231 6 93 241 247 111 221 242 102 32 60 47 190 233 129 92 214 64 255 150 2 226 13 164 44 206 174 165 116 166 171 181 210 59 7 113 69 81 212 193 16 205 221 104 222 47 229 197 15 112 173 115 25 124 81 24 15 182 71 205 201 83 7 80 227 3 73 109 45 108 42 67 127 228 230 65 48 231 76 181 112 30 198 186 127 217 150 126 230 66 211 64 188 237 162 204 150 139 169 18 99 33 158 67 215 133 96 158 227 206 183 24 58 2 215 49 69 234 71 184 37 228 0 179 220 238 145 236 77 52 36 33 107 210 119 81 70 9 160 116 43 233 156 241 112 60 252 159 86 248 201 200 170 80 52 235 254 202 137 188 223 248 162 120 38 206 66 134 140 181 128 85 198 32 87 136 20 32 133 110 58 241 222 203 117 187 193 49 49 69 28 209 186 59 97 166 111 50 115 204 160 53 191 171 56 30 164 17 49 82 63 241 248 117 117 177 110 124 21 175 39 37 168 49 69 179 14 110 113 153 254 13 51 69 82 195 231 220 101 203 53 84 20 118 156 4 116 35 231 152 115 110 103 240 18 166 19 41 16 150 47 60 65 44 243 117 235 226 245 6 93 50 64 117 218 104 29 237 131 91 9 136 144 55 93 220 111 170 211 240 233 112 43 47 157 57 74 176 165 202 228 246 237 60 112 247 158 233 249 151 212 121 9 125 184 244 32 139 99 186 229 20 91 243 89 160 188 158 154 46 217 67 121 126 74 244 82 17 116 146 215 79 86 191 22 233 36 115 131 192 247 103 77 18 102 220 108 115 99 36 71 56 24 9 46 251 31 160 69 51 123 182 249 232 182 6 106 131 251 31 215 204 208 127 118 67 218 45 242 23 204 227 39 7 28 80 110 252 53 152 180 167 144 58 161 157 238 239 252 17 180 86 41 14 226 222 75 39 52 62 28 29 207 150 226 41 145 80 174 100 26 149 133 148 238 251 147 241 193 206 27 59 112 90 150 111 161 61 161 221 246 168 209 103 54 218 186 177 217 112 174 175 56 23 135 93 140 242 194 226 124 35 47 31 132 166 94 130 104 88 143 103 32 76 194 122 46 91 113 66 49 97 89 215 105 96 35 170 252 247 218 36 174 120 150 246 181 176 175 248 13 196 8 30 235 66 246 182 49 151 248 90 243 201 54 126 135 61 163 199 199 212 28 26 225 40 114 160 178 50 88 174 46 189 156 59 231 6 150 203 207 101 163 174 160 146 103 219 142 151 232 100 51 203 136 72 147 70 93 100 200 203 189 223 229 73 104 224 52 51 248 91 132 102 30 182 143 249 156 202 236 56 42 74 188 66 129 7 73 253 31 219 153 40 18 119 154 56 151 16 78 86 180 247 233 161 23 136 81 116 170 132 64 55 121 133 204 47 90 29 139 154 236 174 184 180 190 15 4 132 175 211 73 92 111 192 196 112 56 148 60 213 56 72 57 218 8 124 182 9 198 162 82 177 221 119 138 66 24 18 8 169 220 73 71 176 62 85 18 62 198 165 119 166 94 146 181 40 226 191 113 238 43 38 47 46 226 164 178 89 82 215 232 89 127 217 197 253 187 114 65 63 65 252 4 61 141 67 183 51 237 223 217 2 180 210 64 212 78 141 86 58 78 157 161 61 116 199 94 164 120 167 168 75 145 186 66 211 69 118 204 116 255 207 55 51 29 210 12 17 115 104 123 175 155 245 95 136 142 18 158 3 189 44 25 27 92 224 216 74 151 172 87 68 210 186 152 240 186 173 248 58 197 108 159 84 67 0 160 144 203 147 151 102 124 217 11 220 125 106 158 166 147 60 60 113 220 186 24 188 18 52 155 91 247 12 202 175 5 85 29 0 52 176 66 200 66 160 108 104 1 147 235 38 235 169 237 159 223 32 93 31 65 106 129 120 135 235 94 84 83 243 232 72 104 110 50 49 212 149 88 191 250 242 223 93 57 169 97 157 50 9 52 204 232 28 82 20 243 192 37 37 227 136 229 222 168 129 113 116 219 145 208 20 43 33 158 172 7 34 188 22 108 24 173 35 153 147 232 203 169 6 210 253 153 56 83 110 188 230 147 128 46 47 23 226 110 90 83 239 83 162 92 13 220 204 172 78 74 132 157 102 84 153 90 135 235 93 196 178 47 201 139 229 157 137 124 159 163 191 217 236 134 226 215 155 137 143 143 221 85 68 63 154 29 64 64 116 89 188 200 111 250 97 169 96 96 57 30 209 240 52 205 171 64 110 0 137 245 82 130 12 89 81 177 224 120 68 18 120 198 8 87 29 127 38 145 15 186 48 226 239 150 96 155 9 219 122 95 182 138 63 195 115 198 93 133 221 255 26 18 160 249 250 236 104 215 66 139 54 46 77 252 251 89 102 51 201 160 186 201 189 63 57 117 31 193 191 255 116 127 234 185 134 28 179 116 28 146 156 186 105 37 12 67 35 15 197 236 40 228 10 231 96 56 16 159 139 229 233 130 173 16 129 121 43 102 39 250 6 70 59 124 121 55 33 80 9 169 206 121 190 28 49 134 229 250 29 114 153 195 247 14 204 105 22 251 23 254 164 27 218 109 137 169 50 42 226 234 86 235 85 75 133 247 255 175 80 161 161 176 206 85 158 118 118 36 69 159 104 146 189 131 125 211 182 35 122 76 139 51 191 80 178 137 56 61 203 81 234 8 63 225 39 97 71 74 174 75 171 242 238 60 133 69 86 69 63 251 46 67 95 86 9 130 75 74 114 163 64 90 11 9 11 139 122 52 174 193 246 158 242 193 96 147 39 126 181 106 132 255 157 185 132 11 7 5 169 145 16 118 134 233 23 126 244 64 165 128 122 193 87 183 41 229 152 168 82 25 36 66 141 3 181 181 219 106 193 176 36 66 53 139 208 83 164 9 206 4 20 117 253 8 14 7 9 186 139 92 181 193 241 92 4 52 119 58 214 196 232 32 234 239 58 8 242 222 168 109 132 175 80 65 198 150 160 177 159 51 201 27 248 252 122 226 154 228 115 134 4 156 180 139 107 122 15 91 239 46 216 214 185 221 122 174 106 234 217 68 244 126 234 2 123 74 193 97 36 117 54 63 146 228 122 219 66 57 71 72 133 196 21 79 199 21 188 45 135 40 245 132 30 112 157 180 20 14 13 101 164 202 76 237 221 43 44 149 20 3 43 248 250 10 136 199 104 26 129 19 176 65 62 197 196 133 75 71 81 173 235 240 81 238 249 117 102 123 224 4 105 218 205 182 75 188 55 143 152 253 202 30 154 174 116 163 115 106 205 50 73 222 60 6 78 95 120 91 26 225 90 119 151 44 199 143 223 252 49 27 218 55 217 210 120 19 229 97 7 69 103 155 219 14 85 239 253 54 27 16 212 254 204 207 163 157 248 23 236 222 206 29 168 253 192 210 189 142 31 128 141 254 98 4 213 241 59 234 190 73 241 168 244 8 100 120 135 77 134 103 133 201 128 33 48 65 228 94 85 117 198 180 37 21 162 1 129 196 181 97 234 33 8 168 207 121 39 27 217 97 99 184 201 27 203 255 158 69 214 116 62 158 70 23 51 70 252 28 174 247 142 103 67 46 181 23 0 104 249 192 62 194 0 49 93 55 33 207 114 222 230 27 26 106 82 160 183 58 152 71 32 99 166 175 101 82 134 19 11 0 232 127 61 61 50 92 233 32 205 12 238 189 46 40 195 151 203 86 83 144 150 162 22 242 23 159 58 99 158 5 105 202 194 93 91 157 233 73 187 249 229 132 30 63 20 85 210 22 118 163 234 203 206 145 203 140 65 219 89 191 187 252 132 91 159 215 175 147 67 245 194 2 56 108 132 68 37 221 17 129 131 159 226 223 161 97 176 36 194 219 231 201 65 40 0 39 219 154 225 164 91 135 61 240 191 244 134 1 138 47 85 185 157 109 66 91 71 105 181 76 235 170 147 23 186 127 56 200 49 63 208 13 207 186 4 243 65 186 40 95 215 47 66 47 7 248 252 130 137 207 200 138 232 239 252 66 153 119 179 81 36 25 138 208 179 204 110 53 152 124 41 18 232 172 229 199 202 153 84 242 50 232 87 141 53 251 134 26 196 235 54 51 123 143 226 7 31 184 239 226 142 204 191 50 135 64 98 44 182 45 61 176 132 136 4 181 205 223 234 139 68 40 210 122 212 240 226 211 28 34 92 223 156 168 215 171 114 231 209 42 216 153 231 199 234 251 248 159 168 166 96 245 143 60 233 92 253 90 223 205 211 102 218 87 148 151 4 27 172 184 57 158 153 221 116 252 72 163 1 136 148 214 249 198 74 78 41 201 103 144 14 156 221 244 131 146 249 55 7 121 200 204 75 204 7 54 213 138 176 206 65 35 223 59 114 85 54 246 217 87 29 175 121 112 197 1 173 241 131 37 56 28 105 209 41 218 138 210 84 186 4 197 249 80 253 80 145 202 127 254 17 93 200 58 20 0 223 0 162 240 188 81 199 36 48 178 72 44 199 211 242 30 194 56 58 177 219 10 79 54 225 175 102 213 17 24 226 141 110 99 158 103 150 92 88 2 196 187 80 39 170 92 119 21 48 167 225 76 229 91 253 223 217 4 68 128 188 90 212 99 196 173 207 84 149 82 53 242 116 129 237 123 52 196 71 122 186 31 181 240 205 32 167 129 112 58 176 29 194 100 220 229 120 107 102 217 234 53 209 180 58 241 64 40 194 181 110 116 82 60 207 160 249 64 33 59 225 112 177 224 134 76 4 250 172 23 75 171 8 234 81 4 243 56 228 113 142 237 22 190 194 62 178 91 81 126 196 43 238 200 205 180 64 82 128 150 207 221 165 240 86 153 113 6 196 194 175 52 202 26 50 48 15 94 230 75 68 91 237 222 62 178 80 254 89 238 70 178 78 171 62 194 144 45 248 225 52 71 200 208 6 12 31 36 181 37 201 226 251 66 81 161 107 41 60 19 1 135 23 144 15 215 122 185 188 206 252 153 111 108 74 105 216 74 212 107 84 130 74 0 69 118 172 49 103 176 197 156 113 188 148 91 206 145 218 126 1 253 10 127 224 36 64 130 78 109 1 248 167 79 1 16 34 168 125 235 205 42 199 171 80 229 247 96 11 177 167 247 252 123 239 240 18 119 83 83 161 99 219 226 38 184 14 51 236 51 95 247 107 59 161 221 6 90 107 240 140 104 202 65 8 39 215 107 168 197 207 46 149 14 157 59 51 209 167 51 246 135 95 182 177 10 48 161 114 181 53 86 178 117 123 254 72 50 250 29 193 174 132 95 56 42 95 85 37 47 180 158 150 33 8 90 59 253 131 48 6 120 220 145 222 103 56 71 41 168 14 44 250 97 139 37 231 100 69 123 56 176 114 149 68 148 182 141 129 23 72 108 141 45 162 178 18 249 150 165 82 132 220 157 238 92 245 66 185 221 194 204 215 247 181 6 18 100 150 83 205 182 175 38 81 243 252 184 46 187 101 219 190 156 226 58 157 217 91 231 36 249 30 182 51 75 116 134 144 163 202 187 20 145 20 111 142 70 157 77 176 116 21 150 118 99 225 76 127 92 104 101 174 51 54 90 106 19 252 222 38 6 78 172 235 219 230 150 98 250 139 5 91 36 72 201 240 81 245 182 153 148 213 163 137 7 63 201 172 62 59 201 180 144 227 137 6 198 145 125 217 78 251 31 136 244 33 126 33 194 60 30 27 55 115 110 166 39 6 44 30 190 91 4 197 187 177 45 61 12 234 53 249 21 235 53 232 130 126 44 239 255 118 144 6 207 139 27 64 4 46 16 49 201 255 26 86 174 17 43 33 111 134 22 218 239 218 153 155 108 50 179 19 85 239 168 121 66 164 255 186 171 176 32 95 127 180 121 166 99 152 254 233 192 188 61 96 100 187 169 204 8 182 237 210 69 1 83 186 231 204 223 120 181 160 65 11 155 16 191 243 232 173 79 99 152 172 34 119 167 159 1 209 13 85 243 176 242 9 20 244 115 229 38 24 175 66 228 165 27 131 179 173 160 34 163 50 131 249 143 147 72 93 167 252 131 59 249 226 145 117 211 36 138 147 106 32 87 51 135 41 201 167 35 233 55 180 40 4 110 221 178 53 88 147 17 64 216 201 182 236 121 240 113 72 32 209 172 88 69 141 108 178 182 126 121 153 201 20 93 239 120 239 51 242 174 96 194 86 90 201 25 151 107 183 217 47 2 131 16 21 252 254 164 156 62 57 120 198 21 103 84 241 203 77 235 92 176 143 236 133 140 71 107 87 176 134 127 142 55 193 171 101 173 255 225 194 197 138 15 198 14 94 179 33 128 204 202 129 2 201 134 144]"}
netmaker       | {"time":"2023-11-22T18:30:57.212938596Z","level":"ERROR","source":"handlers.go 85}","msg":"failed to decrypt message for host","id":"d7efad04-7c7c-435d-8559-5235ca98ad7f","error":"could not decrypt message, [133 217 136 222 19 203 95 246 237 48 70 93 51 221 89 139 179 13 253 214 30 179 236 206 89 57 31 188 36 33 227 203 85 73 230 32 132 34 60 241 145 59 14 206 41 83 181 6 66 209 241 3 225 48 176 88 8 236 45 125 45 92 55 121 222 214 57 90 37 53 194 156 128 78 193 131 226 246 167 240 153 46 180 138 124 67 222 236 109 163 158 121 105 35 64 231 127 66 116 215 167 212 179 89 193 120 129 164 145 193 202 178 247 173 218 115 64 212 71 53 8 172 72 62 44 142 170 97 251 247 194 63 31 245 181 127 234 188 137 16 252 240 198 221 184 183 45 97 125 36 133 221 214 228 23 137 94 45 176 134 77 30 83 128 70 186 230 41 69 254 90 88 53 204 89 28 156 96 162 114 201 58 246 150 248 216 16 27 229 104 218 175 219 125 171 223 130 82 204 78 200 45 166 193 252 113 57 178 86 23 170 34 33 79 157 108 191 34 12 122 145 53 175 72 114 26 242 110 7 83 157 223 244 169 152 18 117 39 107 119 156 49 180 139 158 86 70 220 16 171 28 20 44 251 216 93 94 237 105 182 221 34 254 193 156 222 9 3 3 68 176 197 100 9 255 237 94 84 49 82 48 128 15 141 55 167 66 3 4 99 175 149 217 134 246 253 244 204 165 80 116 66 27 228 93 80 48 163 115 235 113 196 228 172 208 19 7 151 56 218 12 7 194 46 117 111 135 128 221 126 254 172 25 182 45 167 116 105 181 39 22 175 172 143 111 36 179 190 152 176 162 98 221 169 41 33 79 236 81 230 159 169 249 33 249 154 98 247 126 60 24 255 96 234 79 103 101 185 137 203 144 130 147 157 238 95 69 191 87 204 91 20 4 126 0 104 135 101 197 175 116 249 124 41 101 70 208 194 223 136 55 155 120 80 14 161 246 81 55 221 110 40 125 242 38 154 47 54 53 7 253 175 29 67 91 131 178 21 154 234 170 220 210 110 29 32 112 26 40 191 4 89 177 175 238 180 187 232 101 145 148 56 178 87 52 80 29 239 188 64 238 63 152 20 219 170 50 194 129 36 108 124 252 53 49 82 25 254 205 70 59 19 111 9 98 208 147 51 23 141 161 238 94 255 200 176 141 196 45 94 149 217 150 83 43 217 112 12 219 80 36 163 97 101 150 148 198 25 190 243 221 25 183 193 249 220 103 44 201 95 85 75 160 123 15 136 165 150 106 101 51 216 215 57 8 227 37 15 221 215 210 170 217 152 183 145 133 62 3 251 27 161 108 33 15 111 145 169 76 220 76 71 78 216 154 96 34 170 61 148 191 149 95 125 237 225 130 1 131 75 179 56 213 239 132 243 197 253 201 118 195 55 199 120 82 196 207 207 1 131 255 199 14 155 146 219 126 210 110 165 26 27 128 150 82 34 91 103 67 227 32 35 219 208 23 87 149 128 11 221 7 122 227 49 116 175 119 189 131 100 108 164 22 245 254 143 14 169 155 88 124 105 186 55 40 39 220 59 214 76 140 147 26 154 245 129 201 21 171 3 68 185 165 100 40 245 57 220 226 248 10 100 101 135 36 224 82 173 190 46 113 165 50 242 68 19 206 230 27 185 120 132 195 71 88 113 150 124 158 29 5 162 121 40 253 29 155 1 225 103 254 157 203 114 227 87 149 32 76 41 224 184 191 4 56 209 193 80 164 104 195 37 59 203 202 148 169 47 150 77 232 241 151 80 248 206 75 204 223 125 164 113 247 195 47 75 31 99 34 172 47 244 249 177 108 221 92 76 106 189 17 34 68 209 76 186 206 216 49 221 79 8 50 218 147 22 37 159 196 246 223 204 48 110 71 19 140 196 211 133 21 14 82 82 114 38 206 44 119 42 208 250 174 218 42 25 9 110 252 157 207 144 184 20 190 154 54 239 163 55 124 255 240 83 183 92 176 239 90 18 183 185 178 31 181 22 58 251 95 148 80 0 16 170 42 93 38 186 82 224 214 45 209 109 87 42 109 171 142 227 57 68 56 183 188 124 126 12 0 110 17 7 168 33 60 43 255 197 236 125 71 67 247 127 159 202 229 83 159 181 144 148 141 95 81 10 36 226 249 207 73 123 47 105 222 53 244 224 239 118 56 131 65 224 132 170 135 138 111 83 174 146 0 194 101 230 24 237 104 36 32 213 68 113 75 137 173 177 21 253 6 139 76 247 2 211 179 136 94 102 93 10 56 9 23 125 202 41 1 206 8 86 90 175 52 99 243 114 64 240 205 113 213 101 113 10 133 51 130 26 99 213 4 195 135 236 60 114 71 3 234 99 216 96 26 187 126 109 252 164 180 98 148 220 98 193 209 1 85 203 44 22 95 62 65 114 176 235 243 66 51 219 74 242 182 254 57 193 36 135 178 37 7 214 225 94 207 16 146 50 105 25 172 192 236 207 103 167 24 28 12 122 235 184 234 178 210 99 223 163 151 173 88 97 106 236 201 56 189 243 8 73 149 0 171 155 82 122 167 95 37 99 175 192 97 117 90 169 22 104 239 85 119 254 122 81 217 250 231 18 60 106 185 151 176 98 77 75 79 196 120 79 108 94 38 152 35 74 140 219 0 104 74 128 57 176 230 161 239 178 129 198 17 251 48 134 14 125 251 177 163 193 176 134 54 31 106 41 49 67 86 43 240 117 154 10 197 19 222 106 168 98 122 207 219 13 205 161 229 63 166 31 19 124 22 97 116 248 6 83 102 216 129 230 89 210 116 45 228 3 190 203 87 112 249 101 125 92 159 207 236 138 64 26 114 119 28 20 205 15 34 149 56 217 92 138 8 193 32 228 107 243 146 223 168 118 116 52 88 148 22 67 71 126 26 9 44 161 121 120 92 219 170 156 178 7 206 114 130 180 63 192 103 251 4 183 130 92 72 17 229 126 93 4 46 115 74 5 54 216 172 77 151 186 99 54 102 61 139 217 96 44 98 90 12 175 7 222 46 68 205 92 227 76 72 130 126 212 115 253 154 17 68 91 221 158 248 5 51 191 227 64 38 192 76 58 45 194 190 19 148 146 101 91 168 242 140 81 105 142 189 63 68 253 134 35 125 29 111 67 242 156 241 234 141 45 223 190 207 17 25 86 233 117 246 164 28 21 166 171 60 120 46 99 182 158 216 192 91 56 65 217 132 236 80 1 249 21 165 230 124 159 160 244 161 169 105 96 217 188 71 230 68 140 74 59 70 26 154 38 94 170 180 14 221 213 135 191 109 254 146 130 255 122 110 194 2 225 253 179 249 147 153 57 228 185 176 105 90 191 206 3 146 126 120 195 231 56 126 99 36 201 26 220 78 54 255 160 34 83 28 34 201 216 198 86 7 177 133 49 184 249 228 234 31 250 92 167 146 41 114 125 13 158 223 112 203 41 75 4 200 168 39 214 57 190 236 22 71 2 54 211 75 154 169 123 50 23 38 247 167 194 187 25 105 151 98 83 37 251 166 210 108 88 220 70 18 187 90 34 137 80 172 201 83 144 58 161 187 131 54 171 187 247 8 21 0 108 147 81 238 196 51 216 41 5 129 115 166 130 56 138 172 240 211 96 99 70 106 244 217 37 254 118 39 224 36 77 147 20 50 196 242 115 110 211 61 50 33 191 151 63 129 198 58 74 102 117 144 167 218 190 126 82 179 168 217 109 163 232 46 229 227 172 93 163 193 55 217 50 4 95 172 64 89 113 125 157 45 229 253 169 124 149 167 226 123 130 189 41 128 125 187 53 229 38 35 31 73 17 209 58 9 81 124 222 35 145 46 215 174 19 133 150 221 222 12 155 132 204 155 98 112 125 120 48 192 24 17 249 219 85 52 120 21 92 217 45 74 165 22 160 233 4 217 37 209 74 181 179 147 239 146 24 237 4 222 136 182 39 218 123 10 71 2 5 202 16 46 132 102 66 78 1 123 35 184 173 83 58 97 61 4 10 37 16 44 246 102 246 29 139 219 81 142 144 158 212 8 126 36 64 164 204 181 38 44 154 3 87 176 202 19 62 119 229 208 58 57 93 113 241 118 129 93 244 120 100 1 185 114 149 45 186 62 84 124 25 49 215 24 73 140 151 126 167 8 215 250 7 237 78 40 219 138 65 14 198 80 207 246 188 36 23 28 229 238 216 89 249 148 194 136 232 62 246 225 51 55 255 53 254 187 70 13 159 217 97 139 120 99 208 21 37 42 163 197 179 31 108 0 202 253 125 247 136 51 151 21 71 27 128 160 9 28 103 161 193 119 163 12 205 52 23 49 131 236 1 140 40 205 66 172 207 184 118 67 95 161 170 69 41 234 218 224 202 114 3 239 237 152 105 199 180 204 241 91 8 112 142 224 116 182 139 251 234 71 56 213 234 86 52 126 222 254 93 21 7 103 98 55 173 1 154 32 111 10 82 208 56 15 44 49 109 244 26 201 91 237 119 160 21 96 33 64 123 148 97 111 237 4 241 114 147 95 51 187 97 70 101 101 232 161 4 91 66 25 255 116 147 88 80 123 31 217 86 204 253 189 136 195 82 144 68 247 130 9 29 129 137 47 4 167 129 223 188 212 11 177 60 15 126 81 196 135 253 169 26 5 78 238 202 56 247 3 80 102 224 139 150 29 95 14 92 5 182 53 159 144 54 241 113 2 35 76 127 65 180 62 76 51 182 115 224 132 7 239 94 191 32 86 135 234 193 19 109 93 80 176 84 58 136 89 95 209 43 88 227 95 131 30 60 200 32 144 19 77 72 154 214 107 29 124 144 80 239 123 87 176 168 10 145 137 146 116 230 251 95 29 88 91 74 76 177 39 95 124 122 9 186 202 250 246 181 108 169 99 150 71 22 235 174 20 108 157 125 206 187 227 45 116 21 94 179 236 202 197 218 8 207 175 227 125 219 245 59 53 114 124 198 30 121 123 106 85 183 19 226 28 98 179 227 233 129 59 60 207 33 150 250 48 206 18 188 247 96 23 75 40 168 255 241 143 130 141 177 12 138 82 191 237 136 138 102 190 248 25 135 59 27 239 98 33 36 0 158 75 211 20 12 187 62 98 246 4 172 5 103 113 24 228 45 181 253 113 106 249 165 197 145 97 179 38 98 30 11 21 39 34 156 233 128 0 236 37 34 214 179 127 44 254 106 65 41 67 210 28 249 23 130 69 234 2 42 63 64 52 144 211 249 252 16 68 126 206 14 133 206 107 123 183 132 247 170 83 187 210 214 75 206 25 84 204 252 188 141 102 18 151 181 170 252 157 223 40 63 201 233 153 171 145 214 51 218 117 181 255 198 95 61 226 212 140 109 30 16 41 27 147 47 219 152 39 133 51 104 30 185 209 192 231 119 180 187 45 89 199 161 138 82 1 26 103 12 73 37 194 44 9 54 197 210 100 16 73 90 201 143 23 70 122 163 228 129 122 39 2 164 111 62 21 64 62 33 13 232 81 218 132 32 145 176 117 109 209 207 40 45 118 212 201 70 202 218 16 163 89 19 236 156 137 119 164 252 101 41 6 137 74 186 236 185 218 32 88 223 10 75 73 202 28 119 195 103 203 157 81 9 171 169 203 243 199 174 137 168 118 39 237 159 103 144 133 12 162 59 138 61 202 45 71 66 196 246 185 163 5 24 223 18 80 241 121 126 66 189 235 172 224 211 62 133 206 5 33 112 155 72 209 147 252 223 218 57 121 188 71 27 84 42 40 169 172 236 226 34 96 112 32 238 4 249 100 52 96 110 210 57 162 192 190 213 228 9 111 175 126 223 154 237 93 64 55 195 86 53 232 8 252 189 215 139 56 40 231 80 194 183 41 225 108 177 222 30 69 246 201 152 62 91 9 200 125 124 21 159 171 88 48 124 198 57 185 171 242 223 189 26 62 31 216 213 160 63 55 49 52 236 194 207 125 25 93 58 109 53 240 66 98 75 250 119 104 91 102 50 32 219 233 62 149 117 148 193 59 29 229 178 237 184 26 233 127 95 132 13 32 112 251 76 155 108 108 135 61 160 20 202 57 44 74 213 34 178 92 132 147 79 207 226 233 135 216 96 59 166 109 4 113 118 214 16 39 154 158 159 43 2 48 20 207 121 206 252 69 21 243 188 156 233 165 233 112 22 217 211 137 66 156 172 10 227 252 144 96 49 195 96 139 184 59 180 0 227 239 62 219 174 113 218 65 181 199 212 190 233 219 10 83 85 32 121 17 30 21 132 28 232 137 120 174 246 33 110 227 41 209 222 207 183 97 150 175 84 105 64 139 218 3 179 71 155 11 6 77 117 113 234 177 91 36 42 100 242 13 106 162 126 34 182 60 203 138 52 162 172 222 226 90 98 132 210 230 74 136 171 160 192 149 192 50 216 80 64 71 7 140 5 105 250 92 226 14 167 92 216 95 251 192 195 10 161 191 135 47 52 59 42 247 196 175 206 46 107 216 211 54 209 210 239 118 105 51 246 117 152 65 190 15 172 204 243 99 199 178 142 177 94 113 202 192 173 12 245 51 30 216 56 38 184 119 8 86 102 47 42 50 13 153 207 11 89 103 132 154 155 200 131 208 11 148 135 118 203 151 127 96 20 99 76 83 136 99 15 15 235 64 214 35 254 105 153 20 142 27 192 74 57 81 22 170 67 22 123 87 169 76 197 218 66 41 90 136 172 68 198 161 160 223 65 128 229 151 230 230 244 95 145 250 192 206 17 61 206 52 115 164 23 224 191 202 213 243 238 73 22 128 157 219 241 191 144 12 194 205 34 57 39 221 217 69 245 137 238 228 172 23 16 222 217 169 226 86 136 201 172 125 191 125 7 44 199 166 20 102 226 230 179 207 47 184 121 157 158 13 129 249 85 127 30 83 116 13 162 203 194 129 168 21 242 189 216 155 185 113 38 137 118 252 162 97 154 237 1 180 237 129 57 213 184 85 239 15 197 56 172 35 118 170 59 152 223 158 51 97 56 83 88 152 110 244 173 144 17 75 183 172 162 163 94 112 25 255 141 235 21 99 51 8 129 184 99 117 198 31 194 131 130 185 152 61 136 94 86 144 30 171 110 229 129 238 52 161 130 212 227 249 163 4 116 214 227 12 192 250 231 45 17 154 11 21 87 3 254 157 80 148 238 69 52 151 60 9 87 245 193 87 122 195 35 159 18 149 96 130 45 75 214 21 204 123 124 106 37 46 174 45 71 0 15 119 190 101 131 11 138 80 112 173 163 83 27 238 194 57 108 217 92 241 104 188 78 223 90 249 81 236 12 155 61 247 48 180 2 159 222 98 150 42 199 198 50 1 152 194 210 120 195 199 135 65 72 138 206 42 22 172 163 128 66 62 28 38 79]"}
netmaker       | {"time":"2023-11-22T18:31:57.210624856Z","level":"ERROR","source":"handlers.go 85}","msg":"failed to decrypt message for host","id":"d7efad04-7c7c-435d-8559-5235ca98ad7f","error":"could not decrypt message, [203 236 96 7 237 153 158 124 242 152 152 255 232 136 51 241 120 237 112 72 27 240 166 151 248 216 206 154 142 77 101 142 101 11 130 217 202 50 206 90 133 161 54 249 2 0 19 86 153 106 140 211 149 59 246 248 112 44 150 190 33 108 58 10 193 232 233 232 17 193 232 9 55 120 253 34 199 30 182 87 121 93 180 147 112 240 174 203 30 101 137 79 103 115 127 49 236 176 152 244 183 255 204 250 229 75 41 13 124 91 230 167 108 93 50 36 226 37 172 95 236 139 61 236 201 198 255 208 134 31 105 155 10 114 191 121 119 65 203 224 94 210 112 27 196 21 112 91 239 187 183 239 49 74 41 163 76 6 120 221 10 108 118 97 89 60 181 238 72 119 229 220 118 230 102 30 179 240 50 59 109 87 71 82 23 131 8 126 250 251 161 46 42 119 232 237 239 242 183 149 130 38 139 152 11 150 203 52 7 200 211 125 223 6 2 151 168 54 131 146 245 200 15 158 127 224 151 146 105 229 100 125 237 241 42 115 102 50 49 239 100 176 96 220 98 135 87 234 226 252 140 156 161 6 29 66 223 234 140 246 215 50 138 207 208 234 242 86 21 234 137 0 115 111 122 27 11 59 84 85 208 45 110 93 251 55 19 6 230 127 66 2 181 168 52 66 108 61 203 94 63 21 211 96 244 234 83 196 44 63 54 82 28 183 31 198 56 250 202 40 188 142 184 202 232 79 206 23 191 59 53 239 82 128 66 215 227 109 252 231 0 221 185 36 225 154 170 241 186 38 114 164 123 196 126 211 253 10 212 26 126 75 7 84 102 63 241 100 48 52 65 129 29 57 183 132 67 118 230 23 255 184 187 167 63 47 131 248 30 86 194 252 138 174 152 87 173 164 253 52 205 141 174 47 213 50 101 51 70 29 3 11 236 128 151 99 251 80 67 32 60 56 119 8 76 172 153 121 20 138 227 243 101 93 100 121 173 78 20 51 193 97 214 152 159 239 17 124 246 150 131 153 115 201 144 76 91 155 237 119 215 226 93 151 59 219 246 142 190 113 153 29 252 202 147 198 176 12 79 23 163 164 177 237 88 206 78 130 221 188 190 151 196 96 112 75 144 104 11 216 254 9 92 135 62 123 238 240 8 18 121 76 187 155 137 20 78 68 132 148 99 95 55 132 105 62 19 145 250 216 17 249 172 149 11 60 129 100 53 82 86 27 241 168 234 137 69 177 166 109 173 153 142 48 7 191 62 153 250 199 145 15 85 58 82 224 132 38 193 86 239 25 21 111 66 231 161 220 254 234 116 166 165 246 134 6 50 241 174 229 202 134 241 187 254 245 228 254 238 92 111 229 172 94 242 105 23 220 16 189 149 5 12 4 126 22 174 49 213 21 143 168 118 163 191 9 177 252 59 141 22 22 84 243 200 235 147 97 84 7 70 76 9 70 138 117 65 206 116 43 222 243 89 205 198 201 127 82 126 156 137 55 194 207 234 58 71 56 45 80 15 167 70 200 65 31 159 243 242 47 40 201 71 29 196 55 204 184 254 220 214 180 34 142 8 102 100 251 189 232 71 148 46 109 194 31 116 175 14 242 204 100 157 245 94 91 151 182 62 103 162 58 60 107 244 252 0 168 86 17 35 164 23 129 88 137 217 26 17 52 27 194 7 97 150 173 105 247 146 95 74 49 36 15 109 125 239 145 204 2 159 21 67 6 5 215 118 27 101 128 43 89 238 148 141 171 150 86 87 31 252 89 214 60 1 17 21 65 180 113 115 151 83 155 104 163 225 137 130 54 223 230 3 125 173 138 25 72 14 154 38 243 57 104 20 37 136 228 174 157 49 228 65 48 231 200 83 53 32 79 119 187 102 0 11 86 123 5 50 82 197 240 185 70 130 37 219 30 87 108 213 81 38 38 241 128 40 126 121 100 156 225 78 145 9 187 128 99 251 131 99 1 118 166 72 35 111 245 11 157 107 214 156 146 26 60 204 61 21 175 7 25 137 94 30 235 43 194 159 150 41 92 127 160 37 180 206 224 30 40 60 180 220 139 99 11 183 72 7 207 236 32 242 243 215 238 7 53 241 117 225 88 13 166 240 140 87 108 82 187 75 15 163 82 53 67 68 122 63 169 2 191 244 132 233 254 232 134 105 218 187 191 41 213 103 116 172 172 200 217 58 96 132 133 78 91 93 9 215 102 25 193 252 149 70 196 105 177 181 105 105 26 227 185 81 9 50 169 147 155 81 71 67 57 62 108 55 207 65 96 70 110 28 85 60 206 30 175 161 150 115 38 151 71 63 135 64 233 134 82 54 66 239 229 220 182 203 133 1 226 26 8 18 28 79 82 59 37 158 182 121 100 217 185 21 210 191 116 222 52 114 177 115 244 43 102 87 173 161 223 24 41 225 193 37 164 194 135 248 36 23 235 159 253 141 84 46 108 8 151 209 174 71 172 61 43 87 41 255 88 94 168 161 134 74 251 36 148 89 176 228 99 77 68 15 231 136 227 166 143 247 223 112 124 251 214 114 67 20 51 111 76 249 156 32 202 52 56 194 211 66 184 92 23 233 13 224 251 195 124 127 106 211 144 66 155 18 101 248 110 23 243 175 161 234 236 84 42 20 241 48 155 77 56 190 177 60 146 18 109 124 35 239 112 244 133 59 35 164 151 93 254 43 241 129 102 52 217 39 96 201 96 89 42 48 37 200 101 29 139 97 36 69 13 198 138 228 1 140 85 11 32 41 97 41 139 131 135 55 231 69 27 209 166 114 227 202 82 165 199 58 231 61 26 214 70 105 191 211 194 211 133 112 88 146 52 79 213 251 27 143 195 78 78 75 56 146 176 110 216 225 81 156 147 156 147 31 89 150 184 165 204 192 99 134 64 38 54 190 229 19 114 202 182 73 187 223 100 5 130 92 252 164 130 74 15 71 167 170 134 54 42 127 244 36 14 19 249 74 132 90 254 123 51 171 191 171 158 251 179 29 170 33 194 219 221 119 161 36 64 83 253 134 111 110 194 140 147 135 21 26 106 146 207 65 194 198 239 83 98 35 95 117 81 24 36 57 197 141 149 59 6 40 101 76 12 83 64 214 252 3 21 147 44 59 103 223 146 122 158 177 113 151 212 213 156 112 186 167 82 246 238 68 53 30 87 10 182 155 18 12 253 56 92 56 144 55 251 249 178 81 243 0 228 234 209 35 31 105 203 87 63 93 215 91 121 151 229 10 239 153 27 230 149 100 86 200 235 140 145 64 96 153 66 12 154 211 16 73 56 211 75 69 129 152 245 226 27 142 92 17 230 159 76 71 153 165 73 227 168 114 164 203 252 6 59 144 92 221 192 34 199 146 163 108 36 132 205 107 117 165 204 6 51 104 127 112 101 42 30 138 64 76 88 236 161 11 34 46 76 184 237 79 75 34 237 47 144 60 205 175 219 103 78 200 228 27 72 235 103 187 104 166 27 102 168 89 155 185 185 53 26 230 62 130 160 71 25 69 51 186 150 165 97 15 82 164 70 97 71 129 181 81 211 237 48 117 144 251 129 236 190 149 165 122 23 94 219 125 174 152 17 147 243 150 6 181 17 10 116 26 18 123 188 129 182 151 113 110 170 62 158 242 155 89 27 0 123 229 72 43 36 140 136 100 86 94 140 237 96 138 111 108 91 107 237 234 27 63 165 50 244 10 62 64 62 76 151 67 124 246 2 161 101 190 84 235 203 8 143 93 122 20 66 122 220 61 22 97 74 199 160 203 110 161 54 115 238 245 186 67 190 75 96 17 29 78 43 163 86 44 107 173 31 134 222 74 216 35 131 252 230 182 89 197 81 237 162 184 28 7 212 78 226 137 95 106 157 254 47 162 213 152 68 112 116 200 142 125 45 225 86 167 135 5 32 80 166 138 213 203 34 2 208 100 172 128 248 152 182 7 58 201 205 129 102 143 89 42 25 122 3 38 77 86 25 78 7 39 194 134 145 74 94 230 86 143 28 196 45 121 137 208 239 74 238 138 227 51 42 130 13 42 88 242 78 244 101 223 47 73 163 11 176 86 188 237 60 71 102 37 136 4 222 132 203 27 97 206 133 198 187 249 251 173 120 117 248 24 107 29 104 100 170 95 168 136 53 236 197 226 75 207 171 19 76 58 72 204 167 26 81 64 115 224 68 218 158 141 204 61 232 136 15 31 184 243 156 38 222 146 232 31 202 168 18 159 206 72 200 177 77 176 142 146 105 155 241 73 130 213 66 180 24 138 73 224 238 41 100 244 11 36 173 249 85 81 119 127 252 175 160 27 84 178 56 71 167 136 74 176 25 114 249 73 172 127 237 173 96 68 142 53 202 242 55 162 80 16 225 182 224 97 66 36 181 181 23 115 73 249 47 45 15 116 31 187 149 186 6 206 146 206 83 221 174 102 195 126 174 182 105 172 144 151 3 13 161 69 168 91 237 4 245 35 117 191 100 223 210 54 124 8 60 231 101 230 117 43 29 149 194 53 207 191 9 90 216 223 109 205 24 82 28 205 136 111 212 22 115 140 125 65 65 97 108 194 7 64 139 230 86 15 94 128 128 138 120 189 240 124 88 101 49 93 191 51 128 20 61 173 35 71 251 131 0 157 171 163 212 162 82 157 180 97 181 227 88 163 40 102 18 78 165 212 194 63 234 126 24 87 161 144 44 55 199 126 118 170 71 173 77 192 226 190 37 100 138 104 196 58 241 125 88 112 168 217 217 11 153 171 10 68 163 113 128 198 234 190 184 42 123 118 105 206 242 23 222 128 101 52 32 139 92 233 79 253 1 248 3 22 168 189 147 231 92 18 81 31 10 109 163 230 131 85 137 123 208 127 137 179 249 241 158 208 209 169 14 128 172 230 48 99 8 179 147 19 1 159 85 159 30 242 196 74 80 130 8 176 149 152 26 66 187 115 135 126 89 44 94 162 19 236 97 112 250 159 73 170 21 232 179 67 138 157 244 240 96 216 13 254 205 87 106 95 138 88 0 81 145 201 28 185 121 136 222 40 157 100 28 91 230 94 188 136 228 120 36 229 136 158 175 152 15 181 84 12 237 113 40 208 137 8 227 23 69 91 172 2 37 29 13 113 161 71 133 43 168 52 183 166 64 23 225 11 11 22 143 185 101 110 40 250 150 171 45 158 97 210 244 106 72 208 135 51 105 25 221 107 140 86 193 193 1 157 63 39 218 152 52 32 192 244 37 91 218 126 44 253 133 245 246 61 9 161 198 232 235 230 20 21 20 141 145 155 250 118 127 71 188 90 36 73 233 104 17 255 224 71 171 241 18 209 116 123 244 230 32 140 251 4 54 207 173 119 103 38 255 50 165 15 79 232 59 7 175 240 51 229 194 251 0 213 102 194 164 34 144 141 102 80 133 21 4 123 84 206 79 244 101 23 158 81 248 144 115 96 20 232 128 107 8 117 71 95 254 80 64 10 124 101 123 249 251 218 12 192 148 157 40 33 16 188 166 117 45 248 42 86 213 109 82 69 239 98 229 48 22 59 61 155 32 213 155 5 153 185 197 244 31 63 114 77 224 78 184 154 2 147 64 245 113 162 23 68 191 154 187 250 223 105 81 154 53 78 250 234 173 3 202 116 44 80 158 181 231 188 12 182 110 77 167 162 208 82 210 151 95 118 120 160 8 1 158 81 22 182 213 114 76 162 149 112 25 100 102 83 148 150 214 31 165 196 253 127 146 35 126 99 96 179 148 163 56 237 245 11 87 112 81 77 18 118 6 86 94 255 152 170 254 121 106 78 181 111 111 97 180 34 180 207 219 209 233 69 26 15 176 70 74 228 140 97 73 39 239 127 50 226 216 108 103 50 111 73 150 89 215 176 145 68 129 247 35 111 172 76 191 59 8 158 73 253 192 169 76 40 135 205 212 181 176 31 8 109 162 225 20 57 115 8 153 180 11 250 192 111 68 85 38 29 88 179 33 29 26 208 116 196 11 201 34 255 176 113 250 16 4 209 30 225 116 144 56 72 60 138 115 179 206 14 51 170 136 109 5 166 217 220 70 140 136 38 66 122 196 159 254 17 140 191 166 106 181 46 208 245 79 173 36 252 223 127 18 1 61 235 115 80 101 8 74 115 145 222 245 162 71 24 0 145 17 32 44 0 128 156 174 113 121 198 105 231 167 14 237 227 222 163 60 156 189 45 181 13 131 185 132 78 107 241 188 103 182 85 95 129 44 101 91 34 90 204 99 68 250 154 46 25 117 83 106 240 66 157 96 172 102 254 141 92 194 31 231 126 90 20 167 68 39 105 136 164 185 24 2 181 202 164 229 2 247 91 68 184 10 183 70 246 217 182 17 80 230 74 133 131 84 20 93 248 126 6 171 10 125 22 63 85 67 144 197 111 248 225 132 82 85 190 48 89 68 168 7 195 183 224 218 234 26 217 93 122 32 15 140 187 118 110 0 254 188 229 91 153 21 42 136 78 252 180 210 57 240 116 207 100 142 28 74 120 76 50 58 209 114 236 197 159 47 236 246 205 132 182 224 149 199 54 187 68 68 101 132 143 198 235 101 71 123 15 13 143 65 223 245 231 16 188 252 94 94 178 3 82 205 195 36 233 201 201 71 103 182 49 168 239 38 192 160 20 30 171 255 63 235 183 214 7 216 91 187 163 137 6 183 140 100 251 66 18 98 76 168 240 26 60 142 226 181 233 41 245 82 245 163 66 100 22 31 222 3 250 23 176 7 125 85 19 40 98 9 159 45 69 205 234 193 165 128 245 223 250 198 87 93 168 101 158 10 165 52 204 48 246 116 164 166 206 78 36 102 82 50 148 218 3 117 124 137 209 225 142 36 2 173 172 193 143 32 246 6 137 18 212 21 253 181 242 215 110 23 39 10 130 107 214 197 240 181 34 134 206 92 236 78 81 78 6 26 154 148 219 155 122 116 98 32 134 112 135 34 126 38 194 127 37 194 77 28 172 116 158 188 216 242 27 26 113 218 192 29 139 134 65 142 218 137 229 221 84 60 34 1 87 164 114 213 209 169 193 110 235 137 143 166 59 114 174 103 172 88 45 175 107 242 252 249 196 173 230 21 19 56 224 183 57 201 43 211 112 59 7 165 103 102 56 175 198 102 33 89 27 88 179 98 178 108 201 67 124 94 240 252 188 173 212 78 162 108 204 132 156 18 72 217 45 106 106 201 44 199 177 155 156 213 7 15 166 131 249 225 228 119 236 90 223 169 215 128 134 9 41 241 21 19 230 122 3 157 76 85 117 186 206 247 175 108 82 48 192 83 109 96 237 153 168 147 125 97 191 78 234 111 98 202 51 50 107 68 121 133 178 30]"}
mq             | 1700676574: New client connected from 69.30.231.99:0 as d7efad04-7c7c-435d-8559-5235ca98ad7f (p2, c1, k10, u'netmaker').
mq             | 1700676938: Client d7efad04-7c7c-435d-8559-5235ca98ad7f disconnected.
mq             | 1700676957: New client connected from 69.30.231.99:0 as d7efad04-7c7c-435d-8559-5235ca98ad7f (p2, c1, k10, u'netmaker').
netmaker-ui    | 172.18.0.6 - - [22/Nov/2023:18:17:08 +0000] "GET /nmui-config.js HTTP/1.1" 304 0 "https://dashboard.netmaker.axenxio.net/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36" "181.131.240.188"
netmaker-ui    | 172.18.0.6 - - [22/Nov/2023:18:17:24 +0000] "GET /icons/warning.svg HTTP/1.1" 200 564 "https://dashboard.netmaker.axenxio.net/networks/axenxio" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36" "181.131.240.188"
netmaker-ui    | 172.18.0.6 - - [22/Nov/2023:18:22:51 +0000] "GET /wp-login.php HTTP/1.1" 200 1233 "http://dashboard.netmaker.axenxio.net/wp-login.php" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/95.0" "184.168.117.19"
turn           | turn ERROR: 2023/11/22 18:32:16 error when handling datagram: failed to handle ChannelBind-request from 181.131.240.188:1866: no allocation found 181.131.240.188:1866:[::]:3479
turn           | turn ERROR: 2023/11/22 18:32:16 error when handling datagram: failed to handle ChannelBind-request from 181.131.240.188:1866: no allocation found 181.131.240.188:1866:[::]:3479
turn           | turn ERROR: 2023/11/22 18:32:17 error when handling datagram: failed to handle ChannelBind-request from 181.131.240.188:1866: no allocation found 181.131.240.188:1866:[::]:3479

Contributing guidelines

cocoonkid commented 5 months ago

I encountered the same issue today and a 3 weeks ago. on different hosts. (but otherwiese identical config (ansible)) only completely resetting the netclient and re-registering solved it.

image

netclient 24.1

abhishek9686 commented 5 months ago

I encountered the same issue today and a 3 weeks ago. on different hosts. (but otherwiese identical config (ansible)) only completely resetting the netclient and re-registering solved it.

image

netclient 24.1

@cocoonkid is this a self-hosted setup or SaaS?

cocoonkid commented 5 months ago

selfhosted. outside of 2 running docker containers the systems are Identical Rocky Linux 9.3. Identical because setup with ansible. I will update all of them today to the latest netclient too.

abhishek9686 commented 5 months ago

I encountered the same issue today and a 3 weeks ago. on different hosts. (but otherwiese identical config (ansible)) only completely resetting the netclient and re-registering solved it.

image

netclient 24.1

can you share the complete log file? you can send it to me over discord, if that's fine

cocoonkid commented 5 months ago

I would do but I upgraded the netmaker host to 24.2 and all clients and the OS itself fully so docker got restarted. So no serverside logs.

The client I rm'ed /etc/netclient and enrolled anew with fresh IP to the network it was supposed to live in.

Journalctl had the service logs still. DM'ed you now.

cocoonkid commented 4 months ago

It happened again today, again on an random client.. Whole network is on latest netclient and nm-server.

I DM @abhishek9686 about it. But same error again.

{"time":"2024-07-03T17:43:48.913852722Z","level":"ERROR","source":"handlers.go 95}","msg":"failed to decrypt message for host","id":"da00efc7-baa0-4ee7-b79f-2819aea88e28","name":"starry-fahrenheit","error":"could not decrypt message, [49 14

Here is a screenshot from syslog when it suddenly starts going weird. netclient] 2024-07-03 00:25:07 error publishing checkin cannot publish ... Mqclient not connected --> this one happens all the time every few minutes so that is expected and the tunnels stilll work. But then it died for real.

image

RobertLukan commented 4 months ago

I am getting the same issue out of the blue. I was running on 0.23 for a while. I have upgraded to 0.24.3. But issue is still there. It is limited only to certain hosts which is quite odd.

I have manually upgraded the affected endpoints and now looks better, lets see if it stays stable.

abhishek9686 commented 4 months ago

I am getting the same issue out of the blue. I was running on 0.23 for a while. I have upgraded to 0.24.3. But issue is still there. It is limited only to certain hosts which is quite odd.

we couldn't get to the bottom of this behaviour yet. By any chance is there any resource scarce on this machine where this issue has been observed? Also only remediation if you encounter this behaviour is to delete the host and re-join the network.

RobertLukan commented 4 months ago

I dont think so. Could it be some certificate expiring and not getting renewed ? I have not checked exactly how certificates are used for MQTT/API.

cocoonkid commented 4 months ago

we couldn't get to the bottom of this behaviour yet. By any chance is there any resource scarce on this machine where this issue has been observed? Also only remediation if you encounter this behaviour is to delete the host and re-join the network.

In my case I have all servers under zabbix monitoring so ressources are clearly not an issue. They are all under 20% load in average.

Upgraded all servers to 0.24.3 today and continue to watch for symptoms. Currently I have a ping running targting the netmaker server and if that ping fails it executes on the clients:

netclient leave <network>

netclient join -t <code>

(Originally it also did an rm -rf /etc/netclient but I found this unnecessary.)

I found that this also changes the IP sometimes when doing this via systemd. When doing it manually the server keeps its wireguard IP. Is there a way to set it to static dhcp so i can make sure the servers keep their IP?

I don't seem to get consistent behaivour..very weird.

This is the script I am using currently to fix a failed tunnel (and adjust the zabbix monitoring IP accordingly IF it changed.)


import subprocess
import logging
import signal
import sys
import time

TUNNEL_TEST_HOSTNAME = "'<netmaker-wireguard-ip address>"
TARGET_NETWORK = '<name-of-network>'
ENROLLMENT_CODE = '<enrollment-code>'
DNS_SERVER = '<netmaker-wireguard-ip address>'

CONFIG_FILE = "/etc/zabbix/zabbix_agent2.conf"

hostname = subprocess.check_output("hostname", shell=True).decode().strip()
WIREGUARD_HOSTNAME = hostname + ".customer_endpoints"

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

def check_ip(TUNNEL_TEST_HOSTNAME):
    """Check if the target IP is responding."""
    try:
        ip_address = resolve_dns(TUNNEL_TEST_HOSTNAME)
        if not ip_address:
            logging.error(f"DNS resolution failed for {TUNNEL_TEST_HOSTNAME}")
            return False
        time.sleep(1)
        logging.info(f"Resolved {TUNNEL_TEST_HOSTNAME} to {ip_address}")
        output = subprocess.check_output(["ping", "-c", "3", "-w", "3", ip_address], stderr=subprocess.STDOUT)
        time.sleep(1)
        logging.info(f"Ping output: {output.decode().strip()}")
        return True
    except subprocess.CalledProcessError as e:
        logging.error(f"Ping failed: {e.output.decode().strip()}")
        return False
    except Exception as e:
        logging.error(f"Unexpected error when checking IP: {e}")
        return False

def leave_network():
    """Leave the network using netclient."""
    logging.info("Leaving the network...")
    try:
        subprocess.check_output(["netclient", "leave", TARGET_NETWORK], stderr=subprocess.STDOUT)
    except subprocess.CalledProcessError as e:
        logging.error(f"Failed to leave network: {e.output.decode().strip()}")
    except Exception as e:
        logging.error(f"Unexpected error when leaving network: {e}")

def join_network():
    """Join the network using netclient."""
    logging.info("Rejoining the network...")
    try:
        subprocess.check_output(["netclient", "join", "-t", ENROLLMENT_CODE], stderr=subprocess.STDOUT)
    except subprocess.CalledProcessError as e:
        logging.error(f"Failed to join network: {e.output.decode().strip()}")
    except Exception as e:
        logging.error(f"Unexpected error when joining network: {e}")

def signal_handler(sig, frame):
    logging.info('Received termination signal. Exiting...')
    sys.exit(0)

def resolve_dns(name):
    try:
        output = subprocess.check_output(["dig", "+short", name, f"@{DNS_SERVER}"], stderr=subprocess.STDOUT, timeout=5)
        logging.debug(f"dig output: {output.decode().strip()}")
        ip_address = output.decode().strip()
        if ip_address:
            return ip_address
    except subprocess.TimeoutExpired:
        logging.error("dig command timed out")
    except subprocess.CalledProcessError as e:
        logging.error(f"dig command failed: {e.output.decode().strip()}")
    except Exception as e:
        logging.error(f"Unexpected error in resolve_dns: {e}")
    return None

def get_current_listen_ip():
    with open(CONFIG_FILE, 'r') as file:
        for line in file:
            if line.startswith("ListenIP="):
                return line.strip().split('=')[1]
    return None

def update_listen_ip(new_ip):
    with open(CONFIG_FILE, 'r') as file:
        lines = file.readlines()

    with open(CONFIG_FILE, 'w') as file:
        for line in lines:
            if line.startswith("ListenIP="):
                file.write(f"ListenIP={new_ip}\n")
            else:
                file.write(line)

def restart_zabbix_agent():
    subprocess.run(["systemctl", "restart", "zabbix-agent2.service"], check=True)

def zabbix_ip_watcher():
    new_ip = resolve_dns(WIREGUARD_HOSTNAME)
    print(new_ip)
    if new_ip:
        current_ip = get_current_listen_ip()
        if new_ip != current_ip:
            update_listen_ip(new_ip)
            logging.info(f"Changed {current_ip} to {new_ip}")
            restart_zabbix_agent()
            logging.info("Zabbix Agent restarted with new ListenIP")
        logging.info("zabbix_ip_watcher executed no change.")

def wireguard_ip_watcher():
    """Main watchdog function."""
    signal.signal(signal.SIGINT, signal_handler)
    signal.signal(signal.SIGTERM, signal_handler)

    if check_ip(TUNNEL_TEST_HOSTNAME):
        logging.info(f"{TUNNEL_TEST_HOSTNAME} is responsive")
    else:
        logging.warning(f"{TUNNEL_TEST_HOSTNAME} is not responsive. Reconnecting network.")
        leave_network()
        time.sleep(1)
        join_network()
        time.sleep(8)
        if check_ip(TUNNEL_TEST_HOSTNAME):
            logging.info(f"{TUNNEL_TEST_HOSTNAME} is responsive AGAIN")
            zabbix_ip_watcher()

if __name__ == "__main__":
    wireguard_ip_watcher()

It runs as systemd service templated via ansible jinja. A timer is running that executess it every minute.

[Unit]
Description=WireGuard IP Watcher Service
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /manifests/{{ inventory_hostname }}/configuration/wireguard_ip_watcher/wireguard_ip_watcher.py
User=root
{% if customer_endpoint | default(false) %}
Environment=WIREGUARD_HOSTNAME={{inventory_hostname}}.endpoints
{% endif %}

[Install]
WantedBy=multi-user.target

The really weird thing so far is I have the symptoms only in one of three networks so far.

But the servers are ALL setup in their base with the same ansible playbook so it should happen everywhere..

All servers now log with verbosity 4.

I"ll catch the next fail with additional logs and update here.

jonasasx commented 4 months ago

I have the same issue. I noticed some peer's ports are changed randomly when "could not decrypt message" error appears. Command netclient pull restores connection.

yabinma commented 3 months ago

In the changes recently(including v0.24.3), some of the communication from netclient to server changed from mq message to restful api call. It's unlikely there will be "could not decrypt message" issue happened in Netmaker server side. Please keep an eye on it.

In the context, the "could not decrypt message" issue happened in client side as reported. Unfortunately, I did not re-produce it yet. I added more debug log for further investigation. That would be helpful if sharing more information when the issue happened again in netclient side. When it occurs, please check the netclient.yml and servers.yml files in /etc/netclient/, to check if the traffickey and traffickeyprivate are right in the place.

yabinma commented 3 months ago

code fixed added for the decrypt issue on client side in v0.25.0. Please test and verify.

cocoonkid commented 2 months ago

Updated all nodes and will report. I had no real time to grep the logs the last weeks. unbelievable, sorry for that.