Open Sxndxr opened 11 months ago
Hi. Almost identical similar issue here. Just to add some additional context of my case that may be helpful:
I have been trying to debug the error but I have had no luck so far, any help or discoveries on the matter would be greatly appreciated if shared.
Thanks!
Could you please check if there are any error logs available at /var/log/netbird/client.log
and /var/log/netbird/netbird.err
?
Here you go, these are the logs from the client that is having the issues mentioned. As far as I have seen, the other machines (clients that work properly and the management server, don't have these logs).
Checking timestamps, the line that aligns perfectly with the moment in which the issues have started is this one:
2023-08-03T10:50:03Z ERRO client/internal/wgproxy/factory_linux.go:15: failed to initialize ebpf proxy: failed to set memlock rlimit: operation not permitted
@gmatasba @Sxndxr the issue was fixed in v0.22.2, can you test on your end?
I had a similar issue with netbird client v0.21.8:
netbird up --management-url <YOUR_MANAGEMENT_URL> --setup-key <YOUR_SETUP_KEY>
logs
2023-08-14T16:01:13+02:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 936.119771ms due to error rpc error: code = DeadlineExceeded desc = context deadline exceeded 2023-08-14T16:01:24+02:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 2.102444321s due to error rpc error: code = DeadlineExceeded desc = context deadline exceeded Error: login backoff cycle failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Updating the client to v0.22.4 fixed it for me. Yay.
I have exactly the same Problem on one (I have several) Manjaro Linux computer with the latest netbird Client from AUR (not docker).
Tried to remove the package and reinstalled it, but had no luck so far.
Any idea what causes this problem?
@mlsmaycon I personally solved the issue by creating a very small separate machine in order to use it as a routing peer and no longer have the set-up I described earlier. I will try to replicate it, test with the new version and then share the results here.
Since today a second machine fails to connect. When I execute "netbird up", I get:
2023-08-22T14:52:40+02:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 502.410121ms due to error rpc error: code = Unknown desc = getting pkce authorization flow info failed with error: context deadline exceeded Error: login backoff cycle failed: rpc error: code = Unknown desc = getting pkce authorization flow info failed with error: context deadline exceeded
Daemon version: 0.22.6 CLI version: 0.22.6
What is going on?
Getting the same error on v0.22.7 with Debian 12 LXC. Interesting enough I was able to get it connected but after restarting the container that's when the errors started popping up and even started locking up the LXC container every 20 secs and dropping my SSH connection.
I think you have to update Netbird. I guess that the bug has been fixed in the new version.
Le mer. 30 août 2023 à 05:03, Trevin Corkery @.***> a écrit :
Getting the same error on v0.22.7 with Debian 12 LXC. Interesting enough I was able to get it connected but after restarting the container that's when the errors started popping up and even started locking up the LXC container every 20 secs and dropping my SSH connection.
— Reply to this email directly, view it on GitHub https://github.com/netbirdio/netbird/issues/1048#issuecomment-1698418127, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUQXX7WG3S23PMVNL6LVCNTXX2URVANCNFSM6AAAAAA3CPJA34 . You are receiving this because you were mentioned.Message ID: @.***>
Same here with newest version:
Exactly same when --management-url provided
I still have the problem on two machines and the latest versions (0.22.7).
The issue might be a problem with gRPC connectivity. Could you run the agent in the foreground with the following steps and share the output?
First, you need to stop the current agent:
sudo netbird service stop
Then run:
sudo bash -c 'GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info netbird up -F -l debug'
@mlsmaycon Here is the log that I get. I just try to install and run netbird on my 2 machines. The desktop runs normally while the laptop encounters this issue. Both have the same network configurations.
// Edit: My bad. Turn out on my laptop, I didn't link stub-resolv.conf to /etc/resolv.conf. It's now working normally.
I still have this issue with the latest releases.
sudo bash -c 'GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info netbird up -F -l debug'
2023-09-19T09:17:12+02:00 DEBG client/internal/login.go:93: connecting to the Management service https://api.wiretrustee.com:443
2023/09/19 09:17:12 INFO: [core] [Channel #1] Channel created
2023/09/19 09:17:12 INFO: [core] [Channel #1] original dial target is: "api.wiretrustee.com:443"
2023/09/19 09:17:12 INFO: [core] [Channel #1] parsed dial target is: {Scheme:api.wiretrustee.com Authority: URL:{Scheme:api.wiretrustee.com Opaque:443 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/19 09:17:12 INFO: [core] [Channel #1] fallback to scheme "passthrough"
2023/09/19 09:17:12 INFO: [core] [Channel #1] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/api.wiretrustee.com:443 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/19 09:17:12 INFO: [core] [Channel #1] Channel authority set to "api.wiretrustee.com:443"
2023/09/19 09:17:12 INFO: [core] [Channel #1] Resolver state updated: {
"Addresses": [
{
"Addr": "api.wiretrustee.com:443",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2023/09/19 09:17:12 INFO: [core] [Channel #1] Channel switches to new LB policy "pick_first"
2023/09/19 09:17:12 INFO: [core] [Channel #1 SubChannel #2] Subchannel created
2023/09/19 09:17:12 INFO: [core] [Channel #1] Channel Connectivity change to CONNECTING
2023/09/19 09:17:12 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING
2023/09/19 09:17:12 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "api.wiretrustee.com:443" to connect
2023/09/19 09:17:12 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0006da0a8, {CONNECTING
Any ideas?
I updated netbird client from 0.22.7 to 0.23.2 on my CentOS 7.9 host and have got the same issue:
2023-09-22T18:43:01+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-09-22T18:43:02+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023-09-22T18:43:02+03:00 ERRO client/server/server.go:124: failed login: context deadline exceeded
At the same time there isn't this issue on fedora 37 with 0.23.2
UPDATE: reinstall netbird client 0.22.7 which worked well before and have got the same issue DEBUG:
sudo bash -c 'GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info netbird up -F -l debug'
2023-09-25T10:46:57+03:00 DEBG client/internal/login.go:93: connecting to the Management service https://netbird.hidden.tech:33073
2023/09/25 10:46:57 INFO: [core] [Channel #1] Channel created
2023/09/25 10:46:57 INFO: [core] [Channel #1] original dial target is: "netbird.hidden.tech:33073"
2023/09/25 10:46:57 INFO: [core] [Channel #1] parsed dial target is: {Scheme:netbird.hidden.tech Authority: URL:{Scheme:netbird.hidden.tech Opaque:33073 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:46:57 INFO: [core] [Channel #1] fallback to scheme "passthrough"
2023/09/25 10:46:57 INFO: [core] [Channel #1] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/netbird.hidden.tech:33073 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:46:57 INFO: [core] [Channel #1] Channel authority set to "netbird.hidden.tech:33073"
2023/09/25 10:46:57 INFO: [core] [Channel #1] Resolver state updated: {
"Addresses": [
{
"Addr": "netbird.hidden.tech:33073",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2023/09/25 10:46:57 INFO: [core] [Channel #1] Channel switches to new LB policy "pick_first"
2023/09/25 10:46:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel created
2023/09/25 10:46:57 INFO: [core] [Channel #1] Channel Connectivity change to CONNECTING
2023/09/25 10:46:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING
2023/09/25 10:46:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "netbird.hidden.tech:33073" to connect
2023/09/25 10:46:57 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc000010ba0, {CONNECTING <nil>}
2023/09/25 10:47:02 INFO: [core] [Channel #1] Channel Connectivity change to SHUTDOWN
2023/09/25 10:47:02 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to SHUTDOWN
2023/09/25 10:47:02 INFO: [core] [Channel #1 SubChannel #2] Subchannel deleted
2023/09/25 10:47:02 INFO: [core] [Channel #1] Channel deleted
2023-09-25T10:47:02+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-09-25T10:47:02+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023/09/25 10:47:02 INFO: [core] Creating new client transport to "{\n \"Addr\": \"netbird.hidden.tech:33073\",\n \"ServerName\": \"netbird.hidden.tech:33073\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}": connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023/09/25 10:47:02 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "netbird.hidden.tech:33073",
"ServerName": "netbird.hidden.tech:33073",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023-09-25T10:47:02+03:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 781.98239ms due to error context deadline exceeded
2023-09-25T10:47:03+03:00 DEBG client/internal/login.go:93: connecting to the Management service https://netbird.hidden.tech:33073
2023/09/25 10:47:03 INFO: [core] [Channel #3] Channel created
2023/09/25 10:47:03 INFO: [core] [Channel #3] original dial target is: "netbird.hidden.tech:33073"
2023/09/25 10:47:03 INFO: [core] [Channel #3] parsed dial target is: {Scheme:netbird.hidden.tech Authority: URL:{Scheme:netbird.hidden.tech Opaque:33073 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:03 INFO: [core] [Channel #3] fallback to scheme "passthrough"
2023/09/25 10:47:03 INFO: [core] [Channel #3] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/netbird.hidden.tech:33073 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:03 INFO: [core] [Channel #3] Channel authority set to "netbird.hidden.tech:33073"
2023/09/25 10:47:03 INFO: [core] [Channel #3] Resolver state updated: {
"Addresses": [
{
"Addr": "netbird.hidden.tech:33073",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2023/09/25 10:47:03 INFO: [core] [Channel #3] Channel switches to new LB policy "pick_first"
2023/09/25 10:47:03 INFO: [core] [Channel #3 SubChannel #4] Subchannel created
2023/09/25 10:47:03 INFO: [core] [Channel #3] Channel Connectivity change to CONNECTING
2023/09/25 10:47:03 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING
2023/09/25 10:47:03 INFO: [core] [Channel #3 SubChannel #4] Subchannel picks a new address "netbird.hidden.tech:33073" to connect
2023/09/25 10:47:03 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0001aa078, {CONNECTING <nil>}
2023/09/25 10:47:08 INFO: [core] [Channel #3] Channel Connectivity change to SHUTDOWN
2023/09/25 10:47:08 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to SHUTDOWN
2023/09/25 10:47:08 INFO: [core] [Channel #3 SubChannel #4] Subchannel deleted
2023/09/25 10:47:08 INFO: [core] [Channel #3] Channel deleted
2023-09-25T10:47:08+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-09-25T10:47:08+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023/09/25 10:47:08 INFO: [core] Creating new client transport to "{\n \"Addr\": \"netbird.hidden.tech:33073\",\n \"ServerName\": \"netbird.hidden.tech:33073\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}": connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023-09-25T10:47:08+03:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 1.993228132s due to error context deadline exceeded
2023/09/25 10:47:08 WARNING: [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "netbird.hidden.tech:33073",
"ServerName": "netbird.hidden.tech:33073",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023-09-25T10:47:10+03:00 DEBG client/internal/login.go:93: connecting to the Management service https://netbird.hidden.tech:33073
2023/09/25 10:47:10 INFO: [core] [Channel #5] Channel created
2023/09/25 10:47:10 INFO: [core] [Channel #5] original dial target is: "netbird.hidden.tech:33073"
2023/09/25 10:47:10 INFO: [core] [Channel #5] parsed dial target is: {Scheme:netbird.hidden.tech Authority: URL:{Scheme:netbird.hidden.tech Opaque:33073 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:10 INFO: [core] [Channel #5] fallback to scheme "passthrough"
2023/09/25 10:47:10 INFO: [core] [Channel #5] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/netbird.hidden.tech:33073 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:10 INFO: [core] [Channel #5] Channel authority set to "netbird.hidden.tech:33073"
2023/09/25 10:47:10 INFO: [core] [Channel #5] Resolver state updated: {
"Addresses": [
{
"Addr": "netbird.hidden.tech:33073",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2023/09/25 10:47:10 INFO: [core] [Channel #5] Channel switches to new LB policy "pick_first"
2023/09/25 10:47:10 INFO: [core] [Channel #5 SubChannel #6] Subchannel created
2023/09/25 10:47:10 INFO: [core] [Channel #5] Channel Connectivity change to CONNECTING
2023/09/25 10:47:10 INFO: [core] [Channel #5 SubChannel #6] Subchannel Connectivity change to CONNECTING
2023/09/25 10:47:10 INFO: [core] [Channel #5 SubChannel #6] Subchannel picks a new address "netbird.hidden.tech:33073" to connect
2023/09/25 10:47:10 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0001aa408, {CONNECTING <nil>}
2023/09/25 10:47:15 INFO: [core] [Channel #5] Channel Connectivity change to SHUTDOWN
2023/09/25 10:47:15 INFO: [core] [Channel #5 SubChannel #6] Subchannel Connectivity change to SHUTDOWN
2023/09/25 10:47:15 INFO: [core] [Channel #5 SubChannel #6] Subchannel deleted
2023/09/25 10:47:15 INFO: [core] [Channel #5] Channel deleted
2023-09-25T10:47:15+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-09-25T10:47:15+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023/09/25 10:47:15 INFO: [core] Creating new client transport to "{\n \"Addr\": \"netbird.hidden.tech:33073\",\n \"ServerName\": \"netbird.hidden.tech:33073\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}": connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023/09/25 10:47:15 WARNING: [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {
"Addr": "netbird.hidden.tech:33073",
"ServerName": "netbird.hidden.tech:33073",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023-09-25T10:47:15+03:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 2.225361164s due to error context deadline exceeded
2023-09-25T10:47:17+03:00 DEBG client/internal/login.go:93: connecting to the Management service https://netbird.hidden.tech:33073
2023/09/25 10:47:17 INFO: [core] [Channel #7] Channel created
2023/09/25 10:47:17 INFO: [core] [Channel #7] original dial target is: "netbird.hidden.tech:33073"
2023/09/25 10:47:17 INFO: [core] [Channel #7] parsed dial target is: {Scheme:netbird.hidden.tech Authority: URL:{Scheme:netbird.hidden.tech Opaque:33073 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:17 INFO: [core] [Channel #7] fallback to scheme "passthrough"
2023/09/25 10:47:17 INFO: [core] [Channel #7] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/netbird.hidden.tech:33073 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:17 INFO: [core] [Channel #7] Channel authority set to "netbird.hidden.tech:33073"
2023/09/25 10:47:17 INFO: [core] [Channel #7] Resolver state updated: {
"Addresses": [
{
"Addr": "netbird.hidden.tech:33073",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2023/09/25 10:47:17 INFO: [core] [Channel #7] Channel switches to new LB policy "pick_first"
2023/09/25 10:47:17 INFO: [core] [Channel #7 SubChannel #8] Subchannel created
2023/09/25 10:47:17 INFO: [core] [Channel #7] Channel Connectivity change to CONNECTING
2023/09/25 10:47:17 INFO: [core] [Channel #7 SubChannel #8] Subchannel Connectivity change to CONNECTING
2023/09/25 10:47:17 INFO: [core] [Channel #7 SubChannel #8] Subchannel picks a new address "netbird.hidden.tech:33073" to connect
2023/09/25 10:47:17 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0006660a8, {CONNECTING <nil>}
2023/09/25 10:47:22 INFO: [core] [Channel #7] Channel Connectivity change to SHUTDOWN
2023/09/25 10:47:22 INFO: [core] [Channel #7 SubChannel #8] Subchannel Connectivity change to SHUTDOWN
2023/09/25 10:47:22 INFO: [core] [Channel #7 SubChannel #8] Subchannel deleted
2023/09/25 10:47:22 INFO: [core] [Channel #7] Channel deleted
2023-09-25T10:47:22+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-09-25T10:47:22+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023-09-25T10:47:22+03:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 1.731637762s due to error context deadline exceeded
2023/09/25 10:47:22 INFO: [core] Creating new client transport to "{\n \"Addr\": \"netbird.hidden.tech:33073\",\n \"ServerName\": \"netbird.hidden.tech:33073\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}": connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023/09/25 10:47:22 WARNING: [core] [Channel #7 SubChannel #8] grpc: addrConn.createTransport failed to connect to {
"Addr": "netbird.hidden.tech:33073",
"ServerName": "netbird.hidden.tech:33073",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023-09-25T10:47:24+03:00 DEBG client/internal/login.go:93: connecting to the Management service https://netbird.hidden.tech:33073
2023/09/25 10:47:24 INFO: [core] [Channel #9] Channel created
2023/09/25 10:47:24 INFO: [core] [Channel #9] original dial target is: "netbird.hidden.tech:33073"
2023/09/25 10:47:24 INFO: [core] [Channel #9] parsed dial target is: {Scheme:netbird.hidden.tech Authority: URL:{Scheme:netbird.hidden.tech Opaque:33073 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:24 INFO: [core] [Channel #9] fallback to scheme "passthrough"
2023/09/25 10:47:24 INFO: [core] [Channel #9] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/netbird.hidden.tech:33073 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:24 INFO: [core] [Channel #9] Channel authority set to "netbird.hidden.tech:33073"
2023/09/25 10:47:24 INFO: [core] [Channel #9] Resolver state updated: {
"Addresses": [
{
"Addr": "netbird.hidden.tech:33073",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2023/09/25 10:47:24 INFO: [core] [Channel #9] Channel switches to new LB policy "pick_first"
2023/09/25 10:47:24 INFO: [core] [Channel #9 SubChannel #10] Subchannel created
2023/09/25 10:47:24 INFO: [core] [Channel #9] Channel Connectivity change to CONNECTING
2023/09/25 10:47:24 INFO: [core] [Channel #9 SubChannel #10] Subchannel Connectivity change to CONNECTING
2023/09/25 10:47:24 INFO: [core] [Channel #9 SubChannel #10] Subchannel picks a new address "netbird.hidden.tech:33073" to connect
2023/09/25 10:47:24 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0006f8090, {CONNECTING <nil>}
2023/09/25 10:47:29 INFO: [core] [Channel #9] Channel Connectivity change to SHUTDOWN
2023/09/25 10:47:29 INFO: [core] [Channel #9 SubChannel #10] Subchannel Connectivity change to SHUTDOWN
2023/09/25 10:47:29 INFO: [core] [Channel #9 SubChannel #10] Subchannel deleted
2023/09/25 10:47:29 INFO: [core] [Channel #9] Channel deleted
2023-09-25T10:47:29+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-09-25T10:47:29+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
Error: foreground login failed: backoff cycle failed: context deadline exceeded
Any news? This issue is very annoying and a deal breaker for me, if no solution is around the corner...
Could you please upgrade to the most recent version, v0.23.6? After the update, follow these steps:
sudo netbird service stop
sudo netbird service uninstall
sudo netbird service install
sudo netbird service start
and test the client's connection with netbird up
There is no v0.23.6 available for my centos - v0.23.4 is the latest available.
I've tried your solution but result the same:
Downloading packages:
netbird_0.23.4_linux_amd64.rpm | 8.4 MB 00:00:05
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : netbird-0.23.4-1.x86_64 1/1 Post Install of an clean install
Netbird service has been installed
Netbird service has been started
Verifying : netbird-0.23.4-1.x86_64 1/1
Installed:
netbird.x86_64 0:0.23.4-1
Complete!
:~$ sudo netbird service stop
Netbird service has been stopped
:~$ sudo netbird service uninstall
Netbird has been uninstalled
:~$ sudo netbird service install
Netbird service has been installed
:~$ sudo netbird service start
Netbird service has been started
:~$ sudo netbird up --management-url https://netbird.hidden.tech:33073 --setup-key 'hidden'
Error: unable to get daemon status: rpc error: code = FailedPrecondition desc = failed connecting to Management Service : context deadline exceeded
:~$ sudo netbird down
sudo netbird up --management-url https://netbird.hidden.tech:33073 --setup-key 'hidden'
2023-10-04T10:39:55+03:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 890.429447ms due to error rpc error: code = DeadlineExceeded desc = context deadline exceeded
2023-10-04T10:40:06+03:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 1.12732371s due to error rpc error: code = DeadlineExceeded desc = context deadline exceeded
Error: login backoff cycle failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
:~$ tail /var/log/netbird/client.log
2023-10-04T10:40:01+03:00 ERRO client/server/server.go:124: failed login: context deadline exceeded
2023-10-04T10:40:06+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-10-04T10:40:06+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023-10-04T10:40:06+03:00 ERRO client/server/server.go:124: failed login: context deadline exceeded
2023-10-04T10:40:12+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-10-04T10:40:12+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023-10-04T10:40:12+03:00 ERRO client/server/server.go:124: failed login: context deadline exceeded
2023-10-04T10:40:17+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-10-04T10:40:17+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023-10-04T10:40:17+03:00 ERRO client/server/server.go:124: failed login: context deadline exceeded
:~$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Any other suggestion?
~ > sudo netbird service stop
Netbird service has been stopped
~ > sudo netbird service uninstall
Netbird has been uninstalled
~ > sudo netbird service install
Netbird service has been installed
~ > sudo netbird service start
Netbird service has been started
~ > netbird up
Error: unable to get daemon status: rpc error: code = FailedPrecondition desc = failed connecting to Management Service : context deadline exceeded
~ > netbird version
0.23.6
I also stumbled across this kind of error today. For me the solution was to adjust my /etc/resolv.conf. there were two entries for nameserver + search domain. after removing the second (original host) entry and keeping the one created by netbird, a "netbird up" worked again.
it seems that "netbird up" failed to resolve its required login servers.
I have this problem:
Error: unable to get daemon status: rpc error: code = FailedPrecondition desc = failed while getting Management Service public key: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); malformed header: missing HTTP content-type
I also stumbled across this kind of error today. For me the solution was to adjust my /etc/resolv.conf. there were two entries for nameserver + search domain. after removing the second (original host) entry and keeping the one created by netbird, a "netbird up" worked again.
it seems that "netbird up" failed to resolve its required login servers.
This also fixed it for me, apparently adding a custom DNS nameserver in netbird that's only accessible via the VPN causes this.
In my case, it was because I changed my Netbird host url (like... old.example.com → new.example.io).
Fix was to delete (or move to /tmp) the config.json in /etc/netbird/config.json,
and ran sudo netbird service uninstall
. Not sure if which handled the issue though
Just want to bring attention to this issue again. We had the same problem today with one of our servers. Server rebooted due to a failure (non Netbird related) and did not show up in Netbird as online. After running the command netbird up
manually, we got the same error message as some of you above:
Error: status failed: failed connecting to Management Service : context deadline exceeded
We figured out that Netbird replaces or adds its own Nameserver (if configured) and search into the /etc/resolv.conf
. The order was Netbird-configured-search, Netbird-configured-nameserver and then our two nameservers. We also noticed problems with the general DNS resolution to e.g. google.de. Our Netbird instance address resolves via a public DNS and therefore this setup won't work.
After removing all entries except our two public nameservers and waiting a few seconds, we could successfully connect to our Netbird instance with netbird up
. This, obviously, overwrote the /etc/resolv.conf
again. This will likely result in the same scenario if the machine goes down again the next time.
Is there any way to avoid this problem here?
Just want to bring attention to this issue again. We had the same problem today with one of our servers. Server rebooted due to a failure (non Netbird related) and did not show up in Netbird as online. After running the command
netbird up
manually, we got the same error message as some of you above:Error: status failed: failed connecting to Management Service : context deadline exceeded
We figured out that Netbird replaces or adds its own Nameserver (if configured) and search into the
/etc/resolv.conf
. The order was Netbird-configured-search, Netbird-configured-nameserver and then our two nameservers. We also noticed problems with the general DNS resolution to e.g. google.de. Our Netbird instance address resolves via a public DNS and therefore this setup won't work.After removing all entries except our two public nameservers and waiting a few seconds, we could successfully connect to our Netbird instance with
netbird up
. This, obviously, overwrote the/etc/resolv.conf
again. This will likely result in the same scenario if the machine goes down again the next time.Is there any way to avoid this problem here?
I found the same behavior. After removing the servers from resolv.conf, netbird connected.
link to slack thread https://netbirdio.slack.com/archives/C05T5K65X7U/p1704806688445639
I have this problem:
Error: unable to get daemon status: rpc error: code = FailedPrecondition desc = failed while getting Management Service public key: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); malformed header: missing HTTP content-type
this is the error I also have. Where you able to resolve your issue?
for nam
This issue (the DeadlineExceeded ..) re-appears for us when running the latest available current version (0.27.3
) on alma linux. We are unable to get netbird running.
Our fix was downgrading to 0.26.3. We have checked the /etc/resolv.conf
file for additional entries as well as using the command by @mlsmaycon to increase grpc logging level, but neither of those made us any wiser.
Same for me on desktop with Ubuntu 22.04.4 LTS and the latest version 0.27.4 via the recommended installation method: curl -fsSL https://pkgs.netbird.io/install.sh | sh
I tried a variety of permutations:
netbird up
sudo netbird up
netbird up --setup-key 12345...
They all give me the same error: Error: login failed: rpc error: code = PermissionDenied desc = peer login has expired, please log in once more
I edited my /etc/resolv.conf
and just put a single entry nameserver 1.1.1.1
but still no luck
Reboot made no difference
Update: upgrading to Ubuntu 24.04 fixed it for me.
Same here, on openwrt netbird version 0.27.3
`Error: unable to get daemon status: rpc error: code = FailedPrecondition desc = failed connecting to Management Service : context deadline exceeded
Error: status failed: failed connecting to Management Service : context deadline exceeded
Peers detail: Daemon version: 0.27.3 CLI version: 0.27.3 Management: Disconnected, reason: rpc error: code = FailedPrecondition desc = failed connecting to Management Service : context deadline exceeded Signal: Disconnected Relays: Nameservers: FQDN: NetBird IP: N/A Interface type: N/A Quantum resistance: false Routes: - Peers count: 0/0 Connected
Netbird service has been stopped
-l debug'
2024-05-13T04:13:57Z DEBG client/internal/login.go:93: connecting to the Management service https://api.netbird.io:443
2024/05/13 04:13:57 INFO: [core] [Channel #1] Channel created
2024/05/13 04:13:57 INFO: [core] [Channel #1] original dial target is: "api.netbird.io:443"
2024/05/13 04:13:57 INFO: [core] [Channel #1] parsed dial target is: {Scheme:api.netbird.io Authority: URL:{Scheme:api.netbird.io Opaque:443 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2024/05/13 04:13:57 INFO: [core] [Channel #1] fallback to scheme "passthrough"
2024/05/13 04:13:57 INFO: [core] [Channel #1] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/api.netbird.io:443 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2024/05/13 04:13:57 INFO: [core] [Channel #1] Channel authority set to "api.netbird.io:443"
2024/05/13 04:13:57 INFO: [core] [Channel #1] Resolver state updated: {
"Addresses": [
{
"Addr": "api.netbird.io:443",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2024/05/13 04:13:57 INFO: [core] [Channel #1] Channel switches to new LB policy "pick_first"
2024/05/13 04:13:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel created
2024/05/13 04:13:57 INFO: [core] [Channel #1] Channel Connectivity change to CONNECTING
2024/05/13 04:13:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING
2024/05/13 04:13:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "api.netbird.io:443" to connect
2024/05/13 04:13:57 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {CONNECTING
ping api.netbird.io PING api.netbird.io (35.186.199.111): 56 data bytes 64 bytes from 35.186.199.111: seq=0 ttl=114 time=11.965 ms 64 bytes from 35.186.199.111: seq=1 ttl=114 time=12.270 ms 64 bytes from 35.186.199.111: seq=2 ttl=114 time=12.027 ms
cat /etc/resolv.conf search lan nameserver 127.0.0.1 nameserver ::1
I was thinking it could be caused by DNS , but It is able to resolve the api link just fine. I tried to get help in the slack but not results. Any suggestions?
Same here, on openwrt netbird version 0.27.3
`Error: unable to get daemon status: rpc error: code = FailedPrecondition desc = failed connecting to Management Service : context deadline exceeded
netbird status -d
Error: status failed: failed connecting to Management Service : context deadline exceeded
netbird status -d output:
Peers detail: Daemon version: 0.27.3 CLI version: 0.27.3 Management: Disconnected, reason: rpc error: code = FailedPrecondition desc = failed connecting to Management Service : context deadline exceeded Signal: Disconnected Relays: Nameservers: FQDN: NetBird IP: N/A Interface type: N/A Quantum resistance: false Routes: - Peers count: 0/0 Connected
sudo netbird service stop
Netbird service has been stopped
sudo bash -c 'GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info netbird up -F
-l debug' 2024-05-13T04:13:57Z DEBG client/internal/login.go:93: connecting to the Management service https://api.netbird.io:443 2024/05/13 04:13:57 INFO: [core] [Channel #1] Channel created 2024/05/13 04:13:57 INFO: [core] [Channel #1] original dial target is: "api.netbird.io:443" 2024/05/13 04:13:57 INFO: [core] [Channel #1] parsed dial target is: {Scheme:api.netbird.io Authority: URL:{Scheme:api.netbird.io Opaque:443 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}} 2024/05/13 04:13:57 INFO: [core] [Channel #1] fallback to scheme "passthrough" 2024/05/13 04:13:57 INFO: [core] [Channel #1] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/api.netbird.io:443 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}} 2024/05/13 04:13:57 INFO: [core] [Channel #1] Channel authority set to "api.netbird.io:443" 2024/05/13 04:13:57 INFO: [core] [Channel #1] Resolver state updated: { "Addresses": [ { "Addr": "api.netbird.io:443", "ServerName": "", "Attributes": null, "BalancerAttributes": null, "Type": 0, "Metadata": null } ], "ServiceConfig": null, "Attributes": null } (resolver returned new addresses) 2024/05/13 04:13:57 INFO: [core] [Channel #1] Channel switches to new LB policy "pick_first" 2024/05/13 04:13:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel created 2024/05/13 04:13:57 INFO: [core] [Channel #1] Channel Connectivity change to CONNECTING 2024/05/13 04:13:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING 2024/05/13 04:13:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "api.netbird.io:443" to connect 2024/05/13 04:13:57 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {CONNECTING } 2024-05-13T04:13:57Z ERRO util/grpc/dialer.go:33: Failed to dial: dial: dial tcp 35.186.199.111:443: connect: invalid argument 2024/05/13 04:13:57 INFO: [core] Creating new client transport to "{Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }": connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:13:57 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }. Err: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:13:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:13:57 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:13:57 INFO: [core] [Channel #1] Channel Connectivity change to TRANSIENT_FAILURE 2024/05/13 04:13:58 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:13:58 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {IDLE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:13:58 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING 2024/05/13 04:13:58 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "api.netbird.io:443" to connect 2024/05/13 04:13:58 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {CONNECTING } 2024-05-13T04:13:58Z ERRO util/grpc/dialer.go:33: Failed to dial: dial: dial tcp 35.186.199.111:443: connect: invalid argument 2024/05/13 04:13:58 INFO: [core] Creating new client transport to "{Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }": connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:13:58 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }. Err: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:13:58 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:13:58 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:00 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:00 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {IDLE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:00 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING 2024/05/13 04:14:00 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "api.netbird.io:443" to connect 2024/05/13 04:14:00 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {CONNECTING } 2024-05-13T04:14:00Z ERRO util/grpc/dialer.go:33: Failed to dial: dial: dial tcp 35.186.199.111:443: connect: invalid argument 2024/05/13 04:14:00 INFO: [core] Creating new client transport to "{Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }": connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:00 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }. Err: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:00 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:00 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:02 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:02 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {IDLE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:02 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING 2024/05/13 04:14:02 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "api.netbird.io:443" to connect 2024/05/13 04:14:02 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {CONNECTING } 2024-05-13T04:14:02Z ERRO util/grpc/dialer.go:33: Failed to dial: dial: dial tcp 35.186.199.111:443: connect: invalid argument 2024/05/13 04:14:02 INFO: [core] Creating new client transport to "{Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }": connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:02 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }. Err: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:02 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:02 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:06 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:06 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {IDLE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:06 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING 2024/05/13 04:14:06 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "api.netbird.io:443" to connect 2024/05/13 04:14:06 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {CONNECTING } 2024-05-13T04:14:06Z ERRO util/grpc/dialer.go:33: Failed to dial: dial: dial tcp 35.186.199.111:443: connect: invalid argument 2024/05/13 04:14:06 INFO: [core] Creating new client transport to "{Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }": connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:06 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }. Err: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:06 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:06 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x3476b10, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:07 INFO: [core] [Channel #1] Channel Connectivity change to SHUTDOWN 2024/05/13 04:14:07 INFO: [core] [Channel #1] ccBalancerWrapper: closing 2024/05/13 04:14:07 INFO: [core] [Channel #1] Closing the name resolver 2024/05/13 04:14:07 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to SHUTDOWN 2024/05/13 04:14:07 INFO: [core] [Channel #1 SubChannel #2] Subchannel deleted 2024/05/13 04:14:07 INFO: [core] [Channel #1] Channel deleted 2024-05-13T04:14:07Z ERRO management/client/grpc.go:68: failed creating connection to Management Service context deadline exceeded 2024-05-13T04:14:07Z ERRO client/internal/login.go:96: failed connecting to the Management service https://api.netbird.io:443 context deadline exceeded 2024-05-13T04:14:07Z WARN client/cmd/root.go:214: retrying Login to the Management service in 734.073652ms due to error context deadline exceeded 2024-05-13T04:14:08Z DEBG client/internal/login.go:93: connecting to the Management service https://api.netbird.io:443 2024/05/13 04:14:08 INFO: [core] [Channel #3] Channel created 2024/05/13 04:14:08 INFO: [core] [Channel #3] original dial target is: "api.netbird.io:443" 2024/05/13 04:14:08 INFO: [core] [Channel #3] parsed dial target is: {Scheme:api.netbird.io Authority: URL:{Scheme:api.netbird.io Opaque:443 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}} 2024/05/13 04:14:08 INFO: [core] [Channel #3] fallback to scheme "passthrough" 2024/05/13 04:14:08 INFO: [core] [Channel #3] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/api.netbird.io:443 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}} 2024/05/13 04:14:08 INFO: [core] [Channel #3] Channel authority set to "api.netbird.io:443" 2024/05/13 04:14:08 INFO: [core] [Channel #3] Resolver state updated: { "Addresses": [ { "Addr": "api.netbird.io:443", "ServerName": "", "Attributes": null, "BalancerAttributes": null, "Type": 0, "Metadata": null } ], "ServiceConfig": null, "Attributes": null } (resolver returned new addresses) 2024/05/13 04:14:08 INFO: [core] [Channel #3] Channel switches to new LB policy "pick_first" 2024/05/13 04:14:08 INFO: [core] [Channel #3 SubChannel #4] Subchannel created 2024/05/13 04:14:08 INFO: [core] [Channel #3] Channel Connectivity change to CONNECTING 2024/05/13 04:14:08 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING 2024/05/13 04:14:08 INFO: [core] [Channel #3 SubChannel #4] Subchannel picks a new address "api.netbird.io:443" to connect 2024/05/13 04:14:08 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {CONNECTING } 2024-05-13T04:14:08Z ERRO util/grpc/dialer.go:33: Failed to dial: dial: dial tcp 35.186.199.111:443: connect: invalid argument 2024/05/13 04:14:08 INFO: [core] Creating new client transport to "{Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }": connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:08 WARNING: [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }. Err: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:08 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:08 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:08 INFO: [core] [Channel #3] Channel Connectivity change to TRANSIENT_FAILURE 2024/05/13 04:14:09 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:09 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {IDLE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:09 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING 2024/05/13 04:14:09 INFO: [core] [Channel #3 SubChannel #4] Subchannel picks a new address "api.netbird.io:443" to connect 2024/05/13 04:14:09 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {CONNECTING } 2024-05-13T04:14:09Z ERRO util/grpc/dialer.go:33: Failed to dial: dial: dial tcp 35.186.199.111:443: connect: invalid argument 2024/05/13 04:14:09 INFO: [core] Creating new client transport to "{Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }": connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:09 WARNING: [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }. Err: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:09 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:09 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:11 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:11 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {IDLE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:11 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING 2024/05/13 04:14:11 INFO: [core] [Channel #3 SubChannel #4] Subchannel picks a new address "api.netbird.io:443" to connect 2024/05/13 04:14:11 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {CONNECTING } 2024-05-13T04:14:11Z ERRO util/grpc/dialer.go:33: Failed to dial: dial: dial tcp 35.186.199.111:443: connect: invalid argument 2024/05/13 04:14:11 INFO: [core] Creating new client transport to "{Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }": connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:11 WARNING: [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }. Err: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:11 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:11 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:13 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:13 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {IDLE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:13 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING 2024/05/13 04:14:13 INFO: [core] [Channel #3 SubChannel #4] Subchannel picks a new address "api.netbird.io:443" to connect 2024/05/13 04:14:13 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {CONNECTING } 2024-05-13T04:14:13Z ERRO util/grpc/dialer.go:33: Failed to dial: dial: dial tcp 35.186.199.111:443: connect: invalid argument 2024/05/13 04:14:13 INFO: [core] Creating new client transport to "{Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }": connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:13 WARNING: [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }. Err: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:13 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:13 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:17 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:17 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {IDLE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:17 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING 2024/05/13 04:14:17 INFO: [core] [Channel #3 SubChannel #4] Subchannel picks a new address "api.netbird.io:443" to connect 2024/05/13 04:14:17 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {CONNECTING } 2024-05-13T04:14:17Z ERRO util/grpc/dialer.go:33: Failed to dial: dial: dial tcp 35.186.199.111:443: connect: invalid argument 2024/05/13 04:14:17 INFO: [core] Creating new client transport to "{Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }": connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:17 WARNING: [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "api.netbird.io:443", ServerName: "api.netbird.io:443", }. Err: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:17 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument" 2024/05/13 04:14:17 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0x34777d0, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing: dial: dial tcp 35.186.199.111:443: connect: invalid argument"} 2024/05/13 04:14:18 INFO: [core] [Channel #3] Channel Connectivity change to SHUTDOWN 2024/05/13 04:14:18 INFO: [core] [Channel #3] ccBalancerWrapper: closing 2024/05/13 04:14:18 INFO: [core] [Channel #3] Closing the name resolver 2024/05/13 04:14:18 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to SHUTDOWN 2024/05/13 04:14:18 INFO: [core] [Channel #3 SubChannel #4] Subchannel deleted 2024/05/13 04:14:18 INFO: [core] [Channel #3] Channel deleted 2024-05-13T04:14:18Z ERRO management/client/grpc.go:68: failed creating connection to Management Service context deadline exceeded 2024-05-13T04:14:18Z ERRO client/internal/login.go:96: failed connecting to the Management service https://api.netbird.io:443 context deadline exceeded 2024-05-13T04:14:18Z WARN client/cmd/root.go:214: retrying Login to the Management service in 2.018719926s due to error context deadline exceeded`
ping api.netbird.io PING api.netbird.io (35.186.199.111): 56 data bytes 64 bytes from 35.186.199.111: seq=0 ttl=114 time=11.965 ms 64 bytes from 35.186.199.111: seq=1 ttl=114 time=12.270 ms 64 bytes from 35.186.199.111: seq=2 ttl=114 time=12.027 ms
cat /etc/resolv.conf search lan nameserver 127.0.0.1 nameserver ::1
I was thinking it could be caused by DNS , but It is able to resolve the api link just fine. I tried to get help in the slack but not results. Any suggestions?
update: Installed netbird verions 0.17.0 and worked right away, so apparently was a issue with previous mentioned version.
Might be affected by this as well, or it might be slightly different issue. All Proxmox PVE and PBS instances will eventually disconnect from peers, or end up with Error: unable to get daemon status: rpc error: code = FailedPrecondition desc = failed connecting to Management Service : context deadline exceeded
after being connected for a while. Bringing it down and up solves it for an undefined amount of time before the issue returns (sometimes takes a day, or about an hour).
So far I've just setup a cronjob */20 * * * * /usr/bin/netbird down && /usr/bin/netbird up
which seems to keep things connected.
Tested on 4 different PVE hosts (v7.x and v8.x) and 3 different PVB instances (v2.x and v3.x) and they all suffer from this. Does not seem to affect other servers we got.
Hi, I also had an issue with registration I used a Proxmox Ubuntu LXC and installed the latest netbird client version. There was only this one line in the client.log: " ERRO management/client/grpc.go:287: failed while getting Management Service public key: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/html" " The other log files were all empty. I deleted the LXC and created a new one and everything worked fine with the registration.
PS: This comment is just to let you know about the error. Reach out to me if you need further information.
Hello, I started using Netbird yesterday on my linux machine. It was working fine, but once i restarted my machine it displays the message "2023-08-03T10:19:37+01:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 1.098619925s due to error rpc error: code = Unknown desc = getting pkce authorization flow info failed with error: context deadline exceeded" when i run "sudo netbird up" even with my setup-key. I tried uninstall/reinstall but nothing worked. Can you help please ?
Steps to reproduce the behavior:
The expected behavior is to have the same as yesterday when i installed it. I expected it to display "connected" after i run the command "sudo netbird up".
NetBird status -d output: Daemon status: LoginFailed
Run UP command to log in with SSO (interactive login):
netbird up
If you are running a self-hosted version and no SSO provider has been configured in your Management Server, you can use a setup-key:
netbird up --management-url --setup-key
More info: https://www.netbird.io/docs/overview/setup-keys
Screenshots
![image](https://github.com/netbirdio/netbird/assets/86080510/2f994e60-a727-4a79-b1b9-3ee5e2618be5)
Thanks for help