Destination Host Unreachable, ping: sendmsg: Required key not available

FarisZR commented 2 years ago

Hello, I have been using wiretrustee to expose a local server to a remote VPS.

However, every once in a while it stops working, and this happens:

Logs

# ping 100.64.0.2
PING 100.64.0.2 (100.64.0.2) 56(84) bytes of data.
From 100.64.0.1 icmp_seq=1 Destination Host Unreachable
ping: sendmsg: Required key not available
From 100.64.0.1 icmp_seq=2 Destination Host Unreachable
ping: sendmsg: Required key not available
From 100.64.0.1 icmp_seq=3 Destination Host Unreachable

Removing both peers and re-adding them sometimes fixes it, or peers still can't ping each other.

setup details

one peer is running wiretrustee inside docker, and the other is running a standard Debian installation inside an LXC, with appropriate options

mlsmaycon commented 2 years ago

Hello @fariszr, can you share some details of your setup?

Are you using the Managed version of Wiretrustee?
Which version of Wiretrustee are the peers using? you can find that with wiretrustee version command
Are the peers running on different nodes?
Did you give privileged permissions to the Docker and LXC peers? for docker, we need to run with the flag --cap-add=NET_ADMIN

FarisZR commented 2 years ago

Hello @fariszr, can you share some details of your setup?

Are you using the Managed version of Wiretrustee?

Which version of Wiretrustee are the peers using? you can find that with wiretrustee version command

Are the peers running on different nodes?

Did you give privileged permissions to the Docker and LXC peers? for docker, we need to run with the flag --cap-add=NET_ADMIN

1- yes 2 – Debian, 0.4.0. Docker, the latest image available from Docker hub using the latest tag. 3 – Yes, Totally different servers and hosting providers. 4 – LXC is Unprivileged, which worked, and I don't think it's the problem. I added the net_admin cap, It stopped showing Required key not available but ping still doesn't work.

Wiretrustee logs

wiretrustee-wiretrustee-1  | time="2022-03-09T19:xx:xxZ" level=info msg="connected to peer p+xxxxxxxxxxxx= [laddr <-> raddr] [172.21.0.1:40908 <-> xx.xxx.xxx.192:55404]"

Compose file

version: '3.3'
services:
    wiretrustee:
        network_mode: host
        privileged: true
        restart: always
        userns_mode: host
        environment:
            - WT_SETUP_KEY=XXXXXXXXX
        volumes:
            - ./wiretrustee-client:/etc/wiretrustee
        image: wiretrustee/wiretrustee:latest
        cap_add:
            - NET_ADMIN

mlsmaycon commented 2 years ago

Hello @fariszr sorry for the late response, we've found a bug on our managed service that was affecting some network updates to be propagated. Can you check if the connections are more stable now?

In case you still facing this issue, would you mind joining our slack? I believe we could have a more dynamic conversation there :) https://join.slack.com/t/wiretrustee/shared_invite/zt-vrahf41g-ik1v7fV8du6t0RwxSrJ96A

ngtrthanh commented 2 years ago

Hello, I have been using wiretrustee to expose a local server to a remote VPS.

However, every once in a while it stops working, and this happens:

Logs
# ping 100.64.0.2
PING 100.64.0.2 (100.64.0.2) 56(84) bytes of data.
From 100.64.0.1 icmp_seq=1 Destination Host Unreachable
ping: sendmsg: Required key not available
From 100.64.0.1 icmp_seq=2 Destination Host Unreachable
ping: sendmsg: Required key not available
From 100.64.0.1 icmp_seq=3 Destination Host Unreachable
Removing both peers and re-adding them sometimes fixes it, or peers still can't ping each other.

setup details

one peer is running wiretrustee inside docker, and the other is running a standard Debian installation inside an LXC, with appropriate options

Hi, I got same issue. And I realized that I install pivpn with wireguard before. Remove pivpn (and WG) by pivnp -u will solve problem. FYI

FarisZR commented 2 years ago

Hi, i have just updated Wiretrustee on both machines, and it now works for some reason. And Yes, my docker node is also running wireguard on docker, however i don't this is really the cause, as it now works normally with the vpn.

I will keep you updated if any issue pops up.

On Sat, Mar 12 2022 at 06:02:51 AM -0000, wiretrustee/wiretrustee - reply+aipxadsdtufjdqx4tnie2owahfvarevbnhheky5rui at reply.github.com @.***> wrote:

Hello, I have been using wiretrustee to expose a local server to a remote VPS.

However, every once in a while it stops working, and this happens:

Logs

ping 100.64.0.2

PING 100.64.0.2 (100.64.0.2) 56(84) bytes of data. From 100.64.0.1 icmp_seq=1 Destination Host Unreachable ping: sendmsg: Required key not available From 100.64.0.1 icmp_seq=2 Destination Host Unreachable ping: sendmsg: Required key not available From 100.64.0.1 icmp_seq=3 Destination Host Unreachable

Removing both peers and re-adding them sometimes fixes it, or peers still can't ping each other.

setup details

one peer is running wiretrustee inside docker, and the other is running a standard Debian installation inside an LXC, with appropriate options

Hi, I got same issue. And I realized that I install pivpn with wireguard before. Remove pivpn (and WG) by pivnp -u will solve problem. FYI

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.***>

FarisZR commented 2 years ago

The issue is back

FarisZR commented 2 years ago

aaaand.. Its gone again. It's very annoying when I'm remoting into my server, and it doesn't work while I'm outside the network.

FarisZR commented 2 years ago

An update on this issue. Its back again, and it's a dealbreaker, I'm searching for an alternative, as it stands with my usage wiretrustee/net bird is not reliable at all. It works once and breaks 10 times, when I update the debian package on the LXC it works then it breaks randomly, does this issue could have anything with being late for updates?

I think this is an issue resides from the LXC side, as I have been using wiretrustee on full servers and it's mostly great. Another issue is there is no mobile client and no exit node support. I want to be able to get the fastest speed when in LAN, and access the server remotely, without having to deal with multi-zone DNS. #289 #213

As for slack, unfortunately I don't use Slack, we can continue here, or you can open a matrx.org room which can be bridged to slack.

ykhedar commented 2 years ago

Hi, I am having similar issue with a self hosted system. The server is deployed on a AWS EC2 instance with all necessary ports open and the dashboard working fine without any problem. The client is a home computer with Ubuntu 20 and a mac. I am using version 0.5.1 for all of the devices and the dashboard shows them to be online. I am just not able to even ping the server. Is this somehow related to https://github.com/netbirdio/netbird/issues/302 ?

lyca-knight commented 2 years ago

Same here. I self-hosted on Linux. (Managed Service same error). And Connected 3 Linux Clients. Works perfectly fine. My Mac M1 shows online. But I can't ping or reach the other Clients. Removing and Reading works sometimes, but same again if I change networks.

braginini commented 2 years ago

Hi @lyca-knight, The problem is only on Mac, right? Linux clients are fine when switching networks?

What version are you running? Could you please run sudo wireguard show on Mac and show the output?

You might need to install wireguard-tools package.

Feel free to DM me on Slack

lyca-knight commented 2 years ago

Hey @braginini, thanks for your reply :) Yeah, only on Mac. The Linux Clients are Cloud Boxes from Hetzner, so no networking switching here.

I reinstalled the Mac to see if it's an error with my setting or something. But same error on a clean installation. And without networking, switching. I am not able to ping or connect to the other machines. On the Dashboard all Clients are online. I installed wireguard-tools package but the sudo wireguard show command gives only command, not found.

Yeah, I will leave you a DM on Slack

mlsmaycon commented 2 years ago

Hi @lyca-knight,

Could you please run sudo wg show ?

Also, could you share the NetBird IP of your Mac with the output of the above command?

hicwic commented 1 year ago

hi, i know it's a old issue, but i having this problem with a self hosted netbird. Tested with you managed and all is working. I don't know how to start to check/configure to resolve this issue

no apparent errors, even on self hosted side
nodes on same lan connect to each other succefully
remote nodes con't connect to other

got this error while pinging: ping: sendmsg: Required key not available

i'll take any advices

thx

mlsmaycon commented 1 year ago

Hello, @hicwic sorry for not replying sooner. This error usually indicates a problem with the P2P connection, it means that the peers didn't negotiate the best connection option.

Recently we've improved our core connectivity layer and this kind of issue might be mitigated.

masx200 commented 8 months ago

I found that when "Interface type: Kernel", I can't access 100.124.255.254? write to '100.124.255.254': Required key not available; But how do I switch to "Interface type: Userspace"?

OpenWrt 23.05.2 r23630-842932a63d / LuCI openwrt-23.05 branch git-23.306.39416-c86c256

dshoreman commented 7 months ago

I had the same ping errors^* today on a new self-hosted install.

The client for 2 of 3 peers (one being the server machine itself) for some reason was stopped. I ran netbird up a second time, then they show connected and could ping each other.

The third peer was already running Wireguard. Maybe the quick install option could detect if wg0/51820 are in use and increment if necessary, but a manual tweak was easy enough:

Open /etc/netbird/config.json
Change WgIface and WgPort to something else (I used wg1 with port 51821)
Apply changes with netbird service restart

^{*This issue was top of search so maybe this helps someone else... but more likely me when I forget in a month.}

grebois commented 7 months ago

Same problem here with new self-hosted install

braginini commented 7 months ago

What version of the netbird client are you running? @grebois

grebois commented 7 months ago

@braginini 0.26.2, 0.27.0 and 0.27.1

madindehead commented 7 months ago

I'm expericing this issue right now - self-hosted install using the quick install script with Zitadel. Everything started correctly and the clients connected to the networks properly. However doing a ping gives me this error.

Unsure if it's port related or not.

GustavooLucio commented 6 months ago

had this on 0.27.2 ubuntu when pinging a specific peer, rebooting the machine fixed it

snorflenet commented 3 months ago

Brand new self hosted install of 0.28.4 had the issue. I found this thread and then I went digging more. I tried restarting nodes and networking, no dice.

My issue was solved when I opened wireguard port UDP 51820 to allow for tunneled traffic, specifically on the cloud side. If wireguard is trying to penetrate a hard NAT traversal and doesn't have free access to this port on at least one side, particularly with the public IP, then it can be challenging to bring up a proper p2p connection.

This isn't netbird's fault, though they could bring attention to the issue in the docs. Maybe it is there, but I didn't run across it in my last install.

I hope this helps.

netbirdio / netbird