search

cloudflare / cloudflared

Cloudflare Tunnel client (formerly Argo Tunnel)
https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide
Apache License 2.0
8.88k stars 780 forks source link

Application connection #710

Open Aeisu opened 2 years ago

Aeisu commented 2 years ago

The issue i'm having is I think I am being disconnected from my server once an initial connection is made.

I run a Final Fantasy XI private server on my home network, to bypass CGNAT I use a cloudflared tunnel which is tested works. I added an enrolment rule to Zero Trust so only a particular e-mail could log in. I installed and setup WARP on the client machine and used a secondary internet connection to test connectivity. This was a success I was then able to connect to the server on the local network through the tunnel and get a response from the server I was able to login to the game client with this initial connection but then is seems to stop receiving data altogether. The game on the client side loses connection after a period of time and can't reconnect afterwards. I think the problem is that there is no keep alive for any inbound connections going to the server IP and wondering if there is a way to make that happen possibly through ingress rules I am not sure of the possibility since i am connecting through a WARP client and not through other means.

DevinCarr commented 2 years ago

If you are looking for keep-alive between cloudflared and your origin service, you can look here in the ingress rules: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/local-management/ingress/#keepalivetimeout

Given that you are serving game server traffic, my guess would be that the game is likely serving traffic over UDP, so also make sure that your private network is setup to enable UDP: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/private-net/private-hostnames-ips/#enable-udp-support

Aeisu commented 2 years ago

the game client runs on TCP and UDP and I had those selected already. I'm not sure what the keep alive timeout would look like though inside the yaml file, all I have currently enabled is warp routing.

DevinCarr commented 2 years ago

Okay, my bad, I misunderstood how you were using the tunnel. Since you are using warp-to-tunnel only, the ingress rules won't apply, but you can update it like so for warp-routing config:

warp-routing:
  enabled: true
  connectTimeout: 5s
  tcpKeepAlive: 30s

(reference: https://github.com/cloudflare/cloudflared/blob/master/ingress/config.go#L46)

Aeisu commented 2 years ago

thanks! I gave that a try and it seems the problem is something other then that. I'm wondering if the client is getting back the data on the same port as the server sends it out or if it is on another port. When i look on the Zero Trust logs I see my incoming connection but the source port and the destination port are different. wouldn't this mean that returning data isn't sent to the same port on the client side?

DevinCarr commented 2 years ago

I'm not too sure offhand. We might need to look into your cloudflared logs, you can enable debugging logs by updating your command like so: cloudflared tunnel --loglevel debug run

Additionally if you could provide your tunnel id, I could look on our server logs.

Aeisu commented 2 years ago

the tunnel id: 8b01487c-2a37-4ddf-8f04-81e0bd72ae08 The specific address i'm trying to reach is 192.168.1.66 the TCP ports are 54001, 54002, 54003, 54230, and 54231 UDP 54230. I know the game server runs fine across the local network at least so it's just through the tunnel that is causing me issues.

Aeisu commented 2 years ago

I did two actions

  1. Logged onto an existing character for the game and got then had the data received plummet to 0
  2. Created a new account and character and the same thing happened. I enabled the debug log as you suggested but i'm not sure where it populates any data.
DevinCarr commented 2 years ago

I noted the image and the tunnel details, but for your privacy-sake, I have removed the image. I'll look in the logs on our side to see if there is anything I can see there.

I enabled the debug log as you suggested but i'm not sure where it populates any data.

Enabling debugging will increase the amount of logging cloudflared will emit. If you are logging to a file, it will be there, otherwise, if you are running in a terminal via cloudflared tunnel run the logs will be emitted there.

DevinCarr commented 2 years ago

On my side, I don't see any TCP errors, those connections seem to be fine. I do see a lot of UDP errors that indicate that your tunnel isn't active (which I assume that's because you took it down recently during testing). Additionally, it looks like your routes are probably setup fine for the tunnel. You might want to make sure that your services are bound correctly in that IP range and that the IP range is available for your origin machine. For instance, the machine that you are hosting the game server on, if you run ip addr is the destination IP available? In your example, is your server socket bound to 192.168.1.66 and is that network address available?

Aeisu commented 2 years ago

I have that network address manually assigned to the server computer through the router I have the server set to run specifically on that IP as well so it doesn't go anywhere else. the only other thing the server does is reference a MySQL table which i confirmed worked by creating a new account and character.

Aeisu commented 2 years ago

-- NETWORK SETTINGS

-- All settings are attached to the xi.settings object. This is published globally, and be accessed from C++ and any script.

-- This file is concerned mainly with networking between the database, client, and server executables.

xi = xi or {} xi.settings = xi.settings or {}

xi.settings.network = { SQL_HOST = "127.0.0.1", SQL_PORT = 3306, SQL_LOGIN = "root", SQL_PASSWORD = ".", SQL_DATABASE = ".",

LOGIN_DATA_IP   = "192.168.1.66",
LOGIN_DATA_PORT = 54230,
LOGIN_VIEW_IP   = "192.168.1.66",
LOGIN_VIEW_PORT = 54001,
LOGIN_AUTH_IP   = "192.168.1.66",
LOGIN_AUTH_PORT = 54231,

MAP_PORT = 54230,

SEARCH_PORT = 54002,

-- Central message server settings (ensure these are the same on both all map servers and the central (lobby) server
ZMQ_IP   = "192.168.1.66",
ZMQ_PORT = 54003,

This is what i'm working with server side 
So everything on the server end is all locked in i believe and so is the IP it connects too
DevinCarr commented 2 years ago

Looking into the cloudflared logs with debugging enabled will be able to tell me more considering the setup. Do you some logs snippets you could share?

Aeisu commented 2 years ago

well this is odd, I seem to have a stable connection that isn't disconnecting me with 0 received data.

Aeisu commented 2 years ago
24.114.87.123 | 192.168.1.66 | allow | 12fd5416700000e861059b4400000001 | Aug 1 2022 • 7:25:57 PM Policy NameAllow PortsPolicy IDb075bc32-d8ad-4387-8f39-2a2b1d371957Policy DescriptionFFXI PortsSource Port52367Source IP CountryCADestination Port54230ProtocoludpVirtual NetworkdefaultEmailaisu_silvermoon@hotmail.comUser ID32057254-8104-40da-8619-463e92203306Device IDec4edd9c-11c9-11ed-8909-46b53d3ee5f2 24.114.87.123 192.168.1.66 allow 12fd5416700000e861059b4400000001 Aug 1 2022 • 7:25:57 PM Policy Name Allow Ports Policy ID [b075bc32-d8ad-4387-8f39-2a2b1d371957](https://dash.teams.cloudflare.com/6ad3ebd9618a87eebadeed936edf69a5/gateway/policies/rule/b075bc32-d8ad-4387-8f39-2a2b1d371957) Policy Description FFXI Ports Source Port 52367 Source IP Country CA Destination Port 54230 Protocol udp Virtual Network default Email aisu_silvermoon@hotmail.com User ID 32057254-8104-40da-8619-463e92203306 Device ID ec4edd9c-11c9-11ed-8909-46b53d3ee5f2
Aeisu commented 2 years ago

would this maybe be due to packet size? I think there is a limitation on the server side for that

Aeisu commented 2 years ago

I think I may have solved my issue by stumbling around.... I h ave changed areas twice without any issues at all. The thing i changed was adding in a gateway policy for trafic going to the desired IP with a large port range 192.168.1.66 Ports 50000-65535 to be safe

Aeisu commented 2 years ago

that seemed to be a fluke, it's back to doing the disconnection now.

Aeisu commented 2 years ago
24.114.87.123 | 192.168.1.66 | allow | 12fd64bd050000e86106001400000001 | Aug 1 2022 • 7:44:08 PM Policy NameAllow PortsPolicy IDb075bc32-d8ad-4387-8f39-2a2b1d371957Policy DescriptionFFXI PortsSource Port52367Source IP CountryCADestination Port54230ProtocoludpVirtual NetworkdefaultEmailaisu_silvermoon@hotmail.comUser ID32057254-8104-40da-8619-463e92203306Device IDec4edd9c-11c9-11ed-8909-46b53d3ee5f2 24.114.87.123 192.168.1.66 allow 12fd64bd050000e86106001400000001 Aug 1 2022 • 7:44:08 PM Policy Name Allow Ports Policy ID [b075bc32-d8ad-4387-8f39-2a2b1d371957](https://dash.teams.cloudflare.com/6ad3ebd9618a87eebadeed936edf69a5/gateway/policies/rule/b075bc32-d8ad-4387-8f39-2a2b1d371957) Policy Description FFXI Ports Source Port 52367 Source IP Country CA Destination Port 54230 Protocol udp Virtual Network default Email aisu_silvermoon@hotmail.com User ID 32057254-8104-40da-8619-463e92203306 Device ID ec4edd9c-11c9-11ed-8909-46b53d3ee5f2 This was the attempt after the successful one
DevinCarr commented 2 years ago

These are logs from the Zero Trust Dashboard. Could you provide some logs from the cloudflared instance? It will help a bit more and provide more details that we don't have access to in the published logs.

Aeisu commented 2 years ago

2022-08-01T23:57:02Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=94baa82d-e4ae-4bd0-8696-3dca92c97dd2 2022-08-01T23:57:02Z DBG tcp proxy stream finished successfully flowID=fb199fd7-d21a-411c-af8b-f10fcce90a52 2022-08-01T23:57:02Z DBG tunnel->origin copy: readfrom tcp 192.168.1.66:56743->192.168.1.66:54001: Read on stream 437 canceled with error code 0 2022-08-01T23:57:03Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=2e1b3296-0f96-4bce-b264-255c90a9ea45 2022-08-01T23:57:03Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=fcd88fce-0424-4bab-9c19-bf7be10dd32b 2022-08-01T23:57:03Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=6c9b4073-b6f6-41b2-af63-eee86215d378 2022-08-01T23:57:03Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=09059a78-6a88-4d6f-8c75-5af8be57b45c src=192.168.1.66:51495 2022-08-01T23:57:04Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=09059a78-6a88-4d6f-8c75-5af8be57b45c 2022-08-01T23:57:04Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=09059a78-6a88-4d6f-8c75-5af8be57b45c 2022-08-01T23:57:06Z DBG Registered session connIndex=2 dst=192.168.1.66:54230 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 src=192.168.1.66:51497 2022-08-01T23:57:06Z DBG Received datagram from edge bytes=136 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:07Z DBG Send datagram back to edge bytes=228 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:07Z DBG Received datagram from edge bytes=53 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:07Z DBG Send datagram back to edge bytes=455 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:08Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=c5b9a3d5-faac-42a8-8685-e4659f47bbea 2022-08-01T23:57:08Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=36e8d399-fd63-4226-86b8-8b24881c3dae src=192.168.1.66:51498 2022-08-01T23:57:08Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=36e8d399-fd63-4226-86b8-8b24881c3dae 2022-08-01T23:57:08Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=36e8d399-fd63-4226-86b8-8b24881c3dae 2022-08-01T23:57:09Z DBG Received datagram from edge bytes=185 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:09Z DBG Send datagram back to edge bytes=797 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:10Z DBG Received datagram from edge bytes=67 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:10Z DBG Send datagram back to edge bytes=924 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:11Z DBG Received datagram from edge bytes=76 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:11Z DBG Send datagram back to edge bytes=667 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:12Z DBG Received datagram from edge bytes=70 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:12Z DBG Send datagram back to edge bytes=1104 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:12Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=757dc5a6-1dd6-4723-9eea-c563a63967e9 2022-08-01T23:57:12Z DBG Received datagram from edge bytes=67 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:12Z DBG Send datagram back to edge bytes=545 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:12Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=9bb91a78-e6fc-42a9-82aa-4c1ed9c544d1 src=192.168.1.66:51501 2022-08-01T23:57:12Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=9bb91a78-e6fc-42a9-82aa-4c1ed9c544d1 2022-08-01T23:57:12Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=9bb91a78-e6fc-42a9-82aa-4c1ed9c544d1 2022-08-01T23:57:12Z DBG Received datagram from edge bytes=67 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:12Z DBG Send datagram back to edge bytes=701 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:13Z DBG Received datagram from edge bytes=67 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:13Z DBG Send datagram back to edge bytes=1069 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:13Z DBG Received datagram from edge bytes=67 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:13Z DBG Send datagram back to edge bytes=299 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:14Z DBG Received datagram from edge bytes=68 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:14Z DBG Send datagram back to edge bytes=915 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:14Z DBG Received datagram from edge bytes=68 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:14Z DBG Send datagram back to edge bytes=784 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:14Z DBG Received datagram from edge bytes=67 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:14Z DBG Send datagram back to edge bytes=142 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:15Z DBG Received datagram from edge bytes=67 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:15Z DBG Send datagram back to edge bytes=49 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:15Z DBG Received datagram from edge bytes=72 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:15Z DBG Send datagram back to edge bytes=785 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:16Z DBG Received datagram from edge bytes=67 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:16Z DBG Send datagram back to edge bytes=776 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:16Z DBG Received datagram from edge bytes=67 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:16Z DBG Send datagram back to edge bytes=128 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:16Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=2698d150-9320-4166-ac87-685d3bf7c948 2022-08-01T23:57:16Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=f9585046-d3a2-4d6c-b1c7-8630ad508622 src=192.168.1.66:51503 2022-08-01T23:57:16Z DBG Received datagram from edge bytes=68 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:16Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=f9585046-d3a2-4d6c-b1c7-8630ad508622 2022-08-01T23:57:16Z DBG Send datagram back to edge bytes=49 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:16Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=f9585046-d3a2-4d6c-b1c7-8630ad508622 2022-08-01T23:57:17Z DBG Received datagram from edge bytes=71 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:17Z DBG Send datagram back to edge bytes=787 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:17Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:17Z DBG Send datagram back to edge bytes=777 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:18Z DBG Received datagram from edge bytes=67 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:18Z DBG Send datagram back to edge bytes=128 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:18Z DBG Received datagram from edge bytes=68 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:18Z DBG Send datagram back to edge bytes=49 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:19Z DBG Received datagram from edge bytes=71 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:19Z DBG Send datagram back to edge bytes=49 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:19Z DBG Received datagram from edge bytes=73 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:19Z DBG Send datagram back to edge bytes=49 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:19Z DBG Received datagram from edge bytes=73 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:19Z DBG Send datagram back to edge bytes=49 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:20Z DBG Received datagram from edge bytes=49 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:20Z DBG Send datagram back to edge bytes=49 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:20Z DBG Received datagram from edge bytes=49 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:20Z DBG Send datagram back to edge bytes=49 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:20Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=2bf73d74-2812-4432-b0b1-75c33f8838e1 2022-08-01T23:57:21Z DBG Received datagram from edge bytes=61 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:21Z DBG Send datagram back to edge bytes=63 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:21Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=faf26c8a-1e9c-4bb1-b8a1-0cb9fed9865b src=192.168.1.66:51505 2022-08-01T23:57:21Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=faf26c8a-1e9c-4bb1-b8a1-0cb9fed9865b 2022-08-01T23:57:21Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=faf26c8a-1e9c-4bb1-b8a1-0cb9fed9865b 2022-08-01T23:57:21Z DBG Received datagram from edge bytes=53 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:21Z DBG Send datagram back to edge bytes=49 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:23Z DBG Received datagram from edge bytes=136 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:24Z DBG Send datagram back to edge bytes=223 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:24Z DBG Received datagram from edge bytes=53 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:24Z DBG Send datagram back to edge bytes=453 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:25Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=359e5719-1def-470a-ac23-b979181ecfc7 2022-08-01T23:57:25Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=94521bae-40e5-46ab-8754-a9348efc54fb src=192.168.1.66:51507 2022-08-01T23:57:25Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=94521bae-40e5-46ab-8754-a9348efc54fb 2022-08-01T23:57:25Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=94521bae-40e5-46ab-8754-a9348efc54fb 2022-08-01T23:57:26Z DBG Received datagram from edge bytes=175 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:26Z DBG Send datagram back to edge bytes=797 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:27Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:27Z DBG Send datagram back to edge bytes=924 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:28Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:28Z DBG Send datagram back to edge bytes=667 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:28Z DBG Received datagram from edge bytes=68 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:28Z DBG Send datagram back to edge bytes=844 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:28Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:28Z DBG Send datagram back to edge bytes=639 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:29Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:29Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:29Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=4be33bab-9b42-4f89-af0f-c9e8434b75a1 2022-08-01T23:57:29Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:29Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:29Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=a2a23fb6-482a-4f52-9f10-296e38bf1101 src=192.168.1.66:51509 2022-08-01T23:57:29Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=a2a23fb6-482a-4f52-9f10-296e38bf1101 2022-08-01T23:57:29Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=a2a23fb6-482a-4f52-9f10-296e38bf1101 2022-08-01T23:57:30Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:30Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:30Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:30Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:31Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:31Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:31Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:31Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:32Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:32Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:33Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:33Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:33Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:33Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:33Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=6e51827c-935c-4a33-98d5-11e57e0ab9c8 2022-08-01T23:57:34Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=1535612d-13cd-4616-b223-f67663269767 src=192.168.1.66:51511 2022-08-01T23:57:34Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=1535612d-13cd-4616-b223-f67663269767 2022-08-01T23:57:34Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=1535612d-13cd-4616-b223-f67663269767 2022-08-01T23:57:34Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:34Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:35Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:35Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:36Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:36Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:37Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:37Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:38Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=f51d35bf-b2b0-4ed5-91f9-1590fcc666cd 2022-08-01T23:57:38Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=cb26f3f4-ca61-4419-8cd2-7fcf9206fac2 src=192.168.1.66:51513 2022-08-01T23:57:38Z DBG Received datagram from edge bytes=65 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:38Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:38Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=cb26f3f4-ca61-4419-8cd2-7fcf9206fac2 2022-08-01T23:57:38Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=cb26f3f4-ca61-4419-8cd2-7fcf9206fac2 2022-08-01T23:57:39Z DBG Received datagram from edge bytes=65 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:39Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:40Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:40Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:41Z DBG Received datagram from edge bytes=65 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:41Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:42Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=66a4bb39-83ec-4f9c-a37d-6ed79c73eca9 2022-08-01T23:57:42Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=c5c5053c-a9d2-4c8f-bec5-3d3e1176f6e7 src=192.168.1.66:51515 2022-08-01T23:57:42Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=c5c5053c-a9d2-4c8f-bec5-3d3e1176f6e7 2022-08-01T23:57:42Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:42Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:42Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=c5c5053c-a9d2-4c8f-bec5-3d3e1176f6e7 2022-08-01T23:57:44Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:44Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:45Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:45Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:46Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:46Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:46Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=7bfaa1dc-0655-48e3-b883-7564cd41f585 2022-08-01T23:57:47Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=e9fcfb83-08d1-421e-8c1b-10feb06a6c20 src=192.168.1.66:51517 2022-08-01T23:57:47Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=e9fcfb83-08d1-421e-8c1b-10feb06a6c20 2022-08-01T23:57:47Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=e9fcfb83-08d1-421e-8c1b-10feb06a6c20 2022-08-01T23:57:48Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:48Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:49Z DBG Received datagram from edge bytes=65 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:49Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:51Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=2cb5615d-c0e2-4366-a87a-072d0f1803f1 2022-08-01T23:57:51Z DBG Received datagram from edge bytes=66 connIndex=2 sessionID=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:51Z DBG dropped packet exceeding MTU connIndex=2 len=1281 mtu=1201 session=83453369-6d3d-4312-8a34-0e2d5f17ed82 2022-08-01T23:57:51Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=4f110684-b47f-4bcf-bb72-aadf2f62ca8d src=192.168.1.66:51519 2022-08-01T23:57:51Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=4f110684-b47f-4bcf-bb72-aadf2f62ca8d 2022-08-01T23:57:51Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=4f110684-b47f-4bcf-bb72-aadf2f62ca8d

DevinCarr commented 2 years ago

Okay, it looks like you mentioned before, the packets are exceeding the MTU of our session and are being dropped. Our MTU is 1201 and the packets are 1281.

Aeisu commented 2 years ago

i'm not too sure what i can do to alleviate the burden in order for it to accept the desired packet size.

Aeisu commented 2 years ago

thank you for all the help in figuring out the problem, one step closer to having everything running smoothly!

DevinCarr commented 2 years ago

We are currently aware of this limitation in our network for private networking and are currently looking into solutions to this. For now, we are limited to the MTU as described in the logs. Sorry!

Aeisu commented 2 years ago

I tried to limit the MTU using windows netsh but i'm not sure if it is the tunnel being funny or the netsh causing problems now. I do keep getting this error repeatedly on a normal tunnel connection and the debug connection

2022-08-02T12:47:11Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with quic protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/

abelinkinbio commented 2 years ago

Typically this debug error log indicates that cloudflared is unable to establish the necessary connections to our network over QUIC. As Devin mentioned, I'd recommend setting the protocol flag to QUIC to ensure you're able to proxy UDP traffic and to help rule out this being this issue. This behavior could be caused by a FW policy blocking UDP egress over 7844. Are these messages sporadic or fairly consistent?

Aeisu commented 2 years ago

they were consistent on startup. I added a windows FW rule for egress on ports 443 and 7844 and i haven't seen it appear again

Aeisu commented 2 years ago

this is what i'm getting in debug now

2022-08-02T14:05:40Z DBG tcp proxy stream finished successfully flowID=12916960-d759-43a3-9aa0-eec29eb28fa4 2022-08-02T14:05:40Z DBG tunnel->origin copy: readfrom tcp 192.168.1.66:50486->192.168.1.66:54231: Read on stream 201 canceled with error code 0 2022-08-02T14:05:40Z DBG tcp proxy stream started flowID=4b88d16f-6b5f-4724-83a9-bb91ac6d0aa2 2022-08-02T14:05:41Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=962647d5-ddcb-441e-a0b4-41738a2f0ca3 2022-08-02T14:05:42Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=92ea5257-eb69-4c05-9de8-1859ae8b5609 src=192.168.1.66:50892 2022-08-02T14:05:42Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=92ea5257-eb69-4c05-9de8-1859ae8b5609 2022-08-02T14:05:42Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=92ea5257-eb69-4c05-9de8-1859ae8b5609 2022-08-02T14:05:46Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=57208079-c1bc-4e2c-bbe8-3183538b2387 2022-08-02T14:05:46Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=a98fd1c9-124b-4b3b-a7d5-1e4ac63ee9cd src=192.168.1.66:50894 2022-08-02T14:05:46Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=a98fd1c9-124b-4b3b-a7d5-1e4ac63ee9cd 2022-08-02T14:05:46Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=a98fd1c9-124b-4b3b-a7d5-1e4ac63ee9cd 2022-08-02T14:05:50Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=08837664-d321-482a-998e-6aba9a366368 2022-08-02T14:05:50Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=e4ed4960-74e8-4154-9c74-d2f16919c1fc src=192.168.1.66:50896 2022-08-02T14:05:51Z DBG tcp proxy stream started flowID=f1060d2b-d8d5-4d84-b540-e95d50755aab 2022-08-02T14:05:51Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=e4ed4960-74e8-4154-9c74-d2f16919c1fc 2022-08-02T14:05:51Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=e4ed4960-74e8-4154-9c74-d2f16919c1fc 2022-08-02T14:05:54Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=dfe035c7-6e21-4891-aa10-c0c1f60fb536 2022-08-02T14:05:55Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=0820b902-56bd-4b16-a9e3-8c49f87dfdb7 src=192.168.1.66:50898 2022-08-02T14:05:55Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=0820b902-56bd-4b16-a9e3-8c49f87dfdb7 2022-08-02T14:05:55Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=0820b902-56bd-4b16-a9e3-8c49f87dfdb7 2022-08-02T14:05:59Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=cc6bc603-5aaf-4422-8455-5a21eb9d1d8e 2022-08-02T14:05:59Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=443aae52-4b10-4140-849a-025d231aafae src=192.168.1.66:50900 2022-08-02T14:05:59Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=443aae52-4b10-4140-849a-025d231aafae 2022-08-02T14:05:59Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=443aae52-4b10-4140-849a-025d231aafae 2022-08-02T14:06:02Z DBG tcp proxy stream finished successfully flowID=f1060d2b-d8d5-4d84-b540-e95d50755aab 2022-08-02T14:06:02Z DBG tunnel->origin copy: readfrom tcp 192.168.1.66:50488->192.168.1.66:54001: Read on stream 241 canceled with error code 0 2022-08-02T14:06:03Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=3da648e2-bf3a-494c-9945-47666e0d4a2f 2022-08-02T14:06:03Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=51490f92-0eca-4dee-a4a5-ec0888068cf3 src=192.168.1.66:50902 2022-08-02T14:06:04Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=51490f92-0eca-4dee-a4a5-ec0888068cf3 2022-08-02T14:06:04Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=51490f92-0eca-4dee-a4a5-ec0888068cf3 2022-08-02T14:06:07Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=5e6b37b2-3fda-4dd5-9fc7-dee424c1a6f5 2022-08-02T14:06:08Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=8d459020-ba66-4230-8a6f-09d5de2c4272 src=192.168.1.66:50904 2022-08-02T14:06:08Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=8d459020-ba66-4230-8a6f-09d5de2c4272 2022-08-02T14:06:08Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=8d459020-ba66-4230-8a6f-09d5de2c4272 2022-08-02T14:06:12Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=94954dbe-8c1e-465f-8d96-85d0a350fce8 2022-08-02T14:06:12Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=def609a8-6109-4e72-a5f8-f31489eaa6e4 src=192.168.1.66:50906 2022-08-02T14:06:12Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=def609a8-6109-4e72-a5f8-f31489eaa6e4 2022-08-02T14:06:12Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=def609a8-6109-4e72-a5f8-f31489eaa6e4 2022-08-02T14:06:16Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=669925bd-ff00-40b9-8efd-e1e5552b7159 2022-08-02T14:06:16Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=8c8c81e3-8045-4a36-b3ee-3fab8e3134d7 src=192.168.1.66:54024 2022-08-02T14:06:16Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=8c8c81e3-8045-4a36-b3ee-3fab8e3134d7 2022-08-02T14:06:17Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=8c8c81e3-8045-4a36-b3ee-3fab8e3134d7 2022-08-02T14:06:20Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=063fd760-8246-4a37-95cd-62d3fbf61f2d 2022-08-02T14:06:21Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=e2b1eae7-d039-42e4-b830-4d67322c1a35 src=192.168.1.66:54026 2022-08-02T14:06:21Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=e2b1eae7-d039-42e4-b830-4d67322c1a35 2022-08-02T14:06:21Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=e2b1eae7-d039-42e4-b830-4d67322c1a35 2022-08-02T14:06:25Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=dc5ef964-b6c3-4d5f-b594-d0a25544b91e 2022-08-02T14:06:25Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=d4122b40-c9d8-432c-9dbf-67cd70de50e8 src=192.168.1.66:54028 2022-08-02T14:06:25Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=d4122b40-c9d8-432c-9dbf-67cd70de50e8 2022-08-02T14:06:25Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=d4122b40-c9d8-432c-9dbf-67cd70de50e8 2022-08-02T14:06:29Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=a6951588-fe01-4054-9879-d8dc751e0e15 2022-08-02T14:06:29Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=dea2ac01-bfa9-4b01-a2c6-b061fd1f5acc src=192.168.1.66:54030 2022-08-02T14:06:29Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=dea2ac01-bfa9-4b01-a2c6-b061fd1f5acc 2022-08-02T14:06:29Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=dea2ac01-bfa9-4b01-a2c6-b061fd1f5acc 2022-08-02T14:06:33Z DBG Session terminated error="session closed by remote due to terminated by edge" connIndex=2 sessionID=292b80e5-0e72-4e9c-8cdb-de6df4ff992c 2022-08-02T14:06:34Z DBG Registered session connIndex=2 dst=192.168.1.177:8610 sessionID=c0dc7f10-5049-4f73-9957-e80822f0bb4a src=192.168.1.66:54033 2022-08-02T14:06:34Z DBG Received datagram from edge bytes=112 connIndex=2 sessionID=c0dc7f10-5049-4f73-9957-e80822f0bb4a 2022-08-02T14:06:34Z DBG Send datagram back to edge bytes=32 connIndex=2 sessionID=c0dc7f10-5049-4f73-9957-e80822f0bb4a

it' snot making sense that i'm not getting errors for MTU being over the limit like i was yesterday

joliveirinha commented 2 years ago

This last doesn't show anything strange. Looking into the logs in our side, all UDP sessions are closed normally from the client side (WARP), due to inactivity on the UDP session for a period of 1 minute.

I don't know the details on the protocol being used by this game but you should note the following: 1) Cloudflare Tunnel only allows traffic to be initiated from WARP (WARP -> Tunnel). This means, that returning packets need to be bound to a "flow"/"session" that was previously initiated by WARP->Tunnel-> cloudflared. 2) If the protocol happens to wait more than 1 minute without exchanging data, the session will be closed and even if the server sends back packets to cloudflared to be sent to client, those will be dropped by cloudflared since the session doesn't exist anymore. 3) Also, note that if the protocol used by the game tries to negotiate another port for the server to contact the client via that new port, that won't work also due to 1), since that by itself would be a new session initiated by the server which is not supported currently.

In conclusion, without knowing the details about the protocol, it is not easy to know if the sessions are being dropped because the game protocol waits long periods to exchange data, thus causing session to be closed. Currently we don't have support to extend this "keep-alive" period for UDP.

If you tcpdump traffic directly from client<->server (without tunnel), can you confirm those long periods of time without data, or packets being destined to client on a different port that the one initially used?

Aeisu commented 2 years ago

I have been trying to get this to work with various command line options using pktmon for windows 10 but i don't think what it is giving me is accurate to the issue at hand. here is a snippit of the realtime data. I put the server to run localy 127.0.0.1 and turned off wifi to exclude any internet traffic.

C:\Windows\system32>pktmon start -c -m real-time Collected Data: Packet counters, packet capture

Capture Type: All packets

Monitored Components: All

Packet Filters: None Processing...

11:18:29.186466000 PktGroupId 177, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.177 tell 192.168.1.66, length 28 11:18:29.186474800 Drop: PktGroupId 177, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:30.123633200 PktGroupId 562949953421461, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.177 tell 192.168.1.66, length 28 11:18:30.123640400 Drop: PktGroupId 562949953421461, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:31.123310700 PktGroupId 178, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.177 tell 192.168.1.66, length 28 11:18:31.123317300 Drop: PktGroupId 178, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:32.202933400 PktGroupId 179, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 58, LoggedSize 58 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype IPv4 (0x0800), length 58: 192.168.1.66.55611 > 255.255.255.255.8610: UDP, length 16 11:18:32.202941100 Drop: PktGroupId 179, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 58, LoggedSize 58 Drop: unknown 802.11 frame type (3) 11:18:32.203076300 PktGroupId 180, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 58, LoggedSize 58 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype IPv4 (0x0800), length 58: 192.168.1.66.55611 > 255.255.255.255.8610: UDP, length 16 11:18:32.203077200 Drop: PktGroupId 180, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 58, LoggedSize 58 Drop: unknown 802.11 frame type (3) 11:18:34.216702000 PktGroupId 181, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.161 tell 192.168.1.66, length 28 11:18:34.216710500 Drop: PktGroupId 181, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:34.431359600 PktGroupId 562949953421462, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.162 tell 192.168.1.66, length 28 11:18:34.431368200 Drop: PktGroupId 562949953421462, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:34.630387500 PktGroupId 182, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.163 tell 192.168.1.66, length 28 11:18:34.630390400 Drop: PktGroupId 182, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:34.834873900 PktGroupId 183, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.164 tell 192.168.1.66, length 28 11:18:34.834875700 Drop: PktGroupId 183, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:35.038068000 PktGroupId 184, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.165 tell 192.168.1.66, length 28 11:18:35.038070600 Drop: PktGroupId 184, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:35.117362100 PktGroupId 281474976710696, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.161 tell 192.168.1.66, length 28 11:18:35.117370300 Drop: PktGroupId 281474976710696, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:35.117378900 PktGroupId 281474976710697, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.162 tell 192.168.1.66, length 28 11:18:35.117379200 Drop: PktGroupId 281474976710697, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:35.243084400 PktGroupId 185, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.166 tell 192.168.1.66, length 28 11:18:35.243091800 Drop: PktGroupId 185, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:35.444080700 PktGroupId 186, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.167 tell 192.168.1.66, length 28 11:18:35.444082600 Drop: PktGroupId 186, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:35.614047500 PktGroupId 187, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.163 tell 192.168.1.66, length 28 11:18:35.614049600 Drop: PktGroupId 187, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:35.614052300 PktGroupId 188, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.164 tell 192.168.1.66, length 28 11:18:35.614052600 Drop: PktGroupId 188, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:35.614053300 PktGroupId 189, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.165 tell 192.168.1.66, length 28 11:18:35.614053700 Drop: PktGroupId 189, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:35.645287200 PktGroupId 190, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.168 tell 192.168.1.66, length 28 11:18:35.645289300 Drop: PktGroupId 190, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3) 11:18:35.848079700 PktGroupId 191, PktNumber 1, Appearance 1, Direction Tx , Type Ethernet , Component 54, Edge 1, Filter 0, OriginalSize 42, LoggedSize 42 08-5A-11-02-28-96 > FF-FF-FF-FF-FF-FF, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.169 tell 192.168.1.66, length 28 11:18:35.848081500 Drop: PktGroupId 191, PktNumber 1, Appearance 2, Direction Tx , Type WiFi , Component 3, Filter 0, DropReason Media Disconnected , DropLocation 0xE0001005, OriginalSize 42, LoggedSize 42 Drop: unknown 802.11 frame type (3)