search

celzero / firestack

Userspace wireguard and network monitor
https://rethinkdns.com/app
Mozilla Public License 2.0
86 stars 15 forks source link

feature request: implement Endpoint-Independent Mapping for UDP #77

Closed Lanius-collaris closed 2 weeks ago

Lanius-collaris commented 1 month ago

I know two methods:

  1. use unconnected UDP sockets
  2. bind multiple UDP sockets to one address

This mapping behavior makes NAT traversal between two rethink users possible.

ignoramous commented 1 month ago

Firestack have to give up its reliance on gonet.UDPConn (which handles outgoing unconnected UDP sockets just fine).

This is not going to be trivial though a few projects like SagerNet did once implement this (unsure if it worked as they don't seem to use it anymore).

https://github.com/SagerNet/LibSagerNetCore/blob/1fce969ea5/gvisor/udp.go (old) / https://github.com/SagerNet/sing-tun/blob/aecfc190f4/stack_gvisor_udp.go (new).

Welcome PR, for sure. :D

ignoramous commented 1 month ago

bind multiple UDP sockets to one address

Curious, how does this work?

Lanius-collaris commented 1 month ago

bind multiple UDP sockets to one address

Curious, how does this work?

Set two sockopt, SO_REUSEADDR and SO_REUSEPORT.

Lanius-collaris commented 1 month ago

Firestack have to give up its reliance on gonet.UDPConn

Why? I am not familiar with gvisor, but its document claims gonet.DialUDP() can create unconnected UDPConn. https://pkg.go.dev/gvisor.dev/gvisor@v0.0.0-20240730175157-c671ffdf47b2/pkg/tcpip/adapters/gonet#DialUDP

ignoramous commented 1 month ago

The problem isn't port forwarding (if I may call it that) using gonet.DialUDP, the problem is, the protocol handlers firestack sets up (for TCP and UDP) are not called for unconnected UDP (and listening TCP) sockets.

Firestack have to give up its reliance on gonet.UDPConn (which handles outgoing unconnected UDP sockets just fine).

Don't think even the SagerNet code I shared above would work. In that,

  1. Either: Android does not forward packets from unconnected sockets to the VPN tunnel: Testing this may be easy: Listening on unspecifiedIP:port (over both UDP, TCP) using nc+Termux, say, would not generate any relevant logs in firestack (in Rethink, Configure -> Settings -> Log level must be set to Very Verbose).
  2. Or: firestack must handle UDP on its own and not rely on netstack at all. That is, instead of delivering the UDP packet (injecting it in) to netstack: https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/netstack/fdbased.go#L505 ... deal with it as-is.
Lanius-collaris commented 1 month ago

Android does not forward packets from unconnected sockets to the VPN tunnel

This is incorrect. nc -u -l -p 12345 does not send UDP packets until netcat receives one UDP packet.

ignoramous commented 1 month ago

This is incorrect. nc -u -l -p 12345 does not send UDP packets until netcat receives one UDP packet.

Is there a way, then, for firestack to know if such a listener was started, so it can start an egress listener?

Also, I guess, I am at a loss as to what an "endpoint-independent mapping for UDP" would look like... Does it mean, instead of dialing to an (remote) endpoint, ^0 always announce UDP instead ^1?

What could we do for TCP [^2]? Port forwarding?

[^2]: Proxies (severs) currently have to be excluded for them to work: https://github.com/celzero/rethink-app/issues/1399

Lanius-collaris commented 1 month ago

Is there a way, then, for firestack to know if such a listener was started, so it can start an egress listener?

On Linux you can use inotify API to monitor /proc/net/udp and /proc/net/tcp, without some permissions this method may not work on Android.

Also, I guess, I am at a loss as to what an "endpoint-independent mapping for UDP" would look like... Does it mean, instead of dialing to an (remote) endpoint, 1 always announce UDP instead 2?

Yes.

What could we do for TCP 3? Port forwarding?

Footnotes

1. https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/udp.go#L352 [↩](#user-content-fnref-0-044726a13dad48d457abd69cf7035293)

2. https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/udp.go#L345 [↩](#user-content-fnref-1-044726a13dad48d457abd69cf7035293)

3. Proxies (severs) currently _have_ to be excluded for them to work: https://github.com/celzero/rethink-app/issues/1399 [↩](#user-content-fnref-2-044726a13dad48d457abd69cf7035293)

Allow users to write some port forwarding rules? ( e.g. TCP 0.0.0.0:12346 -> 127.0.0.1:12345 )

ignoramous commented 1 month ago

Yes.

Guess, Proxy.Announce would need to be paired with gonet.DialUDP instead of gonet.NewUDPConn?

https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/netstack/udp.go#L149

Linux ... inotify ... procfs

I guess Netlink cmds might work too (would be surprised if they worked on Android but it might).

Allow users to write some port forwarding rules? (TCP 0.0.0.0:12346 -> 127.0.0.1:12345)

SagerNet seems to be doing some stuff with this, but I am not sure exactly what it does:

Lanius-collaris commented 1 month ago

Guess, Proxy.Announce would need to be paired with gonet.DialUDP instead of gonet.NewUDPConn?

https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/netstack/udp.go#L149

gonet.DialUDP() calls gonet.NewUDPConn() internally. See https://github.com/google/gvisor/blob/4542eb5ba62b/pkg/tcpip/adapters/gonet/gonet.go#L588

I guess Netlink cmds might work too (would be surprised if they worked on Android but it might).

Ah, I think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes don't have the same effective UID.

Allow users to write some port forwarding rules? (TCP 0.0.0.0:12346 -> 127.0.0.1:12345)

SagerNet seems to be doing some stuff with this, but I am not sure exactly what it does:

* (multicast filter) [SagerNet/sing-tun@150b116](https://github.com/SagerNet/sing-tun/commit/150b1162316cb63bbbf5c3c01f3a2b494a56e72d)

* (broadcast filter) [SagerNet/sing-tun@da350ec](https://github.com/SagerNet/sing-tun/commit/da350ecc726daa502b4256a7a70d70a4c6cde6d4)

I guess these two commits allow packets having multicast or broadcast destination address to bypass the userspace netstack.

A off topic question: Why does firestack send SYN+ACK packets immediately after receiving SYN packets?

ignoramous commented 1 month ago

gonet.DialUDP() calls gonet.NewUDPConn() internally. See: https://github.com/google/gvisor/blob/4542eb5ba62b/pkg/tcpip/adapters/gonet/gonet.go#L588

I think, firestack needs to create a new gonet.UDPConn without using udp.ForwarderRequest.CreateEndpoint + gonet.NewUDPConn (as that makes a connected UDP socket, per docs): https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/netstack/udp.go#L143

think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes

Believe apps part of the VPN tunnel in Android are in a different network namespace and as such must not be able to stomp on the VPN app's sockets?

Using the other option (unconnected UDP over sockopt+reuse) the VPN app NATs the traffic from its listening port (bound to actual underlying network) onto the same port bound to the TUN device, if any. Which is the original proposal, that would work, would it not?

Why does firestack send SYN+ACK packets immediately after receiving SYN packets

netstack's gonet API requires for its internal book-keeping (that is, routing subsequent packets from the same TCP flow to an already established gonet.TCPConn).

https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/netstack/tcp.go#L116-L117

Does this break anything apart from confusing apps into thinking they've connected to remote while they may infact be blocked later due to a firewall rule?

Lanius-collaris commented 1 month ago

think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes

Believe apps part of the VPN tunnel in Android are in a different network namespace and as such must not be able to stomp on the VPN app's sockets?

Using the other option (unconnected UDP over sockopt+reuse) the VPN app NATs the traffic from its listening port (bound to actual underlying network) onto the same port bound to the TUN device, if any. Which is the original proposal, that would work, would it not?

I don't know if apps on android are in different namespaces, 😂 but on android every app has its own user, you can check with adb shell sh -c "ps -d|cat -" . The reason why I mentioned the setsockopt trick a few days ago is just I thought you might not like unconnected sockets. Please read https://man.voidlinux.org/man7/socket.7#SO_REUSEPORT

ignoramous commented 1 month ago

Thanks. Now, I recall reading a Cloudflare blog post about so_reuseport (mirror).

Possibly the biggest downside, though, is that a service listening on the wildcard INADDR_ANY address claims the port number exclusively for itself. Binding over the wildcard-listening socket with a specific IP and port fails miserably due to the address already being taken (EADDRINUSE).

bind(3, {sa_family=AF_INET, sin_port=htons(12345), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
bind(4, {sa_family=AF_INET, sin_port=htons(12345), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 
EADDRINUSE (Address already in use)

Unless your service is UDP-only, setting the SO_REUSEADDR socket option, will not help you overcome this restriction. The only way out is to turn to SO_REUSEPORT, normally used to construct a load-balancing socket group. And that is only if you are lucky enough to run the port-conflicting services as the same user (UID). That is a story for another post.

The reason why I mentioned the setsockopt trick a few days ago is just I thought you might not like unconnected sockets.

I don't like (or understand much of) networking in general, but here we are ;)

(but no, I don't hold any special grudge on unconnected sockets)

I don't know if apps on android are in different namespaces

I am not sure, either (though this must be easy to find out...). Even so, the apps (by default) can't really bind to underlying interfaces (wifi, mobile, usb, etc) and so them listening on a wildcard addr:port (say tcp/udp [::]:8080) means listening on (tcp/udp) packets sent to 8080 to the TUN device only? I take what you're saying is, despite that, the Kernel wouldn't let both (the VPN app and the installed app) bind to the same port (8080, in this case) regardless of routing rules (as the effective UIDs are different), yeah?.

Btw, checking if you working on this change (so we both don't end up impl it)?

Lanius-collaris commented 1 month ago

I am not sure, either (though this must be easy to find out...). Even so, the apps (by default) can't really bind to underlying interfaces (wifi, mobile, usb, etc) and so them listening on a wildcard addr:port (say tcp/udp [::]:8080) means listening on (tcp/udp) packets sent to 8080 to the TUN device only?

Not only the TUN device, they can also receive packets sent to [::1]:8080 and [ the Link-Local Address of dummy0 ]:8080.

I take what you're saying is, despite that, the Kernel wouldn't let both (the VPN app and the installed app) bind to the same port (8080, in this case) regardless of routing rules (as the effective UIDs are different), yeah?.

Try this:

package main

import (
    "flag"
    "fmt"
    "syscall"
    "time"
)

const SO_REUSEPORT = 15

func p(e error) {
    if e != nil {
        panic(e)
    }
}
func main() {
    var localAddr syscall.SockaddrInet6
    flag.IntVar(&localAddr.Port, "listen-port", 65535, "")
    flag.Parse()

    tcpSock, err := syscall.Socket(syscall.AF_INET6, syscall.SOCK_STREAM, syscall.IPPROTO_TCP)
    p(err)
    defer syscall.Close(tcpSock)
    err = syscall.SetsockoptInt(tcpSock, syscall.SOL_SOCKET, syscall.SO_REUSEADDR, 1)
    p(err)
    err = syscall.SetsockoptInt(tcpSock, syscall.SOL_SOCKET, SO_REUSEPORT, 1)
    p(err)
    err = syscall.Bind(tcpSock, &localAddr)
    p(err)
    err = syscall.Listen(tcpSock, 8)
    p(err)

    nfd, from, err := syscall.Accept(tcpSock)
    p(err)
    defer syscall.Close(nfd)
    fmt.Printf("accept: nfd: %v ; from: %v\n", nfd, from)
    fmt.Println("sleeping...")
    time.Sleep(600 * time.Second)
}

Btw, checking if you working on this change (so we both don't end up impl it)?

I am not working on this change.

ignoramous commented 1 month ago

Related:

ignoramous commented 2 weeks ago

Not only the TUN device, they can also receive packets sent to [::1]:8080 and [ the Link-Local Address of dummy0 ]:8080.

On Android, unsure if link-local gets sent to TUN (lo isn't, Wifi Calling isn't, Tethering isn't).

hussainmohd-a commented 1 week ago

f78f98d522be5fc4 is demuxed to 1.1.1.1, 8.8.8.8, 9.9.9.9, and 8.8.4.4 endpoints from :37288.

2024-08-30 21:32:07.422 28538-2688  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.422 28538-2688  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:07.423 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.423 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.8.8:53
2024-08-30 21:32:07.423 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.423 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=1.1.1.1:53
2024-08-30 21:32:07.426 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.426 28538-2692  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=9.9.9.9:53
2024-08-30 21:32:07.447 28538-2692  GoLog                   com.celzero.bravedns                 I  protect.go:67: D control: netbinder: Exit: udp4(0.0.0.0:0); err? <nil>
2024-08-30 21:32:07.450 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:497: I udp: mux: f78f98d522be5fc4 new assoc for 10.111.222.1:40413
2024-08-30 21:32:07.450 28538-2697  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 8.8.8.8:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 1} {{} 0} {{} 0}}
2024-08-30 21:32:07.450 28538-2697  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 8.8.8.8:53
2024-08-30 21:32:07.451 28538-3152  GoLog                   com.celzero.bravedns                 E  udp.go:190: E ns: udp: dial: endpoint for 10.111.222.1:40413 => 9.9.9.9:53; err(connect udp 10.111.222.1:40413: port is in use)
2024-08-30 21:32:07.451 28538-3152  GoLog                   com.celzero.bravedns                 E  udp.go:147: E ns: udp: demuxer: dial: connect udp 10.111.222.1:40413: port is in use; src(10.111.222.1:40413) dst(9.9.9.9:53)
2024-08-30 21:32:07.451 28538-3152  GoLog                   com.celzero.bravedns                 E  udpmux.go:261: E udp: mux: f78f98d522be5fc4 route: vend failure 9.9.9.9:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.451 28538-3152  GoLog                   com.celzero.bravedns                 E  udp.go:190: E ns: udp: dial: endpoint for 10.111.222.1:40413 => 1.1.1.1:53; err(connect udp 10.111.222.1:40413: port is in use)
2024-08-30 21:32:07.451 28538-3152  GoLog                   com.celzero.bravedns                 E  udp.go:147: E ns: udp: demuxer: dial: connect udp 10.111.222.1:40413: port is in use; src(10.111.222.1:40413) dst(1.1.1.1:53)
2024-08-30 21:32:07.452 28538-2695  GoLog                   com.celzero.bravedns                 E  udp.go:140: W udp: proxy: 10.111.222.1:40413 ->  9.9.9.9:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.452 28538-2695  GoLog                   com.celzero.bravedns                 E  udpmux.go:261: E udp: mux: f78f98d522be5fc4 route: vend failure 1.1.1.1:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.452 28538-2692  GoLog                   com.celzero.bravedns                 E  udp.go:190: E ns: udp: dial: endpoint for 10.111.222.1:40413 => 8.8.4.4:53; err(connect udp 10.111.222.1:40413: port is in use)
2024-08-30 21:32:07.452 28538-2692  GoLog                   com.celzero.bravedns                 E  udp.go:147: E ns: udp: demuxer: dial: connect udp 10.111.222.1:40413: port is in use; src(10.111.222.1:40413) dst(8.8.4.4:53)
2024-08-30 21:32:07.452 28538-2692  GoLog                   com.celzero.bravedns                 E  udp.go:140: W udp: proxy: 10.111.222.1:40413 ->  8.8.4.4:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.452 28538-2692  GoLog                   com.celzero.bravedns                 E  udpmux.go:261: E udp: mux: f78f98d522be5fc4 route: vend failure 8.8.4.4:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.453 28538-3095  GoLog                   com.celzero.bravedns                 E  udp.go:140: W udp: proxy: 10.111.222.1:40413 ->  1.1.1.1:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.453 28538-2696  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: f78f98d522be5fc4 (proxy? Exit@127.0.0.127:1337) 0.0.0.0:37288 -> 8.8.8.8:53/8.8.8.8:53; mux? true, uid 10671
2024-08-30 21:32:07.453 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 9.9.9.9:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 2} {{} 0} {{} 0}}
2024-08-30 21:32:07.453 28538-2696  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: 20790a938f64d070 (proxy? Exit@127.0.0.127:1337) 0.0.0.0:37288 -> 9.9.9.9:53/9.9.9.9:53; mux? true, uid 10671
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 9.9.9.9:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 1.1.1.1:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 3} {{} 0} {{} 61}}
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: 5c44094c5361f93f (proxy? Exit@127.0.0.127:1337) 0.0.0.0:37288 -> 1.1.1.1:53/1.1.1.1:53; mux? true, uid 10671
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 9.9.9.9:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 1.1.1.1:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:387: I udp: mux: f78f98d522be5fc4 demux from 0.0.0.0:37288 => 9.9.9.9:53 closed
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 9.9.9.9:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 1.1.1.1:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:387: I udp: mux: f78f98d522be5fc4 demux from 0.0.0.0:37288 => 1.1.1.1:53 closed
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 8.8.4.4:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 4} {{} 0} {{} 183}}
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 9.9.9.9:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: acbd3729b62afe4f (proxy? Exit@127.0.0.127:1337) 0.0.0.0:37288 -> 8.8.4.4:53/8.8.4.4:53; mux? true, uid 10671
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:276: I udp: mux: f78f98d522be5fc4 unrouting... 0.0.0.0:37288 => 1.1.1.1:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 1.1.1.1:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 9.9.9.9:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:387: I udp: mux: f78f98d522be5fc4 demux from 0.0.0.0:37288 => 8.8.4.4:53 closed
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:276: I udp: mux: f78f98d522be5fc4 unrouting... 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 1.1.1.1:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 1.1.1.1:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=9.9.9.9:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:276: I udp: mux: f78f98d522be5fc4 unrouting... 0.0.0.0:37288 => 9.9.9.9:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=1.1.1.1:53
2024-08-30 21:32:07.506 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:07.507 28538-3095  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 1.1.1.1:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 5} {{} 264} {{} 183}}
2024-08-30 21:32:07.507 28538-3095  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 1.1.1.1:53
2024-08-30 21:32:07.508 28538-3095  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.508 28538-3095  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=1.1.1.1:53
2024-08-30 21:32:07.511 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 9.9.9.9:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 6} {{} 396} {{} 183}}
2024-08-30 21:32:07.511 28538-2692  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.511 28538-2692  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=9.9.9.9:53
2024-08-30 21:32:07.512 28538-2697  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 9.9.9.9:53
2024-08-30 21:32:07.525 28538-3095  GoLog                   com.celzero.bravedns                 I  protect.go:67: D control: netbinder: Exit: udp4(9.9.9.9:53); err? <nil>
2024-08-30 21:32:07.527 28538-2692  GoLog                   com.celzero.bravedns                 I  protect.go:67: D control: netbinder: Exit: udp4(1.1.1.1:53); err? <nil>
2024-08-30 21:32:07.529 28538-2692  GoLog                   com.celzero.bravedns                 I  exit.go:53: I proxy: exit: dial(udp) to 1.1.1.1:53; err? <nil>
2024-08-30 21:32:07.529 28538-2692  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: 0c296b22bbd49481 (proxy? Exit@127.0.0.127:1337) 10.1.204.164:42507 -> 1.1.1.1:53/1.1.1.1:53; mux? false, uid 10671
2024-08-30 21:32:07.529 28538-2692  GoLog                   com.celzero.bravedns                 I  exit.go:53: I proxy: exit: dial(udp) to 9.9.9.9:53; err? <nil>
2024-08-30 21:32:07.530 28538-2695  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: 1f41c4b4d08808b3 (proxy? Exit@127.0.0.127:1337) 10.1.204.164:45931 -> 9.9.9.9:53/9.9.9.9:53; mux? false, uid 10671
2024-08-30 21:32:08.522 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:08.522 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:08.545 28538-2695  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 8.8.4.4:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 7} {{} 396} {{} 244}}
2024-08-30 21:32:08.545 28538-2695  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:08.545 28538-2695  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: 0b2a4b8786a59117 (proxy? Exit@127.0.0.127:1337) 0.0.0.0:37288 -> 8.8.4.4:53/8.8.4.4:53; mux? true, uid 10671
2024-08-30 21:32:08.545 28538-2695  GoLog                   com.celzero.bravedns                 E  udp.go:190: E ns: udp: dial: endpoint for 10.111.222.1:40413 => 8.8.4.4:53; err(connect udp 10.111.222.1:40413: port is in use)
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 E  udp.go:147: E ns: udp: demuxer: dial: connect udp 10.111.222.1:40413: port is in use; src(10.111.222.1:40413) dst(8.8.4.4:53)
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 I  udpmux.go:387: I udp: mux: f78f98d522be5fc4 demux from 0.0.0.0:37288 => 8.8.4.4:53 closed
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 E  udpmux.go:261: E udp: mux: f78f98d522be5fc4 route: vend failure 8.8.4.4:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 E  udp.go:140: W udp: proxy: 10.111.222.1:40413 ->  8.8.4.4:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 I  udpmux.go:276: I udp: mux: f78f98d522be5fc4 unrouting... 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:08.546 28538-2688  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:08.546 28538-2688  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:08.546 28538-3107  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:08.574 28538-2695  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:08.589 28538-2697  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 8.8.4.4:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 8} {{} 660} {{} 305}}
2024-08-30 21:32:08.589 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:08.590 28538-2692  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:08.590 28538-2692  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:08.609 28538-2692  GoLog                   com.celzero.bravedns                 I  protect.go:67: D control: netbinder: Exit: udp4(8.8.4.4:53); err? <nil>
2024-08-30 21:32:08.614 28538-2697  GoLog                   com.celzero.bravedns                 I  exit.go:53: I proxy: exit: dial(udp) to 8.8.4.4:53; err? <nil>
2024-08-30 21:32:08.614 28538-2697  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: 5049098b0031bfbe (proxy? Exit@127.0.0.127:1337) 10.1.204.164:42248 -> 8.8.4.4:53/8.8.4.4:53; mux? false, uid 10671
2024-08-30 21:32:08.843 28538-3107  GoLog                   com.celzero.bravedns                 I  common.go:50: D intra: 8fec78d73c5a1f36 download(792) done(read udp 10.1.204.164:41362->8.8.8.8:53: i/o timeout) b/w a(10.111.222.1:35376->8.8.8.8:53) => b(10.1.204.164:41362<-8.8.8.8:53)
2024-08-30 21:32:09.620 28538-3107  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:10.616 28538-3107  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:11.227 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:198: I udp: mux: 78140498e6950c35 read timeout(1): read udp4 0.0.0.0:45297: i/o timeout
2024-08-30 21:32:11.227 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:198: I udp: mux: 78140498e6950c35 read timeout(2): read udp4 0.0.0.0:45297: i/o timeout
2024-08-30 21:32:11.227 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:203: I udp: mux: 78140498e6950c35 read done n(0): read udp4 0.0.0.0:45297: i/o timeout
2024-08-30 21:32:11.227 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: 78140498e6950c35 demux 0.0.0.0:45297 => 100.75.223.185:443 close, in: 0, over: 0
2024-08-30 21:32:11.227 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:387: I udp: mux: 78140498e6950c35 demux from 0.0.0.0:45297 => 100.75.223.185:443 closed
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:276: I udp: mux: 78140498e6950c35 unrouting... 0.0.0.0:45297 => 100.75.223.185:443
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:142: D udp: mux: 78140498e6950c35 stop
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:133: I udp: mux: 78140498e6950c35 awaiter: done
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:164: I udp: mux: 78140498e6950c35 drain: closing 0 demuxed conns
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:153: I udp: mux: 78140498e6950c35 stopped; stats: tx: 0, rx: 6250, conns: 1, dur: 123s
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: 78140498e6950c35 demux 0.0.0.0:45297 => 100.75.223.185:443 close, in: 0, over: 0
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: 78140498e6950c35 demux 0.0.0.0:45297 => 100.75.223.185:443 close, in: 0, over: 0
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:510: I udp: mux: 78140498e6950c35 (Exit) dissoc for 10.111.222.1:36198
2024-08-30 21:32:11.629 28538-2697  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:12.642 28538-3107  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:13.656 28538-3107  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)