hussainmohd-a
commented
2 months ago Open hussainmohd-a opened 2 months ago
hussainmohd-a
commented
2 months ago 
ignoramous
commented
2 months ago # recvmsg always fails, TTL set to default 3
08-12 18:17:43.022 18271 13867 I GoLog : split_and_desync.go:231: V split-desync: recvmsg 142.250.195.74:443 failed: resource temporarily unavailable
08-12 18:17:43.056 18271 13865 I GoLog : split_and_desync.go:262: D split-desync: done: 142.250.195.106:443, ok? true, ttl: 3
...
# desync write with default TTL 3 is ok
08-12 18:17:43.057 18271 14135 I GoLog : split_and_desync.go:431: D split-desync: write: n1: 100, n2: 487, err: <nil>
...
# read / write cycle is ok
08-12 18:17:43.057 18271 14135 I GoLog : retrier.go:422: D rdial: copyOnce: rw [10.1.204.164:57700->142.250.195.106:443] 587/587; err <nil>
08-12 18:17:43.057 18271 14135 I GoLog : split_and_desync.go:439: D split-desync: readfrom: copyOnce; sz: 587; err: <nil>
# download (remote -> local) always fails
08-12 18:17:43.060 18271 14134 I GoLog : common.go:51: D intra: d4a825573852a95f download(0) done(read tcp4 10.1.204.164:57700->142.250.195.106:443: read: connection reset by peer) b/w a(10.111.222.1:44394->142.250.195.42:443) => b(10.1.204.164:57700<-142.250.195.106:443)
Happens on Android 10 (v: Linux version 4.9.186-22990479 (dpi@21HHAG04) (gcc version 4.9.x 20150123 (prerelease) (GCC) ) #1 SMP PREEMPT Thu Mar 31 15:41:33 KST 2022)
cc: @Lanius-collaris any clue?
Lanius-collaris
commented
2 months ago @ignoramous Looks like not same as hussainmohd-a's logs (sendto unreachable) . Is the device a hardware device? The emulator of Android Studio doesn't support UDP traceroute, and it is not possible to launch TCB Desynchronization in it (Not all packets will be forwarded and its networking tool always sends packets with default TTL) . If the device is not virtual, maybe the sorcery doesn't work on your device (I found that OPPO A57 doesn't support), but traceroute should work. I haven't tested 1a403a4, so my theory isn't based on 1a403a4
Lanius-collaris
commented
2 months ago Two ways to check if the sorcery works
Http1_1String two times, the sorcery doesn't work.Http1_1String, the sorcery doesn't work.ignoramous
commented
2 months ago Thanks.
If the device is not virtual, maybe the sorcery doesn't work on your device (I found that OPPO A57 doesn't support)
If the client doesn't support, then the error must be on the client (and not a RST from the server like it happens in this case)? In the current scenario, desync fails only on this one Samsung S8 device running Android 10, but works on other test devices we have (all running Android 13 or 14). Perplexing, esp in absence of any client-side errors.
Two ways to check if the sorcery works ... Run tcpdump on Android
Unrooted... will see if pcap shows anything.
if the server receives Http1_1String, the sorcery doesn't work
I doubt if the server receives the fake payload. In this case, recvmsg fails with the TTL set to default (3). Unless server is within 3 hops of the client (unlikely for ALL egress), I wonder if it sees it.
unrelated; why copy the first split of the payload on to firstSegment if it is never sent (not with unix.Sendfile nor with conn.Write)? I see byedpi does the same.
Lanius-collaris
commented
2 months ago unrelated; why copy the first split of the payload on to
firstSegmentif it is never sent (not withsendfilenor withconn.Write)?
Just manipulate the buffer to change the payload for retransmission, because we can't contruct IP packets directly. Calling sendfile or conn.Write can't send a segment with previous SEQ.
If the client doesn't support, then the error must be on the client (and not a RST from the server like it happens in this case)?
RST may be from TUN. Could you capture packets on a router? I guess that firestack failed to overwrite kernel buffer, google's server received the HTTP request sent to port 443, then closed the connection.
But traceroute shouldn't fail, does UDP traceroute work on this device? (You can install traceroute in termux).
In this case, recvmsg fails with the TTL set to default (
3)
Another possibility is that only ICMP messages from first three hops were received, but it's not likely.
Lanius-collaris
commented
2 months ago | - | - |
|---|---|
emulate sendfile(2) with splice(2) |
still fail on 4.9.x |
MSG_ZEROCOPY flag |
require newer kernel |
😞
Lanius-collaris
commented
1 week ago I'd like rethink-app to tell users this issue when they enable it on kernel older than 4.14. Thanks.
ignoramous
commented
1 week ago I'd like rethink-app to tell users this issue when they enable it on kernel older than 4.14. Thanks.
Yes, we'll do so before releasing v055o (which is still undergoing major changes).
emulate sendfile(2) with splice(2)
We could still PR this, if it lets us support more Kernel versions?
Lanius-collaris
commented
5 days ago emulate sendfile(2) with splice(2)
We could still PR this, if it lets us support more Kernel versions?
There is no LTS Linux kernel between 4.9 and 4.14, did you see android devices using non-LTS kernel?
ignoramous
commented
5 days ago you see android devices using non-LTS kernel?
TIL. https://source.android.com/docs/core/architecture/kernel/linux-stable-merges
Lanius-collaris
commented
4 days ago overwriteSplitter.Write() sends original data twice rather than "payload then original data" on some Linux distribution (although I can't reproduce on android), I suggest inserting time.Sleep(time.Microsecond) in https://github.com/celzero/firestack/blob/4f43cdb3ba7165798261aa680b97e5368ae05068/intra/dialers/split_and_desync.go#L454
byedpi's wait_send() is a bit complex, I ignored it before… Sorry.
ignoramous
commented
4 days ago I suggest inserting time.Sleep(time.Microsecond)
byedpi's default delay is 3ms (sec?) (ref). That works?
overwriteSplitter.Write() sends original data twice rather than "payload then original data" on some Linux distribution
How does the delay result in the original payload being memcp/cp'd (as we'd expect it to), though?
ignoramous
commented
3 days ago I suggest inserting time.Sleep(time.Microsecond) in
https://github.com/celzero/firestack/commit/a7d3fc2657680b6e784e37e54b9243d77025f314 (a delay of 3ms)
Lanius-collaris
commented
3 days ago payload: "haha", 1st segment: "1234", 2nd segment: "56\r\n" I have seen three kinds of behavior. 3pcap.zip
Linux localhost 6.6.28-0-virt #1-Alpine SMP PREEMPT_DYNAMIC Wed, 17 Apr 2024 19:38:57 +0000 x86_64 LinuxIn this environment, 1 microsecond is enough, it's confusing.
logs
logs: ```jsx I bootstrap.go:140: I dns: default: DNS reinit rethink w/ 8.8.4.4,2001:4860:4860::8844 I bootstrap.go:68: I dns: default: DNS new 8.8.4.4,2001:4860:4860::8844 rethink I natpt.go:51: I natpt: new; mode(&{{{} 1} {{} 0} {{} 2}}) I proxies.go:157: I proxy: adding reserved proxy: Exit I proxies.go:157: I proxy: adding reserved proxy: Base I proxies.go:157: I proxy: adding reserved proxy: Block I proxies.go:148: I proxy: new I transport.go:910: I dnsx: bootstrap! rethink:53 -> [8.8.4.4 2001:4860:4860::8844] I ipmap.go:238: D ipmap: renew: rethink:53 / seed: [8.8.4.4 2001:4860:4860::8844] / typ: Protected I ipmap.go:264: D ipmap: makeIPSet: rethink, seed: [8.8.4.4 2001:4860:4860::8844], typ: Protected I upstream.go:101: I dns53: (Bootstrap) pre-resolved rethink:53 to 8.8.4.4,2001:4860:4860::8844; ok? true I upstream.go:111: I dns53: (Bootstrap) setup: rethink:53; pre-ips? true; relay? false I bootstrap.go:186: I dns: default: start; DNS with rethink[d.rethink.rethink:53]; ok? true I alg.go:141: I alg: setup done I cacher.go:124: I cache: (CacheDefault) setup: cached.d.rethink.rethink:53; opts: ttl=10m0s;bumps=10;size=256 I transport.go:183: I dns: new! gw? true; default? d.rethink.rethink:53 I goos.go:63: I dns53: goosr: setup done I cacher.go:124: I cache: (CacheGoos) setup: cached.localhost:53; opts: ttl=10m0s;bumps=10;size=256 I transport.go:257: I dns: add transport Goos@localhost:53; cache? true I grounded.go:35: I grounded(BlockAll) setup: 127.0.0.3:53 I cacher.go:124: I cache: (CacheBlockAll) setup: cached.127.0.0.3:53; opts: ttl=10m0s;bumps=10;size=256 I transport.go:257: I dns: add transport BlockAll@127.0.0.3:53; cache? true I servers.go:160: D dnscrypt: refreshing certificates I cacher.go:124: I cache: (CacheDcProxy) setup: cached.; opts: ttl=10m0s;bumps=10;size=256 I transport.go:257: I dns: add transport DcProxy@; cache? true I mdns.go:67: I mdns: setup: 46 I cacher.go:124: I cache: (Cachemdns) setup: cached.224.0.0.251:5353; opts: ttl=10m0s;bumps=10;size=256 I transport.go:257: I dns: add transport mdns@224.0.0.251:5353; cache? true I ips.go:122: I dialers: ips: mapper ok? true I multiserver.go:459: I dnscrypt: refreshRoutes: remove all relays I ipmap.go:135: I ipmap: new resolver; ok? true I servers.go:106: V dnscrypt: getAll: servers [0/0] I multiserver.go:471: I dnscrypt: refreshRoutes: 0/0 for 0 servers I dns64.go:280: I dns64: ipv4only.arpa w underlying network resolver I dns64.go:313: D dns64: id(__local464); add? nat64 ip(64:ff9b:1:fffe::c000:aa / 16) E dns64.go:83: W dns64: err reg underlay(address ipv4only.arpa.: no suitable address found) / local(