completely broken on ami linux: getPidByAddr failed

[CaledoniaProject]

I'm running Amazon Linux release 2023.4.20240513 (Amazon Linux), and graftcp suddenly stopped working, looks like it failed to associate the socket with the PID

# ./mgraftcp --select_proxy_mode=only_socks5 --socks5 127.0.0.1:9000 --enable-debug-log curl ifconfig.me -vv
[2024-05-23 09:38:33] [INFO] graftcp-local start listening :0...
* Host ifconfig.me:80 was resolved.
* IPv6: 2600:1901:0:bbc3::
* IPv4: 34.117.118.44
*   Trying 34.117.118.44:80...
* Connected to ifconfig.me (127.0.0.1) port 80
> GET / HTTP/1.1
> Host: ifconfig.me
> User-Agent: curl/8.5.0
> Accept: */*
>
[2024-05-23 09:38:33] [ERROR] getPidByAddr(172.x.x.x:56982, 127.0.0.1:36967) failed
* Recv failure: Connection reset by peer
* Closing connection
curl: (56) Recv failure: Connection reset by peer

Any suggestions? proxychains works fine.

[hmgle]

Could you provide the version information for curl? I'm not sure whether modifying the address parameter of the connect(2) will impact curl. It's possible that curl checks this parameter after the connect syscall returns and closes the socket, so that mgraftcp fails to obtain its PID and target address information through procfs. Perhaps adding a connect_exiting_handle function to restore the connect address parameter might be a solution.

[hmgle]

I've reviewed the code and found that this error occurs before the connect(2) syscall returns, so it's not for the reason mentioned above but due to some other cause.

[CaledoniaProject]

@hmgle Sure, but how can I help you? perhaps you can create a new branch and add several debug output?

[hmgle]

@CaledoniaProject I have added some logs to track the PIDs and their IP address infomation in local, and pushed them to a new branch(debug-local). You can switch to this new branch and try to see if the bug can be reproduced. Thank you.

[CaledoniaProject]

@hmgle see if this helps

~/graftcp/local # ./mgraftcp --select_proxy_mode=only_socks5 --socks5 127.0.0.1:3080 --enable-debug-log curl ifconfig.me
[2024-05-24 21:26:30] [INFO] graftcp-local start listening :0...
[2024-05-24 21:26:30] [DEBUG] connect req[dest-addr:pid]: 34.117.118.44:80:243036
[2024-05-24 21:26:30] [DEBUG] StorePidAddr(243036, 34.117.118.44:80)
[2024-05-24 21:26:30] [DEBUG] ----- pidAddrMap: {
        243036: 34.117.118.44:80
}
[2024-05-24 21:26:30] [ERROR] getInodeByAddrs(172.x.x.x:59160, 127.0.0.1:33023) err: no inode for [xxxxx:E718 0100007F:80FF]
[2024-05-24 21:26:30] [DEBUG] pidMap: pidAddrMap: {
        243036: 34.117.118.44:80
}
[2024-05-24 21:26:30] [ERROR] getPidByAddr(172.x.x.x:59160, 127.0.0.1:33023) failed
curl: (56) Recv failure: Connection reset by peer

[hmgle]

@CaledoniaProject Thank you for your feedback. Is this bug consistently or reliably reproducible for curl in the Amazon Linux? Does the same issue occur with other commands, such as wget? I ran the same version curl/8.5.0 on my machine, but couldn't reproduce the issue. Maybe I need to add more logs to pinpoint this problem.

[CaledoniaProject]

@hmgle 100% reproducible, it just suddenly failed to work. I don't know why, but /proc works fine. I added retry to curl, and tried to capture things in /proc/pid/fd, unlikely to be helpful.

Also tried wget, same error for inode.

[CaledoniaProject]

At first I thought curl might be exiting/closing the socket too fast, however, I have the same error for ncat

# ./mgraftcp --select_proxy_mode=only_socks5 --socks5 127.0.0.1:3080 --enable-debug-log nc google.com 80 -vv
[2024-05-24 22:22:51] [INFO] graftcp-local start listening :0...
Ncat: Version 7.93 ( https://nmap.org/ncat )
NCAT DEBUG: Using system default trusted CA certificates and those in /usr/share/ncat/ca-bundle.crt.
NCAT DEBUG: Unable to load trusted CA certificates from /usr/share/ncat/ca-bundle.crt: error:80000002:system library::No such file or directory
libnsock nsock_iod_new2(): nsock_iod_new (IOD #1)
libnsock nsock_connect_tcp(): TCP connection requested to 142.251.220.78:80 (IOD #1) EID 8
libnsock nsock_trace_handler_callback(): Callback: CONNECT SUCCESS for EID 8 [142.251.220.78:80]
Ncat: Connected to 142.251.220.78:80.
libnsock nsock_iod_new2(): nsock_iod_new (IOD #2)
libnsock nsock_read(): Read request from IOD #1 [142.251.220.78:80] (timeout: -1ms) EID 18
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 26
[2024-05-24 22:22:51] [DEBUG] ----- pidAddrMap: {
}
[2024-05-24 22:22:51] [ERROR] getInodeByAddrs(172.x.x.x:44082, 127.0.0.1:45709) err: no inode for [xxx:AC32 0100007F:B28D]
[2024-05-24 22:22:51] [DEBUG] pidMap: pidAddrMap: {
}
[2024-05-24 22:22:51] [DEBUG] connect req[dest-addr:pid]: 142.251.220.78:80:246075
[2024-05-24 22:22:51] [DEBUG] StorePidAddr(246075, 142.251.220.78:80)
[2024-05-24 22:22:51] [ERROR] getPidByAddr(172.x.x.x:44082, 127.0.0.1:45709) failed
libnsock nsock_trace_handler_callback(): Callback: READ EOF for EID 18 [142.251.220.78:80]

[hmgle]

I think I have found out the cause of this issue.

The Amazon Linux host you are using seems to have rewritten packets with a source address of 127.0.0.1 to "172.x.x.x". Maybe something like iptables or nftables did it. In that case, the /proc/net/tcp entry for this socket will be HEX(127.0.0.1), but not HEX(172.x.x.x). Setting localAddr to "127.0.0.1" in the code's getInodeByAddrs function should correctly handle this situation, but I still need to evaluate whether it's necessary to do so.

I successfully reproduced this issue by adding the following iptables rule:

sudo iptables -t nat -A POSTROUTING -s 127.0.0.1 -d 127.0.0.1 -j SNAT --to-source 172.x.x.x

This is just one possible cause; there might be other ways that could affect the graftcp-local service's accurate acquisition of the client IP.

[CaledoniaProject]

Indeed this works,

func getPidByAddr(localAddr, remoteAddr string, isTCP6 bool) (pid string, destAddr string) {
           parts := strings.Split(localAddr, ":")
           parts[0] = "127.0.0.1"
           localAddr = strings.Join(parts, ":")

           ...
}

I do have iptables DNAT rules, but not for localhost, both the SRC and DEST are public addresses,

# iptables-save
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -s xxxxx/32 -p tcp -m tcp --dport xxxx -j DNAT --to-destination xxxx:xxxx
-A POSTROUTING -j MASQUERADE
COMMIT

relevant sysctl settings

net.ipv4.conf.all.route_localnet = 0
net.ipv4.ip_forward = 1

[hmgle]

@CaledoniaProject I have updated the getPidByAddr function to handle cases where the source IP address in the IP packet header has been modified. However, there is still an unusual case that cannot be handled: when the application binds a non-localhost address before connect call, and the source IP address in the IP packet is modified before it reaches the local server. It's acceptable for it to fail in such an unusual situation.

[osamahamad]

any workaround to solve this ?

graftcp-local -socks5 23.230.167.1XX:1080 -listen :7777

results

root@pentest:/var/www# graftcp curl ipinfo.io/ip
curl: (56) Recv failure: Connection reset by peer

[hmgle]

any workaround to solve this ?

graftcp-local -socks5 23.230.167.1XX:1080 -listen :7777

results

root@pentest:/var/www# graftcp curl ipinfo.io/ip
curl: (56) Recv failure: Connection reset by peer

@osamahamad Need more logs to pinpoint the problem. Could you please share the output of mgraftcp --enable-debug-log --socks5 23.230.167.1XX:1080 curl ipinfo.io/ip?

[osamahamad]

root@pentest:~# mgraftcp --enable-debug-log --socks5 23.230.167.1XX:1080 curl ipinfo.io/ip
[2024-06-18 12:16:26] [INFO] graftcp-local start listening :0...
[2024-06-18 12:16:26] [DEBUG] StorePidAddr(1698396, 34.117.186.192:80)
[2024-06-18 12:16:26] [INFO] Request PID: 1698396, Source Addr: 127.0.0.1:42490, Dest Addr: 34.117.186.192:80
23.230.167.1XXroot@pentest:~#

I'm trying to achieve something u might call a system-wide proxy using graftcp. via

graftcp-local -socks5 23.230.167.1XX:1080 -listen :7777

then redirect outgoing TCP/UDP traffic

iptables -t nat -A OUTPUT -p tcp -j REDIRECT --to-port 7777
iptables -t nat -A OUTPUT -p udp -j REDIRECT --to-port 7777

I assume this results in command being executed through the socks5 proxy for example curl or go scripts without the need of passing it via an argument, for example , using -x in curl. By that i mean

curl ipinfo.io/ip

Will results in socks5 IP being returned

[osamahamad]

nvm i just figured that out @hmgle , it seems that i changed the default listen port on graftcp-local but i didn't specify it on graftcp command via -n argument.

[hmgle]

nvm i just figured that out @hmgle , it seems that i changed the default listen port on graftcp-local but i didn't specify it on graftcp command via -n argument.

If graftcp-local listens on port 7777, graftcp should specify the graftcp-local-port via -p 7777 argument. Oh, I didn't see the content about iptables above. Please ignore my previous reply.