Use somltcp

[cavivie]

@ibigbug I think I might be of some help here, as I recently stabilized the netstack-smoltcp crate.

[ibigbug]

yeah that'd be great. I think @VendettaReborn has been playing with it a bit lately, https://github.com/Watfaq/netstack-smoltcp/blob/main/examples/proxy.rs

and it seems it's feasible to use tun crate for the device setup and use smoltcp's stack to deal with layer 4 logic. so we can replace lwip.

is that something similar to what in your mind?

[cavivie]

yeah that'd be great. I think @VendettaReborn has been playing with it a bit lately, https://github.com/Watfaq/netstack-smoltcp/blob/main/examples/proxy.rs

and it seems it's feasible to use tun crate for the device setup and use smoltcp's stack to deal with layer 4 logic. so we can replace lwip.

is that something similar to what in your mind?

Yes, you are right. I would like to clarify why this is layer 4 and not layer 3. Currently netstack based on smoltcp can work in most scenarios of layer 3.

[cavivie]

Well, I misunderstood you. It seems that you mean tcp/udp.

[ibigbug]

yeah that's waht i meant

[ibigbug]

@cavivie were you planning to creat a PR for this?

[cavivie]

I'd love to, but before that, can you add more about tun in ss, including unresolved problems and known problems and goals.

[ibigbug]

I think the main goal is to replace the lwip w smoltcp, for performance boost, see https://github.com/Watfaq/netstack-smoltcp/blob/main/benchmark.md

And adopting pure rust impls over foreign C bindings.

This is only for the tun input, not much todo with ss.

We already use smoltcp in the user space wirefuard, it consolidates the deps which is another good reason too.

And thanks for you work.

[cavivie]

Thank you for your description. I think I will take time to integrate it recently.

[VendettaReborn]

@cavivie @ibigbug i added some soem auto detect interface feature for rust-tun, and here is the example code: https://github.com/VendettaReborn/netstack-smoltcp/blob/f89b9e9b2e56a6abf1b2be9290e4cf63c7be3e11/examples/forward.rs#L142 . Basically, it's just a code version of route commands. And i've tested it on linux,macos & windows.

Shall we extract the logic of auto-detect interface, auto setup routes into a independent crate? just like sing-tun?

[ibigbug]

cc @Itsusinn

[Itsusinn]

Shall we extract the logic of auto-detect interface, auto setup routes into a independent crate? just like sing-tun?

i would like call it watfaq-tun : )

It will combine both https://crates.io/crates/net-route and https://crates.io/crates/tun

[VendettaReborn]

any updates? cc @cavivie

[Itsusinn]

any updates?

最近挺忙的，大概到六月初才有时间

[cavivie]

Ah, ah, first of all, I'm sorry for the reason why I've been busy recently. I want to disassemble the current steps here:

1. Shall we extract the logic of auto-detect interface, auto setup routes into an independent crate? just like sing-tun?

Are there any disassembled independent clates here? I want to do some work based on this.

2. BTW, netstack-smoltcp may have some things that need to be changed. Can I think of anything else you need to add besides some refactoring of the udp part?

[VendettaReborn]

hh, it's ok, and hope you can rejoin it whenever you're ready.

for the first point, for example, the route's auto-configuration can be done via programming, here is a WIP version: (https://github.com/VendettaReborn/netstack-smoltcp/blob/auto-detect-interface/src/utils.rs), it can works in netstack-smoltcp's example, but need some extra work before adding it into clash-rs.

What's your idea?

[ibigbug]

i would like call it watfaq-tun : )

Or wontun 😄

[VendettaReborn]

i've done some extra jobs on it:

1. forward_lwip && forward_smoltcp. I compared smoltcp with lwip on their basic proxy mechanism, and found that the netstack-smoltcp may not work well on ipv6 (lwip is good)
2. some work on adding ipv6 addr for tun device, which is missed in rust-tun
3. watfaq-tun. code equivalent for ip route & 'ip rule' commands. But with supports of ipv6. Now works on linux&macos.

and watfaq-tun may need some reviews and comments. We can make it more robust and user-friendly before putting into usage in clash-rs.

cc @ibigbug @cavivie @Itsusinn

[ibigbug]

Awesome work! Thanks! Will take a look

[ibigbug]

@VendettaReborn i think we can start integrating your tun with https://github.com/Watfaq/clash-rs/issues/396 and improve it progressively, thoughts?

[Itsusinn]

watfaq-tun. code equivalent for ip route & 'ip rule' commands. But with supports of ipv6. Now works on linux&macos.

I dont think we need net_route crate(at least on linux). It doesn't meet my requirement of impl auto-route(eg. no fwmark support).

[VendettaReborn]

of course, global routing on linux is easy, your impl of fwmark is surely sufficient, but what about macos and windows? I don't expect most of our users to use linux.

[Itsusinn]

of course, global routing on linux is easy, your impl of fwmark is surely sufficient, but what about macos and windows? I don't expect most of our users to use linux.

I could write my route impl on windows. But indeed i cant do the same for macos xD. So i would like take net-route as a platform-specific dep.

[cavivie]

forward_lwip && forward_smoltcp. I compared smoltcp with lwip on their basic proxy mechanism, and found that the netstack-smoltcp may not work well on ipv6 (lwip is good)

I think we may be further along, since ipv6 local routing is supported after this MR, I have successfully tested ipv6 ping in netstack-smoltcp using the latest main branch after this MR: https://github.com/smoltcp-rs/smoltcp/pull/900

[cavivie]

For now, this means we can further work on IPv6 code. Nonetheless, this also seems to mean that the support of the ipv6 part of the smoltcp crate is not perfect enough. We may need to test more to find the problem. I am not sure what specific aspects we need to test for IPv6. Can you give me some advice?

[VendettaReborn]

for the outbound handlers, we have the docker test suites, which will cover tcp&udp. For tun, we haven't figure out some good ways to test it on both ipv4 & ipv6. here is a brief thought: https://github.com/Watfaq/clash-rs/pull/362#issuecomment-2114260969

@cavivie thoughts?

[cavivie]

Here is a pic that some of simple tcp tests shared. rust-tun does not currently support ipv6, I want to do something about rust-tun's shortcomings in ipv6 in the near future to avoid us do some hacky things about ipv6 for tun device at outside of rust-tun crate. What do you think?

[cavivie]

for the outbound handlers, we have the docker test suites, which will cover tcp&udp. For tun, we haven't figure out some good ways to test it on both ipv4 & ipv6. here is a brief thought: #362 (comment)

@cavivie thoughts?

Absolutely agree, usually routing is definitely an important point that hinders testing.

[VendettaReborn]

yeah, i agree. rust-tun hasn't supported ipv6 yet, we can extend the ability and some other feature(like routing rules setup) in a separate crate.

Here is a pic that some of simple tcp tests shared. rust-tun does not currently support ipv6, I want to do something about rust-tun's shortcomings in ipv6 in the near future to avoid us do some hacky things about ipv6 for tun device at outside of rust-tun crate. What do you think?

[cavivie]

a separate crate

What I mean is that we may need to use the rust-tun patch first, and then merge some patches about ipv6 into rust-tun upstream if possible.

[VendettaReborn]

oh, i see, that's better.

[cavivie]

To add tun feature, we want rust-tun to behave like:

• Dual stack is enabled by default, that is, ipv4/ipv6 addresses are bound to the tun device at the same time, and then a method is provided to disable ipv6 or ipv4;
• IPv4 is enabled by default, IPv6 can be optionally added
• IPv6 is enabled by default, IPv4 can be optionally added

Obviously, I think the first two are more reliable, and in particular, I think the second point is more needed. What do you think?

[VendettaReborn]

that would be great

[cavivie]

After some time testing tun2 and various netstacks, I found that in terms of throughput:

• Throughput (nix write): lwip > smoltcp > ipstack
• Throughput (win write): lwip > smoltcp > ipstack
• Throughput (nix read): smoltcp = lwip > ipstack
• Throughput (win read): smoltcp > lwip > ipstack

In terms of stability, ipstack currently performs the worst, and even the throughput is unacceptable on win, about 0.2 to 0.3 Gbps., while the VPN scenario focuses more on netstack's read performance, followed by write performance, and finally stability.

The above data is only a personal test on a specific device. Later, I will publish a test routine for you to test and compare.

[cavivie]

rust run pef test: https://github.com/automesh-network/rust-tun-perf.git

[ibigbug]

Thanks for sharing.

To me I think it's still worth adopting smoltcp even with similar bar of performance to lwip for simplicity and pure rust impl.