Open ntzb opened 1 year ago
I've added a new branch, windows-cfg_if
, which replaces my incorrect assumption that Rust would discard unused symbols during its build process, rather than trying to evaluate everything.
I also incorrectly assumed that nix and other things flagged in your build would still provide those symbols, even if their implementations were effectively just generating errors.
It seems to build in a limited cross-compilation environment here, but I haven't used Windows in over two decades, so I'm afraid I don't have any means of testing it myself.
thanks for looking into this! I'll happily test. it compiles fine now. tried a simple usage, but getting an error. server:
C:\Apps>rperf -s
[2023-09-05T16:50:41Z INFO rperf::server] server listening on 0.0.0.0:5199
[2023-09-05T16:50:54Z INFO rperf::server] connection from 192.168.1.150:57412
[2023-09-05T16:50:54Z ERROR rperf::server] error in client-handler: connection lost
[2023-09-05T16:50:54Z INFO rperf::server] 192.168.1.150:57412 disconnected
client:
C:\Apps\rperf>rperf -c htpc.lan
[2023-09-05T16:50:53Z INFO rperf::client] connecting to server at htpc.lan:5199...
[2023-09-05T16:50:53Z INFO rperf::client] connected to server
[2023-09-05T16:50:53Z ERROR rperf] unable to run client: operation would block
p.s. I tried cross compiling for aarch64 (target aarch64-unknown-linux-gnu
, but seems to hit brick walls, on macos m1. if it's possible, can you try and build for that target as well?)
Can you re-run this with --debug
(or -d
)?
I suspect the block is happening at https://github.com/opensource-3d-p/rperf/blob/14d382683715594b7dce5ca0b3af67181098698f/src/protocol/communication.rs#L45 , though I'm unsure of the circumstances under which the Windows networking subsystem would generate an EWOULDBLOCK response to an established, not-yet-used TCP connection, especially since write_all()
is supposed to take care of overflows. So it might be happening somewhere else.
server:
C:\Apps>rperf -s -d
[2023-09-05T17:22:28Z DEBUG rperf] registering SIGINT handler...
[2023-09-05T17:22:28Z DEBUG rperf] beginning normal operation...
[2023-09-05T17:22:28Z DEBUG rperf::stream::tcp::receiver] using OS assignment for IPv4 TCP ports
[2023-09-05T17:22:28Z DEBUG rperf::stream::tcp::receiver] using OS assignment for IPv6 TCP ports
[2023-09-05T17:22:28Z DEBUG rperf::stream::udp::receiver] using OS assignment for IPv4 UDP ports
[2023-09-05T17:22:28Z DEBUG rperf::stream::udp::receiver] using OS assignment for IPv6 UDP ports
[2023-09-05T17:22:28Z DEBUG rperf::utils::cpu_affinity] enumerated CPU cores: [0, 1, 2, 3, 4, 5]
[2023-09-05T17:22:28Z DEBUG rperf::utils::cpu_affinity] not applying CPU core affinity
[2023-09-05T17:22:28Z INFO rperf::server] server listening on 0.0.0.0:5199
[2023-09-05T17:22:36Z INFO rperf::server] connection from 192.168.1.150:57630
[2023-09-05T17:22:36Z DEBUG rperf::protocol::communication] awaiting length-value from 192.168.1.150:57630...
[2023-09-05T17:22:36Z ERROR rperf::server] error in client-handler: connection lost
[2023-09-05T17:22:36Z INFO rperf::server] 192.168.1.150:57630 disconnected
client:
C:\Apps\rperf>rperf -c htpc.lan -d
[2023-09-05T17:22:35Z DEBUG rperf] registering SIGINT handler...
[2023-09-05T17:22:35Z DEBUG rperf] connecting to server...
[2023-09-05T17:22:35Z DEBUG rperf::stream::tcp::receiver] using OS assignment for IPv4 TCP ports
[2023-09-05T17:22:35Z DEBUG rperf::stream::tcp::receiver] using OS assignment for IPv6 TCP ports
[2023-09-05T17:22:35Z DEBUG rperf::stream::udp::receiver] using OS assignment for IPv4 UDP ports
[2023-09-05T17:22:35Z DEBUG rperf::stream::udp::receiver] using OS assignment for IPv6 UDP ports
[2023-09-05T17:22:35Z DEBUG rperf::utils::cpu_affinity] enumerated CPU cores: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
[2023-09-05T17:22:35Z DEBUG rperf::utils::cpu_affinity] not applying CPU core affinity
[2023-09-05T17:22:35Z DEBUG rperf::protocol::messaging] preparing TCP upload config...
[2023-09-05T17:22:35Z DEBUG rperf::protocol::messaging] preparing TCP download config...
[2023-09-05T17:22:35Z INFO rperf::client] connecting to server at htpc.lan:5199...
[2023-09-05T17:22:35Z INFO rperf::client] connected to server
[2023-09-05T17:22:35Z DEBUG rperf::client] running in forward-mode: server will be receiving data
[2023-09-05T17:22:35Z DEBUG rperf::protocol::communication] sending message of length 172, Object {"family": String("tcp"), "kind": String("configuration"), "length": Number(32768), "receive_buffer": Number(0), "role": String("download"), "streams": Number(1), "test_id": Array [Number(107), Number(61), Number(17), Number(48), Number(58), Number(130), Number(65), Number(229), Number(177), Number(53), Number(253), Number(22), Number(160), Number(115), Number(249), Number(105)]}, to 192.168.1.181:5199...
[2023-09-05T17:22:35Z ERROR rperf] unable to run client: operation would block
I'll hopefully be able to do some testing directly against M1/M2 hardware later this year, running native Mac OS, but for now, I'm just trusting that nix provides the facilities I read in its documentation.
aarch64 for Linux should work, though: we do a lot of testing with SBCs, even down to some 8-bit controllers. I'll need to know more about what you mean by "brick walls".
Yeah... sending 176 bytes really shouldn't be a problem with the socket in the state I expect it's in.
But it looks like mio's implementation changed since 0.6 (in a manner that may have been later retconned by changing the function's name) and now it expects the stream to already be in non-blocking mode, rather than performing that conversion itself. It expects everything to be non-blocking, actually.
Creating another branch off of the Windows one to build this, though I'm afraid I'm working on a number of other things today, so I can't promise super-quick responses.
Ah, it's not that mio's interface changed within 0.6, it's that it behaves very differently on Windows.
And mio 0.7+ removed set_keepalive(), which was the only reason I had introduced it in the first place.
Learning what I can.
I'll hopefully be able to do some testing directly against M1/M2 hardware later this year, running native Mac OS, but for now, I'm just trusting that nix provides the facilities I read in its documentation.
aarch64 for Linux should work, though: we do a lot of testing with SBCs, even down to some 8-bit controllers. I'll need to know more about what you mean by "brick walls".
I don't think it's rperf related, just probably cross compilation rust stuff (missing linker libs, things like that) - I've already burnt a few hours on it today, it's not exactly a well documented procedure (macos m1 targeting aarch64-unknown-linux-gnu
, tried 3 different linkers, ld
cannot find needed libs).
it's just if you have cross compilation working, I'd be happy to see a aarch64 test binary as a shortcut :)
Ah, it's not that mio's interface changed within 0.6, it's that it behaves very differently on Windows.
Learning what I can.
it's fine, surely nothing urgent about this Windows build, any effort is appreciated
I've excised mio in the socket2
branch. While everything works (at least under Linux), there's a timing mismatch in the handoff for TCP that doesn't affect total transmission time or data sent, but does skew the data-received window.
I'm trying to figure out how to minimise its impact, but for now, you should be able to build that for Windows to see if it works.
I ended up putting bits of mio back in place, but it should all be working now. Possibly a little more accurate than before, though on the order of microseconds, nothing too significant.
tested with socket2
clean build, same exact logs unfortunately.
testing between two windows machines.
the server is only using port 5199 right?
This morning, I observed a Windows build linked through cross-compilation from Linux (same process you were using) succeed as a traffic-sending client without issue, so the "operation would block" issue may be something configuration or policy-based (a firewall, perhaps). It's possible the previous implementation would have worked, too.
However, there is a problem when receiving traffic on Windows: only the first interval (one second) of traffic gets through with TCP and nothing with UDP, seemingly because the sender assesses a timeout, concludes the receiver isn't present, and gives up.
I'll look into this when I have time, but it's a lot less obvious. I think I already have access to all the data I need, though (trace-level logs, packet-captures).
As for aarch64/Linux, I'm still able to build without issue (Debian host), but because there's a dependence on dynamic linking (libc specifically), even if I felt comfortable providing precompiled binaries (and you'd feel comfortable accepting them), there's a high chance of a mismatch, so it's probably best to leave resolving the conflict to your own build environment (maybe use a VM on your M1 if you want to build with a native instruction-set), to avoid desync.
interesting, I just (cross) compiled for macos, linux and it works between then on my x86 openwrt machine and m1 mac. Windows machine still fails as both client and server (tried multiple branches).
regarding the aarch64, I get it, it's ok. I'll try cross
Windows should now be able to receive UDP; still working on TCP.
I may have also implemented a working handler for the would-block case you're experiencing, though nobody I've asked to help me test has reported it themselves.
TCP reception should now work, too.
Aside from being able to configure TCP keepalive and rend/receive-buffers, there should be functional parity between POSIX and Windows.
If this checks out, I'll merge things and increase the version-number.
since the windows build is running as client, I'm not really sure what could be blocking.
something in my build process might be bad then.
file size is always the same, and the error remains (operation would block
).
if you test it on a different machine, and multiple other people confirmed it's working, I might have raised a false flag here, not sure if this issue should still be even open
Well, it was in a state where it wouldn't compile for Windows and I did need to rework time-accounting to make it deal with Microsoft's system clock semantics, so the issue definitely has merit.
I'll hopefully be able to see it running on more Windows systems on Monday (I'm in Western Canada, so it's about 3am on Saturday now), and maybe one of them will stall so I can get a live testing candidate.
For whatever it's worth, the Windows environment where it was observed to be working is Windows 10, x64, with very little in the way of additional security.
I don't suppose Windows has any facilities like strace
, does it? I'd be curious to see just what is going into the WinSock call and what it's returning, since I should be handling both the POSIX and WSAPI "not yet able to send" cases now (https://github.com/opensource-3d-p/rperf/blob/cf01ff4e783cb4686ba543b203b879beb5834a69/src/protocol/communication.rs#L65), which would seem to indicate that there's another class of error that is being overlooked here, but that has the same textual presentation as WouldBlock
.
I don't know anything about strace
unfortunately. perhaps one of the suggested tools here https://stackoverflow.com/questions/3847745/systrace-for-windows could be helpful.
I have 2 Windows PCs, 10 x64 and 11 x64, and both do not work (with Windows firewall disabled).
p.s. can you provide the Windows test binary that worked for you, or for anyone you gave it to? or I can provide the .exe that I built, and maybe you can test that one, just to eliminate cross compilation shenanigans
It's temporarily available in [pre-stable-binary link redacted].
md5sum (XZ compressed): b2795f70cb2c9ecc8a6f202ace37121f
your binary works! client AND server (linux <> windows, simple testing). now I really wonder what goes wrong in my cross compilation efforts.
you can take it down as far as I'm concerned..
My build environment is Debian Trixie (Testing)
All I'm doing is cross build --target x86_64-pc-windows-gnu --release
.
Prior steps included apt-get install mingw-w64-x86-64-dev
and cargo install cross
, though it seems like something else isn't present in my bash-history file.
I recall that cross
built a container around the mingw libraries and that cross
(cargo
) is using it to do the actual compilation, but I haven't touched anything inside.
I'm on macOS 13, something in the Apple+rust, aarch64 -> x86, is probably not mature enough.
the regular build (installing linkers and setting up targets) produces an unusable binary (as we've seen), and using cross
fails for every target I'm trying with sh: 1: cargo: not found
which probably means I have to build my own images etc.
I'm making arrangements to buy a Mac to figure this out (I have other things to port to ARM and could use the rough edges of Asahi to learn anyway), though it may take a couple of weeks before I have one available.
I also considered Ashai linux on my m1, but it's got a way to go.. no external monitors is a huge problem for me. if you haven't already, I suggest you take a look at https://github.com/AsahiLinux/docs/wiki/Feature-Support and https://www.reddit.com/r/AsahiLinux/comments/11t2yt9/asahi_linux_users_what_has_been_your_reallife/ btw it's my work laptop, I've already asked for a replacement, and will probably dump it in a few months anyway (hopefully you don't plan on buying a mac because of my cross compilation issues.. honestly, don't worry about it)
I've only been looking at Asahi for the past week (I have an entire Debian-based operating system to explore porting to ARM in general, not specifically to Apple Silicon, but just at a scale where I need to be able to rebuild binaries and things at decent speeds, without cross-compilation surprises), but it looks sufficient for my needs.
I'm going to wait to see if M3 minis are part of the Apple announcement on Tuesday, then make a decision from there, but it's going to happen in some capacity, both for work (like the OS) and projects I maintain outside of work, like this.
Kinda sucks that ARM is still so bound to U-BOOT and board-specific bring-up processes that Apple is the best option for a general-purpose dev-and-build system right now. I'd really love to just be able to put some reasonably specced mITX-like build together.
Although, on that note, how much memory do you usually see your system watermark at when doing any sort of compilation task?
my code is backend JS, so no much compilation going on here :) I agree that M1/2/3 are the best ARM the world has to offer right now. if that's what you need, go for it..
Tried in windows
inside msys2
in ucrt64
and get the following error:
User@DESKTOP-HPEE9O3 UCRT64 ~/rperf
$ ./target/release/rperf.exe -s
[2024-01-19T00:33:32Z INFO rperf::server] server listening on 0.0.0.0:5199
[2024-01-19T00:34:26Z INFO rperf::server] connection from 127.0.0.1:52004
[2024-01-19T00:34:26Z ERROR rperf::server] error in client-handler: connection lost
[2024-01-19T00:34:26Z INFO rperf::server] 127.0.0.1:52004 disconnected
User@DESKTOP-HPEE9O3 UCRT64 ~/rperf
$ ./target/release/rperf.exe -c 127.0.0.1 -p 5199
[2024-01-19T00:34:26Z INFO rperf::client] connecting to server at 127.0.0.1:5199...
[2024-01-19T00:34:26Z INFO rperf::client] connected to server
[2024-01-19T00:34:26Z ERROR rperf] unable to run client: operation would block
I'm expecting to merge a PR in the next day or two that has been tested on Windows, though it's largely the same solution as in the socket2
branch.
Can I ask which branch you used that led to that problem?
@flan cfg_if branch main branch didn't compiled
socket2
was where I took the code after cfg_if
upon encountering that timeout
hi and thanks for rperf. it is mentioned in the readme that
rperf should build and work on all major platforms, though its development and usage focus is on Linux-based systems, so that is where it will be most feature-complete.
trying to switch from iperf to rperf, I need some binaries for Windows and Linux aarch64.compiling for Windows fails with:
it seems that there are some dependencies on *nix tcp/udp