Output of received messages to stdout truncated to 1024 bytes

[rollulus]

Hi, I have a websockets endpoint that outputs messages of around 1500 bytes. When I run websocat ws://my-endpoint the output to my terminal is truncated to 1024 bytes. Then the next message is received, the remaining ~300 bytes of the first message, and the full next message are displayed. Then for the third message, the process repeats, with a truncation to 1024 bytes at first.

I verified that the websockets data itself is correct via inspection through tcpdump.

When I pipe the output to a file, the entire message is written to it. When I pipe the output through less, same. When I pipe the output through cat I get the message: cat: write error: Resource temporarily unavailable

What's an explanation for this behaviour? Could it be related to nonblocking output, e.g. UnixFile::raw_new(std::io::stdout()).set_nonblocking(false);?

[vi]

Are there no warnings issues to console? The behaviour looks like what message-to-line convertor in WebSocat does (but default buffer size is 65536, not 1024).

cat: write error: Resource temporarily unavailable

Maybe current terminal became non-blocking? Does running just cat (without any arguments, pipes or redirections) works or produces similar error?

This snippet would help removing non-blocking mode from current terminal:

perl -we 'use Fcntl qw(F_GETFL F_SETFL O_NONBLOCK); open F, "<&=", 0; my $flags = fcntl(F, F_GETFL, 0); fcntl(F, F_SETFL, $flags & !O_NONBLOCK);'

(or just reopening it)

---

Websocat typically switches stdin/stdout to nonblocking mode while it is operating, but then switches it back on exit (including abrupt termination, but not including panics/SIGKILL).

[rollulus]

Thanks for your response. There are no no warnings issued to the console. My cat works without errors. Some more experiments:

• websocat ws://url >file writes the full (>1024 bytes) message to the file;
• websocat ws://url | grep -e "" updates in 1024 byte chunks;
• websocat ws://url | less works correctly;
• websocat ws://url | cat shows the first 1024 bytes followed by cat: write error: Resource temporarily unavailable. When the next message arrives, websocat terminates: websocat: Broken pipe (os error 32) websocat: error running

[rollulus]

FYI I am on macOS. The behaviour is the same in iTerm and Terminal, and I tried zsh and bash.

[vi]

websocat ws://url | cat cat: write error: Resource temporarily unavailable

Are those command lines verbatim (modulo ws://url)? Is there anything after "cat"?

What happens if you try websocat ws://url | cat > file? What happens if you try cat file | cat (loading a pre-saved message)

Does anything change if you use -b option of websocat. Does anything change if you use this form: websocat -t ws://url threadedstdio: | cat?

[vi]

websocat ws://url | cat cat: write error: Resource temporarily unavailable

Now I understand:

1. websocat and cat both start together. websocat uses terminal's stdin, cat uses terminal's stdout.
2. websocat switches stdin to nonblocking mode. On Mac, this also switches cat's stdout to nonblocking mode (as it is the same device); probably unlike on Linux.
3. websocat writes data to cat, which in turn tries to write it to the terminal.
4. If data is small enough, write will immediately succeed even though terminal is nonblocking mode. But with more data it takes some time to accept data for display and cat gets unexpected EAGAIN.

threadedstdio: may be a workaround here.

[rollulus]

I'm glad you found it. Is there anything I can help with? My experience with Rust is about zero but I'm up for trying things if that helps you.

[vi]

Is there anything I can help with?

Is there something to be done? This issue is probably a design limitation and can't be just fixed.

Does the threadedstdio: workaround work for you?

[rollulus]

Sorry, I don't have the skills yet to give this a try. Thanks for your help, and your useful piece of software!

[michaelwoods]

I use websocat through a series of test scripts on macOS and appending threadedstdio: as the output worked for me.

[vi]

What console IO method should be default on Mac? Async or threaded? Or shall there be some tricky auto-detection?

[yeryomenkom]

The same problem for me. Text frames are truncated until the next frame being received. mac os

[vi]

@yeryomenkom Does the threadedstdio: workaround work for you as well?

[joshtriplett]

I've hit this issue as well, on Linux.

I ran into it when piping the output of websocat ws://... to hd to get a hexdump.

It's not OK to make the terminal non-blocking for other applications in a pipeline. I realize that it's unusual that changing stdin also changes stdout, but nonetheless, this is a common issue, and makes shell pipelines unreliable. And a tool like websocat is very likely to get used in a shell pipeline.

(For that matter, even if stdin and stdout were independent, changing stdout would break things like websocat ws://... & othertool where both tools share stdout.)

Conversely, relying on the blocking or non-blocking status of stdin/stdout means that websocat itself would break if something else in the pipeline changed stdin or stdout to blocking mode.

Please consider making the threaded implementation the default everywhere (including for the one-argument form of websocat, and for stdio:).

[vi]

OK, maybe it can be --fast-stdio option, with the default being threaded everywhere.

Shall I publish websocat 1.7.0 with just this change alone?

[vi]

Published v1.7.0 where the workaround is now the default behaviour.

[rollulus]

Thanks!!