Feature request: overlay to split stream on upon custom tokens

[spoeschel]

At the moment I have the need to forward XML documents from a TCP stream to a WebSocket port (unidirectionally). Each document shall result in a separate WebSocket packet. Unfortunately this currently doesn't work with websocat, as it basically converts each TCP packet into a WebSocket packet. But as TCP is stream-based and WebSocket in contrast packet-based, for example these two cases can (and do) happen here:

• a TCP packet only consists of a part of an XML document (happens when documents are larger than the IP packet size, which is the case here)
• a TCP packet consists of more than one XML document

This requires to somehow buffer/split the received stream, in order to forward distinct XML documents.

It seems to me that a generalized version of the line2packet overlay would be great for that. In the case of such an overlay, one would e.g. need to specifiy a token (instead of having CR/LF hardcoded) that would correspond to the end of the document. So the stream is splitted always when that token is found in the stream. In the XML case this would be the closing root element, e.g. </test>. Note that in contrast to the line2packet overlay, the token would not be discarded, but forwarded as well (an additional option could be added to toggle that).

However as in XML namespaces are important, one token to match would not be sufficient here. In theory it would have to be considered, which prefix the namespace of the root element uses - but of course this would be way to complex/out of scope here. However in the present case it would be sufficient to simply assume that either no namespace prefix (e.g. </test>) or a specific namespace prefix (e.g. </abc:test/>) is used. Hence it would be necessary to be able to match against more than one token here.

To initially solve this, I extended copy<R, W> myself, but this is only a custom quick and dirty hardcoded solution (as I'm not yet that familiar with Rust). So it would be great if there would be a generalized way to do such splitting in websocat itself.

[vi]

converts each TCP packet into a WebSocket packet

In --binary mode. In --text mode it accumulates lines and converts each line to a WebSocket message (and vice versa: messages containing newlines are mangled to get rid of it). So one TCP packet can cause multiple outgoing WebSocket message or none, depending on number of 0x0A bytes in it.

If you are constructing WebSocket's advanced mode command line yourself, look for msg2line: and line2msg: overlays. You can override \x0A to be \x00 instead.

[vi]

I suggest implementing splitting XML documents separately from Websocat. They could be re-joined with zero byte as a separator, which Websocat can use for delimiter between Websocket messages.

Note that each document must fit in Websocat's -B buffer size, which is 64kb by default.

[vi]

So it would be great if there would be a generalized way to do such splitting in websocat itself.

How command line interface for such mode would look like? How would that be useful for other projects and use cases?

Is it really important that XML stream flows directly into Websocat instead of being processed by some intermediate process?

[vi]

I'm considering adding a plugin API (using C native loadable dynamic libraries or WebAssembly modules) for implementing custom specifiers and overlays.

There is even abandoned branch "plugins" branch for that. Would it be useful for your use case?

[vi]

If it is Linux and that TCP connection is actually localhost, then you can also use seqpacket: / seqpacket-listen: address types to have a interprocess connection which preserves messages boundaries. This also should result in one message becoming one WebSocket message, both in text and binary modes.

You can use both file-based UNIX sockets and abstract addresses.

[spoeschel]

Thanks for your detailed responses and hints.

Note that each document must fit in Websocat's -B buffer size, which is 64kb by default.

This is fine, as the resulting XML documents have a size of only about some KBs.

How command line interface for such mode would look like? How would that be useful for other projects and use cases?

There would be the need for a separate parameter to add such a splitting token. Maybe two params, to add text tokens (e.g. assuing UTF-8, where applicable) and binary tokens (I only need the former ones). The params can be specified multiple times and are collected in a list. So e.g. --add-split-token-text "</test>" --add-split-token-text "</abc:test>" here. And maybe an optional parameter to configure whether a found token is discarded (e.g. when splitting on line breaks) or kept.

I could imagine that other people may have the issue, too, when they want to convert messages from a stream-based protocol to a message-based one. The line2packet overlay is actually a special case of this case.

I suggest implementing splitting XML documents separately from Websocat. They could be re-joined with zero byte as a separator, which Websocat can use for delimiter between Websocket messages.

Is it really important that XML stream flows directly into Websocat instead of being processed by some intermediate process?

I wanted to prevent the need to add an additional tool to the workflow, as a separate tool would require to consider the possible errors a TCP connection could be affected of and to re-implement their handling. This would introduce additional locations where bugs could be present and lead to unnecessary redundany in the workflow. So I just wanted to split the actual bitstream with the help of websocat itself.

I'm considering adding a plugin API (using C native loadable dynamic libraries or WebAssembly modules) for implementing custom specifiers and overlays.

There is even abandoned branch "plugins" branch for that. Would it be useful for your use case?

This would be useful; I would allow me to write a plugin that does the described splitting.

If it is Linux and that TCP connection is actually localhost, then you can also use seqpacket: / seqpacket-listen: address types to have a interprocess connection which preserves messages boundaries. This also should result in one message becoming one WebSocket message, both in text and binary modes.

You can use both file-based UNIX sockets and abstract addresses.

While websocat runs on Linux here, the TCP source is a (proprietary) application on a different (Windows) PC. So unfortunately I have to stick with TCP here.

[vi]

consider the possible errors a TCP

Note that Websocat's native handling of TCP hangups and resets may be also subpar.

[vi]

This would introduce additional locations where bugs could be present and lead to unnecessary redundany in the workflow.

Note that Websocat as of itself may be not the best tool for production use where reliability and error handling matters.

Primary use case is using it as a good CLI for WebSockets, glue code, mini-projects, exploratory programming.

If need performance, reliability, proper handling of all edge cases, you may need a customized, integrated program.

[spoeschel]

So far the workflow works fine using websocat and handles all reconnects well. The file rate is at most about one file per second, so there is no performance issue so far.

[spoeschel]

Meanwhile the mentioned Windows application added native WebSocket support, so the mention issue is no longer relevant to me.