search

crossbario / autobahn-testsuite

Autobahn WebSocket protocol testsuite
https://crossbar.io/autobahn/
Apache License 2.0
1k stars 84 forks source link

Allow close frames to pre-empt message echo responses #102

Closed wbudd closed 5 years ago

wbudd commented 5 years ago

There are 9 cases (3.2, 3.3, 3.4, 4.1.3, 4.1.4, 4.2.3, 4.2.4, 4.2.5, and 5.15) where the test suite sends a valid message immediately followed by some invalid frame. The problem—as I see it—is that it only gives the peer a Pass if the peer manages to echo the valid message back before that peer shuts down the WebSocket session. However, as far as I can tell, there is nothing in RFC 6455 to suggest that doing so is required; which means the reported Non-strict evaluation given instead when echos are absent is at least a misnomer ("non-strict with regard to what?").

This problem probably doesn't arise much in single-threaded implementations, but in RingSocket WebSocket message contents are relayed by a Websocket-handling worker thread to a separate backend app thread. That same worker thread is also responsible for processing any outbound messages originating from the app thread (e.g., echo responses) for the same WebSocket session, but during that time it will continue to read and parse any new data coming in over the wire—and immediately shutdown any session for which it detects a malformed WebSocket frame. This means a close frame may be sent out by that worker thread before a data frame response from the app thread has arrived (by which time it's then correctly dropped without making an attempt to send it out on the wire).

I believe RingSocket's behavior in that regard fully complies with RFC 6455, and should therefore pass the above-mentioned test cases—but I welcome any feedback here!

PS: Autobahn-testsuite saved me lots of time testing WebSocket protocol details. Thank you providing this valuable tool!

wbudd commented 5 years ago

...that said, I understand that from autobahn-testsuite's perspective expecting to receive an echo response before a close frame serves as a valuable heuristic, because otherwise it would have no way of telling whether the close frame it receives was triggered by the intended invalid data or some unrelated false positive.

So maybe just always pause for a short amount of time (0.1s should be plenty) before sending malformed frames, to accommodate for any valid control frame <--> data frame response reordering?

oberstet commented 5 years ago

please see the comment added in the results pages:

Test case was executed and passed non-strictly. A non-strict behavior is one that does not adhere to a SHOULD-behavior as described in the protocol specification or a well-defined, canonical behavior that appears to be desirable but left open in the protocol specification. An implementation with non-strict behavior is still conformant to the protocol specification.

not processing everything up to an invalid frame is considered non-strict behavior by the testsuite.

wbudd commented 5 years ago

Thanks for your quick follow-up.

A non-strict behavior is one that does not adhere to a SHOULD-behavior as described in the protocol specification or a well-defined, canonical behavior that appears to be desirable but left open in the protocol specification.

I don't see any "SHOULD" in RFC 6455 applicable to this behavior, so I guess you interpret this variety of fast failing as conflicting with some "well-defined, canonical behavior that appears to be desirable"? If so, this supposedly well-defined behavior ought to be defined/documented somewhere, right? Can you provide a reference?

An implementation with non-strict behavior is still conformant to the protocol specification.

That's nevertheless a good assurance to have.

not processing everything up to an invalid frame is considered non-strict behavior by the testsuite.

I don't think that's an accurate summary of the situation I described, because in fact everything is processed in the sense that the server backend application does receive everything up to any invalid frame.

It just so happens that any outbound message it decides to send back to the originating peer as a response may be processed later than the next inbound message (as well as the close response that may trigger). Obviously though the protocol specification has no such notion of "echoing speed", so I'm again left wondering how these circumstances can conflict with any "well-defined, canonical behavior".