[SignalR] Autobahn-like test suite for SignalR

[szarykott]

Is your feature request related to a problem? Please describe.

I started working on SignalR implementation for Rust, which I believe is missing. I want to have a way to verify if the implementation is correct and has all the required features.

Describe the solution you'd like

I am looking for an automated test suite for SignalR much like the Autobahn test suite for WebSockets. Is there such a thing? I filed it as a feature request, but I am not sure if this is correct. After all, if it is missing or not public, it is a kind of feature request.

If I started the issue in the wrong place, please redirect me to the correct one.

[adityamandaleeka]

Triage: this sounds like a good idea. We'd love to have something like this. We'd need to define what the cases are to test, and also have an example implementation.

Notes: we'll want to minimize the effort required on the client side of the tests, while leaving open the possibility of doing asserts/checks on both sides.

See also gRPC interop tests: https://github.com/grpc/grpc/blob/master/doc/interop-test-descriptions.md

[ghost]

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[SebastianKunz]

I think this could be quite interesting for people developing new clients. Here are my initial thoughts.

• Do we want to provide a test suite only for client side implementations, or also for server side?
• Where should the source code live? This repo, or somewere else?
• How does a client implementation interact with the Autobahn? E.g. like gRPC interop tests via command line?

Test Coverage

Regarding the test coverage I think we want to cover as much as possible of the spec. Here is a brief overview. These are directly extracted from the Hub Protocol Spec.

Protocol

Flows

All supported communication flows should be covered. This means handshake, invocations, streaming, non-blocking-invocations, ping, reconnect and close (peacefull and unexpected). Message headers are currently not used, but are part of the spec so they should be included.

Encoding

The spec defines two encodings (JSON and MessagePack). All flows should be tested with both encodings.

Transport

All flows should should support the different transport layers covered in Transport Protocols Spec. This includes WebSockets, Server-Sent Events and Long Polling. The negotiation phase should be covered as well.

[BrennanConroy]

All flows should be tested with both encodings.

All flows should support the different transport

Things like this should be optional, we don't want to write a test suite that requires people to write all encodings and all transports. Additionally, people should be able to add their own encodings or transports and test them.

There should probably be "core tests" that test the bare minimum and then a bunch of additional optional tests for covering everything else. Then client implementations can choose what they want to test against based on what they support.

Other test cases:

• authorization, via header or query string
• transport fallback
• unexpected hub message types (think new server old client, or not full featured client)
• invalid protocol
• multiple hub messages in a single receive

Do we want to provide a test suite only for client side implementations, or also for server side?

Client side is the more interesting one right now. People are more likely to write a new client than a new server.

Where should the source code live? This repo, or somewere else?

Ideally here, then we can use it for all our clients.

• How does a client implementation interact with the Autobahn? E.g. like gRPC interop tests via command line?

No clue 😄 would need to look at how other test suites like autobahn and grpc do things.

[SebastianKunz]

No clue 😄 would need to look at how other test suites like autobahn and grpc do things.

I had a quick look at grpc and autobahn-testsuite. The grpc-interop-tests describe a set of tests to be implemented for client and server implementations. From what I can see by skimming the docs they have connection-backoff-interop-test-description.md, http2-interop-test-descriptions.md and interop-test-descriptions.md test descriptions. Each specifiying tests for parts of the spec. They use program args to interface with the implementation. That way they pass down a test name which should be run by the client/server. Then they provide a script to run the server and client.

For convenience, we provide a shell script wrapper that invokes both server and client at the same time, with the same test_case. This is the preferred way to run these tests.

Autobahn|Testsuite on the other hand provides a docker image to test against client/server implementations. They also provide some nice tools for developing

Besides the automated testsuite (aka "fuzzing" server/client), wstest also includes a number of other handy modes

• WebSocket echo server and client
• WebSocket broadcast server (and client driver)
• Testee modes to test AutobahnPython against the test suite
• wsperf controller and master (see below for more)
• WAMP server and client, for developing WAMP implementations
• WebSocket Mass-Connect

Instead of using cmd line args, like grpc, a client has to open a new connection to the autobahn-test-server (fuzzingserver) specifiying what test to run via query string params.

[BrennanConroy]

a client has to open a new connection to the autobahn-test-server (fuzzingserver) specifiying what test to run via query string params

This sounds like what I was thinking at one point. We would provide a server app with all the test cases configured (hub methods, hub protocols both good and bad, logic to hijack the request and inject invalid data, etc.), and the client would just write the test cases it cares about and connect to the server either with a query string, or a path to a hub.

[szarykott]

Nice to see it took off.

Client side is the more interesting one right now. People are more likely to write a new client than a new server.

While I agree that people are more likely to create clients, me in particular (as OP of this issue) created server in Rust. I later learnt that to create server I also need to de facto implement client as I need to invoke client's hub methods from the server.

This sounds like what I was thinking at one point. We would provide a server app with all the test cases configured (hub methods, hub protocols both good and bad, logic to hijack the request and inject invalid data, etc.), and the client would just write the test cases it cares about and connect to the server either with a query string, or a path to a hub.

I like the idea. I think it would be equally easy to have a client with corresponding setup.

Things like this should be optional, we don't want to write a test suite that requires people to write all encodings and all transports. Additionally, people should be able to add their own encodings or transports and test them.

100% agree. For instance having support for only text WebSocket would be satisfactory for most users of custom implementation. Those implementations could just mention they support 75% of spec or simply name supported features.

What would be really interesting side effect of writing test cases would be to have all possible features listed in compact form, maybe I did bad job searching but the only place they are listed are those long documents (and source code ofc):

• HubProtcol
• TransportProtocol.

I had some fun establishing a list of features to implement digging through those. And a few surprises when I discovered new things to implement that I missed previously.

As a side not, it would be nice if tests were contenerized to have an easy way to run them in CI.

[SebastianKunz]

I had a look at the reports Autobahn|Testsuite generates. It's quite neat. For every test case they write out json files containing all the information gathered for the testcase. They also provide html files, organized as reports to provide a summary of the testcases. Every testcase has a lot of detailed information about the testcase like description, expectation, actual, wirelog, wire statistics and more. I think developing a client it could be quite nice to have similar features. When a tests fails a user can inspect where it went wrong by e.g. inspecting the network traffic.

What do you think of the idea of developing a POC SignalR-Autobahn first. The main goal is to establish a framework that allows the development of autobahn-signalr tests. It shouldn't be too much about the tests themselves, but rather about the environment. That way we can see what works and what doesn't and what features are needed. This also allows developers to implement their own autobahn-tests without understanding the whole autobahn pipeline. Just like you don't have to understand how xUnit or nUnit work. You just write your tests.

[szarykott]

I am totally for a tracer bullet approach here.

I would gladly see how tests that are developed fit into my implementations of server / client and how to make it more seamless before jumping all in.

I thumbed through Autobahn as well and while it seems easy to implement tests for servers (just use C# client with tests encoded with expectations, run via docker), developing tests for client seems more challenging.

A server with test cases could be provided, but then what? Each implementor would need to write their own tests by hand (possibly getting some of them wrong). Maybe there could be a way to standardize test cases notation somehow and make users only implement parser for those test cases (for code generation for instance)?

[SebastianKunz]

Each implementor would need to write their own tests by hand (possibly getting some of them wrong)

Exactly. But I fear thats just the costs one has to pay for a fully fledged automated test suite.

Maybe there could be a way to standardize test cases notation somehow and make users only implement parser for those test cases

I like this idea, but I am not sure whats the effort / reward ratio here. Might be too much work. But you hit a crucial point here: If we go for "every implementation manually implements the test cases" we need to make sure to provide examples and proper documentation on each test case so there is no confusion on what implementations have to do.

[szarykott]

we need to make sure to provide examples and proper documentation on each test case so there is no confusion on what implementations have to do

This alone would be a satisfactory outcome of this issue. Automation would be a nice addition, a cherry on the cake.