Get JSON report via websocket API

[joffrey-bion]

Currently I can only know the status of a test result be connecting to /getCaseStatus?casetuple=$case, but I have no more information about what's wrong.

Is there any way to fetch the actual JSON report via the web socket client as well? (or even HTTP?)

[oberstet]

the JSON reports are just files generated, and so can be fetched via HTTP I guess ... long ago, I forgot;) what I'm sure: there is no builtin websocket protocol for retrieving the test results. the websocket connection is the one under test, and only used for that - not for controlling/results-access

[joffrey-bion]

@oberstet thanks for your response!

the JSON reports are just files generated, and so can be fetched via HTTP I guess

I'm not aware of an HTTP server running in the autobahn testsuite container. Is there one? Or are you talking about running a separate HTTP server next to the autobahn container?

My main problem is that the file system is not always accessible depending on the platform. I'm working on a Kotlin multiplatform client, and the tests could go and read the JSON reports as files on JVM, nodeJS and some native platforms, but in JS browser or iOS I don't have access to the file system. That's why it would have been way easier to get these JSONs as responses in some endpoint since we're already calling "control endpoints" like /updateReports.

what I'm sure: there is no builtin websocket protocol for retrieving the test results.

Then what is /getCaseStatus?casetuple=$case meant for?

the websocket connection is the one under test, and only used for that - not for controlling/results-access

AFAICT there are control/results endpoints: /getCaseInfo, /getCaseStatus, /updateReports. Is that an internal API or something meant for some other usage?

[oberstet]

the testsuite has a builtin webserver: just add the webport option

https://github.com/crossbario/autobahn-testsuite/blob/master/examples/fuzzingserver/fuzzingserver.json

and start the docker container exposing 8080 with -p 8080:8080

the result looks ugly (the index pages .. the report pages are rendered nicely), but works;)

My main problem is that the file system is not always accessible depending on the platform.

the report files are generated, stored and served from within the docker container, so that should be good (independent of your test env)?

---

Then what is /getCaseStatus?casetuple=$case meant for?

this is one of the HTTP endpoint that are indeed used to control the fuzzing server - but that is HTTP not WebSocket. the problem with putting the control onto the same WebSocket connection that is being fuzzed is obvious. So the only sane way would be to run a 2nd WebSocket connection only for the control. Possible, sure. But that is not implemented right now ..

[joffrey-bion]

the testsuite has a builtin webserver: just add the webport option

Awesome, I didn't know that, thanks!

the report files are generated, stored and served from within the docker container, so that should be good (independent of your test env)?

Yes definitely, if the container can serve those files via HTTP, this is just perfect. I was talking about the absence of such HTTP API, in which case I would have to read the files manually.

this is one of the HTTP endpoint that are indeed used to control the fuzzing server - but that is HTTP not WebSocket

Oh. It did look like a websocket API though here: https://github.com/crossbario/autobahn-testsuite/blob/master/autobahntestsuite/autobahntestsuite/fuzzing.py#L309

the problem with putting the control onto the same WebSocket connection that is being fuzzed is obvious. So the only sane way would be to run a 2nd WebSocket connection only for the control. Possible, sure. But that is not implemented right now ..

Totally agreed, and I actually used additional websocket connections for those endpoints (which seemed to work fine on most client implementations I'm testing right now). So now I'm confused as to this HTTP vs websocket controls, it really did look like it worked as a websocket connection 😆 In any case, I will try with the HTTP server, thanks!

[oberstet]

In any case, I will try with the HTTP server, thanks!

yes, from what I understand, that should do what you want (get at the test result json) and is simple to do.

So now I'm confused as to this HTTP vs websocket controls

I'm sorry, you're right;) long ago I looked into the code. anyways, it indeed works by using the HTTP path given by the fuzzed client when opening a WebSocket connection: if no "fuzzing case" is to be run, use that path to multiplex to the respective control endpoint.

https://github.com/crossbario/autobahn-testsuite/blob/f274c381034bba60c0f00f381b24e8031e00aaaf/autobahntestsuite/autobahntestsuite/fuzzing.py#L296

and after that, immediately close the connection. so either run a test case or run a cmd and end immediately is the logic once a new websocket connection is opened. so no fuzzing and control at the same time - even though only over 1 server, and websocket.

[joffrey-bion]

So I tried and I can confirm that the HTTP server can indeed serve the JSON reports. I just need to GET /cwd/reports/clients/index.json or other JSON files at the same path.

I also tested the endpoints mentioned above (/getCaseStatus, /updateReports etc.), and as expected they are websocket connection endpoints and not supported by the HTTP server. When trying to call them via HTTP, I get a 404 error page:

<html>
    <head><title>404 - No Such Resource</title></head>
    <body>
        <h1>No Such Resource</h1>
        <p>File not found.</p>
    </body>
</html>

Using the one-shot websocket connection (immediately closed) works most of the time, it's what I did so far to get the test status and to generate reports. However sometimes the connection is abruptly closed without returning a text frame with the test result (couldn't pinpoint the exact problem here, though, but it seems to mostly happen when testing several agents in parallel - hence my question in https://github.com/crossbario/autobahn-testsuite/issues/119).

Apart from that, the only drawback of /getCaseStatus is that it only returns the status of the test (OK, FAILED, etc.) but no extra information.

By contrast, with the HTTP server giving me the JSON files, I need to run updateReports after each test to then be able to fetch the JSON file for the test and read detailed results. Depending on the number of tests, it might be a problem to rewrite these files constantly. I also wonder about synchronization issues for parallel tests. Still better than nothing, though :)

[oberstet]

I also tested the endpoints mentioned above (/getCaseStatus, /updateReports etc.),

yes, those the URLs used when opening the websocket connection, and since every websocket connection starts as HTTP, this is where the confusion comes from. those URLs cannot be used to continue with HTTP/GET after the initial HTTP open.

By contrast, with the HTTP server giving me the JSON files, I need to run updateReports after each test to then be able to fetch the JSON file for the test and read detailed results.

yes, unfort. that's how it works (results are just collected in a server class and only updateReports will process that, and process all collected results).