Roadmap to v1.0

[codesections]

After #12, the streaming server's Sever Sent Events functionality is feature-complete. This means that we're getting fairly close to full functionality. I still need to implement the following features (in rough order):

• [x] Add unit tests. Warp has strong support for unit testing, so this should not take that long.
• [x] Read development/production configuration variables in place of current hard-coded defaults. This also shouldn't take all that long.
• [x] Add WebSocket support. This is the point where I'm most uncertain. Warp also has good support for WebSockets, so I think/hope that adding WS support might be as easy as adding several filters and then hooking them in to the existing streams. On the other hand, it is possible that it will require setting up separate streams; I'll need to dig into the docs/examples a bit more to be sure. So this could be fairly fast or could be a bit more time consuming.

Once I've added these features, we should be feature-complete and ready for integration testing with Mastodon proper. I am very much hoping to get to this point at or before May 10 because I will be traveling May 10–22 and will have limited availability. My goal is to have the streaming server testable before I leave.

After the server is feature complete, we'll have the following tasks:

• [ ] Perform integration tests with Mastodon
• [ ] Benchmark the server's performance both in terms of RAM usage and ability to handle concurrent connections. @Gargron, I would be interested to hear how you benchmarked the existing streaming server. Most of the load-testing tools I'm familiar with test time to complete a connection and thus, since SSE connections don't complete, don't show very useful data for either the Node or Rust streaming server. I believe there's better support for load-testing WebSocket connections, though I haven't looked into that as much.
• [ ] Make any changes necessary to improve performance or address issues found during testing.
• [ ] Finalize documentation for public release

Given all of the above and my travel in mid-May, it seems like a release window of late May/early June is a realistic target.

[codesections]

Now that #15, #16, and #17 have been merged, I believe that the repo is at a point where other developers could productively test it in development environments. Specifically, it can be run in at localhost:4000 in place of the Node.js streaming server and allows clients connected to the local Mastodon instance to function normally.

I updated the README with instructions on how to install and test the streaming server; I also noted a partial list of features that are currently missing that will need to be implemented before release.

I will be traveling for the next two weeks and will have very limited availability during that time. When I get back, I will continue working on the features necessary to get the repo to release and incorporate any feedback that other's testing might have generated during that period.

[Gargron]

Benchmark the server's performance both in terms of RAM usage and ability to handle concurrent connections. @Gargron, I would be interested to hear how you benchmarked the existing streaming server.

To be exact I have not benchmarked it at all. I simply run it in production and look at memory usage and the number of open connections of the process. I don't know how to benchmark a streaming server either.

[codesections]

To be exact I have not benchmarked it at all. I simply run it in production and look at memory usage and the number of open connections of the process.

That makes sense. I'll look forward to seeing how it compares once we can run it in a production environment. With just a few clients connected, it uses ~5 MB of RAM (compared to 400+ MB for the development version of the Node.js streaming server). I'm sure the gap will narrow considerably under full load, but that seems like a promising start at least.

[codesections]

closed in favor of #65