Road to v3.0

[jondot]

Hi guys,

Sorry for the time off, we had our first child born and I have made the humane error of underestimating the time it would take from my open-source work. The good news is that I've finally stabilized and my free time has returned to normal, and I'd like to thank you all for putting up with me maybe disappearing for a bit, but I'm really excited and have a ton of new ideas and would like to be as helpful as I can to merge in all of your PRs! (existing and upcoming).

With that in mind, my notebook is also filled with ideas and improvements that Sneakers need, to be able to improve as an open source project that I never got to share, and I'd like to capture that here.

This list includes (but not limited to):

Simplification of Testing

By either converting many of the mocking-heavy code to integration tests, now that we have the proper framework for that. The reasoning is that for a production, mission critical system like Sneakers, a real test that goes through RabbitMQ is better than one that's mocked out, and "paying" the run time for it is also OK because Sneakers is fast and RabbitMQ is fast.

The result would be clearer tests, pragmatic tests, that generate more confidence. In addition to that, I'm also planning of making these declarative. Imagine a yaml file containing a scenario, like messages fixture, that is being loaded onto a queue automatically before each test (much like old Rails fixtures).

I know that arguable this is back to basics in a sense, but "back to basics" is something that generates simplicity and helps newcomers make more with knowing less.

Code Reduction

A line not written is a line not maintained. It is in my plans for a long while to try the challenge of reducing code in Sneakers. Doesn't matter which area. Grooming the codebase is very beneficial for mission critical systems.

Documentation Redux

I have the feeling, for a long while, that while our Wiki is dynamic, it might be less accessible. Before I was busy with the maternity leave, I've already started building a static-site-generator based documentation site, that would be up to date on sneakers.io, and be searchable with Algolia.

Features

The great news is, that Sneakers is already production proven, very stable, and runs in many production environment, and have processed billions of messages already. So, the core of Sneakers works well. It would now be our jobs to incorporate features that serve as power-multipliers for any project, such as Docker support, first-grade Rails support (sneakers-rails anyone?) and so forth. Any idea is welcome here, of course :+1: .

Why?

If you're happy with Sneakers, I guess you can just use and lock to the version that works for you. That was always part of the vision. Messaging infrastructure should not be disruptive, and it should stand our of your way; which is why part of our job is to fend out bloat and unneeded features. However, with that in mind, simplification, clarity, fun factor, performance, and lowering API friction, are always "features" in great need. I hope this makes sense!

Execution

I will be giving a constant amount of time, each day, to build every concern listed. Anyone is welcome to join in on the fun, discuss, and contribute.

Finally I'd like to extend my thanks to everyone that have contributed so far. I consider myself a Karma guy and I am truly happy to see the discussion and benefit Sneakers made for each one of you that happened to be so kind and contribute back, and the thousands of other users out that that just use it. So, thanks!

[jondot]

Docker support is in with this commit

[gottfrois]

Hey @jondot thanks for the amazing work you are putting into this project. I came across it recently and one pain point for me was to try to migrate from sidekiq to sneakers. Even though I am using ActiveJob, I was not able to find proper resource on how to properly integrate sneakers with ActiveJob.

Did I miss something obvious here? Otherwise I think the current documentation is missing this part ;)

Thanks, can't wait to see new features coming.

[graimon]

Hi @jondot,

I would like to propose some kind of middleware feature, where one can easily add common behaviours to all or some Workers, like logging elapsed time, parsing the message, or reading headers.

We have been able to add this by appending or prepending concerns to our Workers, but I believe there could be a built in method for simplifying it.

Here is an example of a very simple gem where we've had to use it: https://github.com/qustodian/tenantify-sneakers

Greetings and thank you very much for your work!

[gabrieljoelc]

first-grade Rails support (sneakers-rails...

@jondot what kind of things would go in sneakers-rails? Initializer and worker generators? I think the other thing to think about here is how we want publishing should work. Currently, we publish messages all over in our Rails applications using the la_gear bus. Do we want a more natural way of supporting that?

[gabrieljoelc]

@jondot @michaelklishin should we make a GH milestone for v3.0 and start adding the features that would be breaking?

@jondot do you want to create a sneakers-rails repo so we can start adding "feature request" issues and a v1.0 milestone or should we create a sneakers org and start putting repos under that? i can just create the repo under my account for now as well if that seems appropriate.

[jondot]

In my opinion milestone is a good idea, it will make goals very visible. As I'm on mobile now, I'll need to check out who might be affected by moving to a sneakers org, that would take a few days to check with major sneakers deployments - but it makes sense.

For now feel free to open up such a repo, and we can see how to consolidate as we go :)

[michaelklishin]

@gabrieljoelc SGTM.

[michaelklishin]

Using a GitHub org is also a good idea. Note that GitHub forwards previous repo URLs for both git operations and HTTPS.

[gabrieljoelc]

I was finally able to articulate something that's been floating around in my mind about sneakers in this comment:

The general ambiguity of the direction of sneakers could play into a decision on a feature like this. I would like to see sneakers core stay more general, and opinionated or use-case-specific features like this be sneakers plugins in separate repos that could be registered as "middleware" or something. I would rather see work done on making sneakers more extendable than adding more configuration options added.

@michaelklishin @jondot thoughts on a way to register some kind of configuration-layer "middleware" as a way to extend sneakers and in contrast to adding more configuration options?

I'm thinking this would allow users more flexibility for low-level configuration of queues, channels, etc. It also might make using the Sneakers::Queue abstraction more useful when making your own retry handlers (i.e. user needs to create multiple queues for implementing exponential backoff).