search

NicolasLM / spinach

Modern Redis task queue for Python 3
https://spinach.readthedocs.io
BSD 2-Clause "Simplified" License
63 stars 4 forks source link

Add Engine.list_queue(), Broker.list_queue() commands #11

Closed 0xDEC0DE closed 3 years ago

0xDEC0DE commented 3 years ago

This adds a simple interface for inspecting the contents of queues, to facilitate apps making rudimentary decisions based on e.g., queue size, types of jobs in the queue, etc.

NicolasLM commented 3 years ago

Thank you for your contribution, I am a bit unsure about this feature.

I believe you have a good understanding of Spinach and its internals and so you can make informed decisions based on the the data in the queue, but I don't expect other users to take the time to fully understands how the data is stored internally.

There are a lot of subtlety related to future jobs, retries, idempotent jobs, locks... that make this data hard to use in practice. I can totally imagine users stumbling upon that call and use it to implement things like a task waiting on another task, that will break in subtle way.

There is also the fact that exposing this data makes it de facto part of the API, meaning that it adds the burden of trying to make it somewhat backwards compatible in the future.

As it stands, this calls seems so easy to add to your own code base (just subclass the broker, you don't even need a Lua script) that I don't think it is worth putting in Spinach itself.

0xDEC0DE commented 3 years ago

This was a wrong-headed approach towards improving the level of available info about the Spinach task queue.

A simple script to inspect the same data and send it into our monitoring solution gave us all the same data, and since it's not tied to our application, we can update it outside of app release cycles. Plus, we got pretty graphs for free.

Closing.