jheld
commented
4 years ago @johnarnold curious for your thoughts here. I'm ready to do some work on this (already started), but don't want to do too much if I'm going down the wrong path.
@mher if you're interested, please consider.
Anyone else?
From #821 PR, I'm looking to make the usage of alternative apps/event classes more pluggable for flower.
https://packaging.python.org/guides/creating-and-discovering-plugins/
Plugin discovery/usage can be name-based. I know celery prefers to basically install all available brokers in-house to the project, but these could be broken out if needed and updated at their own cadence.
I'm looking to propose solutions such as:
There a couple reasons for why I'm looking to do this.
I have a fork which allows use of elasticsearch for the history of the tasks. It's much faster.
It's hard to get large code changes into flower (getting approval & merge). Breaking these out to a plugin architecture of some sort will allow agility to be more natural, even if the core project tends to be more stable.
I have broken out between app and indexer. The app is there for the web server & (if wanted), the event history.
The indexer (name can be changed) is specifically there to be run in history mode to be receiving the events and "indexing" them somewhere. The app can be told how to talk to the history server (e.g. elasticsearch).
You could run the normal app & use elasticsearch as opposed to the in-memory event receiver. But I would like to argue that for production use the runtimes be split. I have a working implementation of this (even having a separate entry point command native in flower's setup.py on my fork).
What do we think? I don't particularly want to be the only contributor to my fork (my company uses it extensively) -- it would be great to see others use it and improve upon it (or the plugin system itself). We could even have the Event in memory receiver still exist, but use it on the indexer command and have the app know to communicate via socket, as others have suggested (before and during my elasticsearch work, here).
The python plugin architecture looks a bit static to me, as opposed to being able to pass classpath/module strings into CLI/config, but maybe I'm not seeing the bigger picture here.
I have made a version of that work on my fork, and then I realized python may have a better way, hence the research.