Support for plugins - Githubissues

heyman commented 12 years ago

I'd like locust to have a plugin system that would provide the ability to extend locust with different functionality. We could then probably refactor out some of the core functionality into plugins that are loaded by default and resides in a contrib package.

I think we could get a long way if we would implement a plugin system with the following features:

A way for plugins to add custom tabs to the web UI
An event that's triggered when a web UI client retrieves new data through AJAX, where a plugin can add data
A client side event when new data has been retrieved, where plugin code can process/present any data that was added in the server side event
A (documented) way of adding endpoints/routes to the web UI from within a plugin
A convention on how to structure plugins
A way to specify what plugins to load in the locust test file(s).
Go through the current internal events and make sure they provide as much data as possible
Provide a good way of writing unit tests for plugins
Good documentation on how to write plugins

I believe the above features would cover a majority of the different plugin use cases one might have. I also think it's good to start small and keeping the plugin API as small as possible, since there's less risk of making bad API decisions and they will be easier to correct :).

Jahaja commented 12 years ago

Agree. This would be a great improvement. It would probably simplify/clean up the current codebase as well.

EnTeQuAk commented 11 years ago

https://github.com/getsentry/sentry and https://github.com/mozilla-services/circus have some good concept of plugins, maybe orientate on these?

sanga commented 10 years ago

Have you guys made any progress on clarifying how plugins might work? We have been running a very slightly hacked up version of locust for about the last 6 months that allows us to run a plugin that publishes metrics from locust to graphite.

I would be happier to not to have to run a patched locust. And also I guess there's no reason why that graphite plugin couldn't be open-sourced once the plugin system is clarified

Jahaja commented 10 years ago

Hi Tim, no I don't think we have unfortunately. I would like to be able to spend more time on this but it's not looking very likely that I will be able to. At least for the next month or so.

heyman commented 10 years ago

Hi Tim!

I've been thinking about how we could implement plugins in a good way, but haven't gotten into any actual implementation yet. Hopefully I'll be able to tinker with it soon.

What specific things is it that you would need in Locust for your graphite hack, in order to not have to run a patched version? It's great to have examples of real use-cases to consider when designing the plugin system.

I've started implementing percentile statistics in the web UI as discussed in #140. I might use that as a use case for the plugin architecture as well.

Jahaja commented 10 years ago

We should create a list of like at least 5 good use-cases. Basically a list of future plugins. This should indeed give us an idea of what's needed of the plugin system. Graphs seems like a good start.

sanga commented 10 years ago

The plugin that I currently have is just a greenlet running in locust that periodically grabs the stats from locust and posts them up to graphite. To do that, I just added a flag --plugin=[path of module to load], that imports that path and hence starts that logger greenlet. However when I think about it now, I guess there's no need for that - I could instead just write a wrapper around locust that starts both locust and the logger greenlet. There was another small patch that just moved some stats logic out of the web app, so I didn't have to call the url handler.

The above is pretty trivial and perhaps not really what would give you a good idea of how to design a plugin. Something more adventurous which I had planned to do with locust would be something like the same thing plus:

log a hatch_complete event up to graphite's event api from locust
collect timing information i.e. in which parts (web frontend, memcache, db etc) of the system the request spent time (that we pass back in an HTTP header) and collect and display that info in locust

That kind of thing would be more what you're interested in I guess - as it'd need changes on master and slave and also the web ui.

sanga commented 10 years ago

and while I think about it, another good candidate might be the ramping functionality that was removed to be reimplemented as a plugin (just as a reminder). At least i would find that useful.

heyman commented 10 years ago

Great! Let's compile a list:

Plugin for...

...showing percentile statistics table in the web UI (which shows the same stats as what is currently available as a CSV download).
...periodically pushing stats to some external system (e.g. Graphite).
...displaying graphs of request statistics over time, in the web UI
...ramping up/down the number of spawned locusts
...history of tests and results
...scheduling of tests
...simple timer that shows how long a test has run, and the ability to schedule a stop of a running test (I remember @hjlarsson requesting this feature some time ago)

heyman commented 10 years ago

@sanga: Slightly off topic from the plugin discussion, but couldn't you just spawn your logger greenlet inside your locust test file(s), (at the module level), and then not have to run a patched version of Locust?

sanga commented 10 years ago

I have a feeling there was some reason I couldn't do that. But no recollection of what that was...

Jahaja commented 10 years ago

Great! Let's compile a list:

Plugin for...

...showing percentile statistics table in the web UI (which shows the same stats as what is currently available as a CSV download). ...periodically pushing stats to some external system (e.g. Graphite). ...displaying graphs of request statistics over time, in the web UI ...ramping up/down the number of spawned locusts

History of tests and results (saved to some backend; redis, sqlite, json files etc)

Scheduling of tests

heyman commented 10 years ago

@Jahaja , I've added those two to the list above, as well as another one. So now we have 7 use-cases, which is probably enough to get us started.

techdragon commented 10 years ago

any progress on this... the need to write extra code to deal with graphite or statsd when it could be standardised is a major pain.

ghost commented 9 years ago

I am also interested in locust pushing data to graphite.

sanga commented 9 years ago

@heyman I'd forgotten about this completely but now I recall the reason why I didn't just start the graphite logging greenlet in our scenario. Because you only want that running on the master not the slaves. I could have changed only the master scenario to include the logger greenlet but because of the way we distribute the scenarios it was more convenient to just write a 5 line patch for locust.

venth commented 8 years ago

Any progress on plugin concept? I'm interested in percentiles

arthurzenika commented 7 years ago

Would be interested in a statsd plugin

gplusplus314 commented 7 years ago

I'm very interested in a plugin architecture. I love the underpinnings of Locust, but I'd like to add some opinionated functionality that some people wouldn't agree with. I don't want to implement these features and force them upon everyone. :)

Say I were to implement the plugin functionality; do any of the primary contributors have any particular design considerations?

The first thing I would like to implement is a way to implement additional HTTP routes/handlers into the main/master process. This would open many doors.

Thanks!

FrancisJen commented 7 years ago

I also hope locust can open a world for plugin, It is possible to be greater for testing engineer who only know python, because they can use python to do automation testing on selenium, but for performance testing. Jmeter is very popular because its support for plugin, if locust can support plugin, It will be more popular.

mbeacom commented 7 years ago

Maybe we could start a GH project for this repo and scope a plugin system? @heyman @aldenpeterson-wf Any thoughts? Are guys on board for setting up a board in projects outlining what has been discussed here and further scoping it out?

aldenpeterson-wf commented 7 years ago

yeah, I'd be interested, that seems like it could be pretty neat.

It'd be useful to be able to more customize locust too without requiring actually modifying the core codebase, too.

mpaddle commented 7 years ago

Any update on this? A plugin system would be really useful. I'm not a heavy user of locust, but some of the functionality I implemented in my tasks could have been provided by a plugin.

aldenpeterson-wf commented 7 years ago

I'm curious what this might look like implementation wise.

I can envision it happening in a variety of ways. One easyish way would just be starting another greenlet with the plugin running and it seems that'd be straightforward to actually add. But it would put a lot of responsibility on the plugin developer to not break things or block. In other words mimicing how some of the internal functions work like the stats outputs - https://github.com/locustio/locust/blob/master/locust/main.py#L485-L490 - which are "pluginesque."

Anyone have thoughts about what might work for a plugin?

cyberw commented 5 years ago

This is a good ticket, but I think it is too big. Also, some of the things mentioned here can be solved without an explicit "plugin" architecture (although the web UI stuff might need tighter integration)

Ok to close? (in favour of accepting new, smaller tickets)

heyman commented 5 years ago

@cyverw Yes, I agree. I think a first step towards providing an official plugin API, would be some refactoring of the code in mainly runners.py and making some of it part of the official Locust API by allowing users to run tests programatically and use Locust as an ordinary python library. But that should definitely be in a separate issue/PR.

locustio / locust

Support for plugins #34