ericholscher / django-test-utils

Utilities for testing Django applications
http://django-test-utils.readthedocs.org/
MIT License
343 stars 69 forks source link

Crawler: plugin, output overhaul #12

Closed acdha closed 14 years ago

acdha commented 14 years ago

There are a bunch of changes here if you wanted to discuss/sprint at djangocon. The two major changes I found useful are:

  1. a bit of cleanup to the plugin system so fewer things are loaded by default and it's easier to load optional plugins. This allows things like not paying the BeautifulSoup parse tax (5x or so on my system) unless you want it or trivially loading code from any python module so the process of writing new modules doesn't involve forking django-test-utils.
  2. Start of a uniform output system - now you can start the crawler with an output_dir parameter (--output-dir on the management command) and each plugin has the opportunity to create whatever output files make sense. My initial goal here is for the query count and guppy plugin output to be plotted in Hudson across test runs to catch performance regressions but it'd make a ton of sense to work on a broader config system so people can pass optional config to the plugins as well (e.g. I only want a top-10 CSV file, save detailed heap dumps, etc.)