review acts_as_xapian code

[mhl]

It would be worth carefully going over the code in acts_as_xapian, particularly looking out for anything that might be causing excessive memory usage and garbage collection.

[mhl]

Scheduled for Monday with @crowbot

[mhl]

It's apparent that working on acts_as_xapian in any significant way is likely to be a big task. We've come up with the following rough plan of what we'd like to do if time allows, however, and these tasks could be added to future sprints to gradually pay off the technical debt.

What we're trying to achieve:

Address problems with Alaveteli resource consumption

For example, we're worried about:

• CPU usage and memory usage in alert_tracks_loop
• Memory usage in application threads in practice, serving a variety of pages, for example:
  • Finding similar requests in the sidebar
  • Front page display of requests

What want to address this is performance integration tests in the test suite (not run by default, though).

The suggested plan for this is to write a script that can run in a reasonably short amount of time to generate a large test dataset for performance testing; we'll be able to instrument this more easily than the live site, and can try diagnostic tools that only work with later Ruby version, for example.

Suggested initial experiments for this, with the memcache caching disabled:

1. Repeatedly hit the front page (monitor for memory)
2. Repeatedly hit random request pages (monitor for memory)
3. Run the alert_tracks_loop job (monitor for CPU and memory)

A potentially big benefit of this work is having a reusable test setup for looking at performance problems like these; each time these come up it seems that lots of analysis has to happen from scratch again.

Make acts_as_xapian more maintainable and easier to extend

We're concerned that this course of action wouldn't produce the concrete improvements to the site that we'd like to see through the amount of work involved, but theoretically a good course of action might be:

• [ ] Add functional tests to check that basic end-to-end functionality still works, e.g. creating a new database, adding terms to it, checking they have been added, reindexing, searching, searching for similar terms, etc. These would be high level tests to check that subsequent refactoring doesn't break core functionality.
• [ ] Write unit tests (in as far as possible) for the class methods of ActsAsXapian, mocking the Xapian Ruby bindings.
• [ ] Start to refactor the class methods of ActsAsXapian; for example:
  • reduce the length of very long methods
  • reduce the amount of global-like state (the many class variables)
• [ ] Publish as a separate gem again
• [ ] Add new features, e.g. live updates to the database to address #1168