search

Snorby / snorby

Ruby On Rails Application For Network Security Monitoring
Other
1k stars 226 forks source link

Mass Classification silently fails on large jobs (with :handler bodies >64k) #325

Open cventers opened 11 years ago

cventers commented 11 years ago

We have an environment with a lot of sensors and daily automatic rule updates. Sometimes a new rule that we have no suppression rule for will go crazy and generate a mountain of false positives (in one case, enough events to generate an "ids" string of about 4 MB in length). When attempting background mass classification on such events, Snorby 2.6.2 (which I have hand patched to fix #291) will fail to kick off the background job at all.

I debugged the problem and determined there are a series of issues with such mass classifications:

  1. Snorby is not checking the delayed_job object returned from Delayed::Job.enqueue() in app/controllers/events_controller.rb to see if it was created successfully or not, so even when ActiveRecord/DataMapper/whatever refuses the INSERT, a success message is shown on the screen instead of an error;
  2. The delayed_jobs table is being created with a TEXT column for :handler, which is limited to 64k;
  3. The datamapper for delayed_jobs is using the default limit of 64k, even when I manually altered our snorby database to use FULLTEXT for :handler;
  4. Even when I manually fixed the datamapper :handler column for delayed_jobs to accept a large body, the mass classification POST request bombs out with:

DataObjects::ConnectionError (Got a packet bigger than 'max_allowed_packet' bytes): app/controllers/events_controller.rb:256:in `mass_action'

At this point I've given up trying to hand patch the issue in our installation. I'm not sure where to begin with the fourth problem, but here are some suggestions on the others:

  1. Check the return value of Delayed::Job.enqueue() and render an error if it didn't work;
  2. Either fix upstream, monkeypatch, delay the Snorby::Search until the innards of the MassClassification handler, or accept that mass classification will be broken for large requests but will at least return an error.