search

airbnb / streamalert

StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.
https://streamalert.io
Apache License 2.0
2.86k stars 334 forks source link

Improvement: Multiprocessing of classification #232

Open jacknagz opened 7 years ago

jacknagz commented 7 years ago

Background

AWS Lambda supports a thread limit of 1024 per execution. We currently are not taking advantage of this feature when processing logs with StreamAlert.

Desired Outcome

Usage of Python's multiprocessing builtin library to parallelize the classification and rules matching engine.

ryandeivert commented 7 years ago

PR #273 is open with a POC for this but should be further investigated.