mozilla-services / foxsec-pipeline

Log analysis pipeline utilizing Apache Beam
Mozilla Public License 2.0
25 stars 9 forks source link

add support for ctxsvc - pipeline nginx logs + session limit analysis #535

Closed kkleemola closed 3 years ago

kkleemola commented 3 years ago

First pass at support needed for ctx svc.

Modifications to nginx parsing to support messages forwarded through pipeline proxy + first pass at a simple session limit heuristic. I'll likely change the windowing on this as I don't think the aggregating windows are the right option here, but this has been running using prod data for testing and I had wanted to try our existing windowing strategies before adding more.