search

mozilla-services / foxsec-pipeline

Log analysis pipeline utilizing Apache Beam
Mozilla Public License 2.0
25 stars 9 forks source link

HTTPRequest in some situations can begin to lag over time #470

Closed ameihm0912 closed 3 years ago

ameihm0912 commented 4 years ago

In some cases, HTTPRequest will begin to lag over time and worker CPU utilization begins to diverge. We can end up in a situation with a handful of workers pinned at max CPU, and many others in the 40-50% range. The reason for this is currently unknown, but may be related to sessions being kept for too long or some leak somewhere in the code.

ameihm0912 commented 4 years ago

I had hoped #464 would help with this but it doesn't seem to have.