google / timesketch

Collaborative forensic timeline analysis
Apache License 2.0
2.58k stars 589 forks source link

Use subqueryload to make loading events with comments faster #3049

Closed tomchop closed 6 months ago

tomchop commented 6 months ago

When a sketch has a lot of commented events, performance starts to seriously degrade. In my dev environment, a sketch (450k total events) with 1 comment on 3k events would take 14 seconds to load.

The bottleneck was in the way comments are queried from the database via the CommentMixin class, which AFAIU dynamically creates a SQLAlchemy table definition. The Explore API goes through all events in a sketch to populate their comment dictionary. Using subqueryload to eagerly load comments in one go turned out to speed the queries up a lot (went from 14 seconds to 0.5 seconds for the same sketch).