search

dbt-labs / dbt-rpc

A server that can compile and run queries in the context of a dbt project. Additionally, it provides methods that can be used to list and terminate running processes.
https://docs.getdbt.com/reference/commands/rpc
Apache License 2.0
19 stars 7 forks source link

[CT-350] Memory leak with repeated compiles #75

Closed barberscott closed 2 years ago

barberscott commented 2 years ago

dbt in rpc mode appears to leak memory as commands are executed, increasing during command execution, but then not falling back to the original memory usage following the command execution.

In this example, tracking the RAM in a container running dbt in rpc mode, you see this phenomenon -- the peaks are runs of the dbt compile command where the effect is most pronounced, but we also see this with partial parse re-compiles, DAG rendering via the DAG-in-IDE feature in dbt Cloud, file save operations, etc.

Screen Shot 2022-03-10 at 9 16 28 PM

ChenyuLInx commented 2 years ago

The issue is caused by keeping a record of too many previous request results. We have garbage collection logic implemented but need to reduce the limit on things we keep in memory

davidharting commented 2 years ago

Thanks for this fix @ChenyuLInx!