Memory Leak? Hide keeps consuming more memory the more project it had created.

[wistuba]

I've started some larger experimentation using Hide where I use multiple processes and sequentially keep creating new projects. I've noticed that over time, the RAM consumption increases until I'm running out of memory.

My flow is very simple and basically does the following:

while True:  # technically using multiprocessing.Pool with 10 workers
  hc = Client()
  project = hc.create_project()
  # ...
  hc.delete_project(project)

The RAM is not consumed by the Python program. Killing it does not free the memory. Only killing hide does.

delete_project seems to correctly delete the docker containers.

My workaround for now: create a small set of projects, finish generation with them, and then kill hide. repeat until done.

[artmoskvin]

Thanks for reporting! Just to confirm, deleting the project does not release the resources (ie memory)?

[wistuba]

That's my understanding. At least the docker container change their status from "up" to "exited (137)". As far as I can tell, not calling delete_project will keep them in status "up".

Let me try to create and share a reproducible example.

[wistuba]

The following script shows the mentioned behavior. Anything I'm missing?

from multiprocessing import Pool

from hide.client.hide_client import HideClient

from hide import model
from hide.devcontainer import ImageDevContainer

def process_func(i):
    print(i)
    hc = HideClient()
    project = hc.create_project(
        repository=model.Repository(url=f"https://github.com/awslabs/Renate"),
        devcontainer=ImageDevContainer(image="continuumio/miniconda3"),
        languages=[model.Language.PYTHON],
    )
    hc.delete_project(project)

def main():
    with Pool(10) as pool:
        pool.map(process_func, [i for i in range(999)])

if __name__ == "__main__":
    main()

[artmoskvin]

The code looks legit. I'm trying to reproduce the problem locally now. Will get back soon.

[artmoskvin]

I can confirm there's a memory leak. For some reason, LSP servers keep running, even though shutting them down is part of the project cleanup procedure. Killing hide solves the problem because hide is a parent process for all LSPs. I will inspect the code for bugs now.

[artmoskvin]

Prepared a PR https://github.com/hide-org/hide/pull/128 We'll merge it tomorrow morning.

[wistuba]

awesome, thank you!