c4lm
commented
2 weeks ago Hey @ryorke1,
Do you have some kind of proxying for that server_api_url? It's supposed to end with /api because orkes-conductor-server image has proxying for /api internally for API queries - do the logs show that the workers are polling? Does the pod have enough CPU for garbage collection?
Have you tried newer or older python images? IIRC there were changes in 3.11 which were supposed to bring more performance at the cost of a bit more memory, and those changes also initially caused a few leaks in some libraries.
I would appreciate it if you could share Dockerfile + compose yaml for the worker with cpu/memory capped so that we could use them to reproduce the issue. It does not seem to occur on my machine w/ python 3.11.9/conductor-python=1.1.7 .
ryorke1
Hello everyone,
My team seems to have stumbled upon a really weird issue with the conductor python sdk where the memory of the worker continues to climb until the worker runs out of resources (in our case our OpenShift cluster OOMKills the pod due to the resource utilization exceeding it's allocated quota). What is odd is the timing of the memory increasing is not consistent. Sometimes it will start as soon as the worker starts while other times it will start 30 to 45 minutes after the application starts. Additionally, once in a while it seems to not have this issue. I have tried tracing through the polling functionality within the SDK and cannot seem to find anything that stands out as the cause of the memory. Wondering if anyone else is experiencing this issue and what is the root cause?
This graph shows the example code below running and being restarts. When you see a long flat line that is when we turned it off and tried looking at code. Considering this workflow does absolutely nothing (execute doesn't even get called) I wouldn't expect to see a bunch of spikes in memory and this should just be flat for the most part.
Steps to reproduce:
Sample Code: ` from conductor.client.automator.task_handler import TaskHandler from conductor.client.configuration.configuration import Configuration from conductor.client.http.models import Task, TaskResult from conductor.client.worker.worker_interface import WorkerInterface
configuration = Configuration( server_api_url=f"http://localhost:3500/", debug=True )
class SillyWorker(WorkerInterface): def execute(self, task: Task) -> TaskResult | None: return None
workers = [ SillyWorker(task_definition_name="silly-worker-1"), SillyWorker(task_definition_name="silly-worker-2"), SillyWorker(task_definition_name="silly-worker-3"), SillyWorker(task_definition_name="silly-worker-4"), SillyWorker(task_definition_name="silly-worker-5"), SillyWorker(task_definition_name="silly-worker-6"), SillyWorker(task_definition_name="silly-worker-7"), SillyWorker(task_definition_name="silly-worker-8"), SillyWorker(task_definition_name="silly-worker-9"), SillyWorker(task_definition_name="silly-worker-10"), SillyWorker(task_definition_name="silly-worker-11"), SillyWorker(task_definition_name="silly-worker-12"), ]
def start_workers():
if name == "main": start_workers() `
Environment: