docker memory lead

[likhita-8091]

Things to check first

• [X] I have checked that my issue does not already have a solution in the FAQ

• [X] I have searched the existing issues and didn't find my bug already reported there

• [X] I have checked that my bug is still present in the latest release

Version

3.9

What happened?

Please try executing this code inside the Docker container and check the Docker Stats, which has increased significantly. Here, a dynamic package is used, and the second job is in another custom.py file.

Here is just a simple example. My real code is actually calling GRPC, calling OpenCV's library to save images, and calling Open3D to save point cloud images. After performing several photo taking tasks, the memory reaches 1g and is not recycled.

How can we reproduce the bug?

Please try executing this code inside the Docker container and check the Docker Stats, which has increased significantly. Here, a dynamic package is used, and the second job is in another custom.py file. @agronholm

import importlib
import logging
import sys
import time
import signal

from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.events import *
from flask import Flask

app = Flask(__name__)

class J:
    def __init__(self, sc: BackgroundScheduler, isError: bool):
        self.sc = sc
        self.is_error = isError
        self.ls = []

    def append(self):
        self.ls.append(None)

    def run(self):
        self.sc.add_listener(self.listen, EVENT_JOB_EXECUTED | EVENT_JOB_ERROR)
        self.sc.start()
        self.sc.add_job(self.job0, trigger="date", name="f1", id="xxx")

    def shutdown(self):
        from apscheduler.schedulers.base import STATE_STOPPED
        if self.sc.state != STATE_STOPPED:
            print("shutdown....")
            self.sc.shutdown(wait=False)

    def listen(self, event):
        print("listen event...", event.exception)
        if event.exception:
            print("error: ", event)
            self.shutdown()

        else:
            print("job0 ok...")
            if len(self.ls) == 2:
                print("job1 end...")

                self.shutdown()

                return
            else:
                print("next job...")
                module = importlib.import_module("custom")
                print(module)
                cal_func = getattr(module, "cal")
                self.sc.add_job(cal_func, trigger="date", name="n", id="nnn")
                self.append()

    def job0(self):
        time.sleep(1)
        print("job init")
        time.sleep(1)
        self.append()

@app.route('/')
def hello_world():
    s = BackgroundScheduler()
    j = J(s, isError=False)

    j.sc._logger = ll
    j.run()

    return 'Hello World!'

def Exit(sig, frame):
    print("weld agent exit...")
    sys.exit(0)

def listen_signal():
    """
    监听停止信号
    """
    signal.signal(signal.SIGINT, Exit)
    signal.signal(signal.SIGTERM, Exit)

if __name__ == '__main__':
    listen_signal()
    logging.basicConfig(level=logging.DEBUG)
    ll = logging.getLogger()
    logging.info("xxx")
    l = logging.getLogger()
    try:
        # import gc
        #
        # gc.enable()
        time.sleep(1)
        print("start。。。")
        app.run()
        time.sleep(100)
    except Exception as e:
        print(e)

custom.py

def cal():
    n = 0
    while 1:
        print("cal log...: ", n)
        b = [2] * (2 * 10 ** 7)
        n += 1
        if n == 3:
            break

[agronholm]

If you cannot reproduce this problem outside of Docker, don't you think Docker is the problem then?

[likhita-8091]

If you cannot reproduce this problem outside of Docker, don't you think Docker is the problem then?

I have tried many times, and I have executed the same code on my own window system without any problems. But there is one in Docker, mainly because I am not an expert in Docker and Linux, so I still hope you can investigate. I believe you. The main thing is that if you don't use your library and just start the function with a simple thread, there won't be this problem. Your suspicion of executing this function through your library is correct; -It may be due to the Python version -It could be due to Docker -It may be due to the Linux system -It may be due to your library

[agronholm]

So if you run it on Linux without Docker, can you still see the problem? I'm not going to investigate this unless there is sufficient cause to believe that this is specifically caused by APScheduler, so it falls to you to eliminate other causes of the problem.

[likhita-8091]

It has been eliminated. I tried to directly start the cal function using Python's threading module, but there was no memory growth and it was immediately reclaimed. So you also need to personally try the library to add a task and give it a try.

[agronholm]

Ok, so this can be reproduced also without Flask?

[likhita-8091]

Yes, I just used flask to replace one of my task triggers

[agronholm]

Ok, so could I trouble you to construct a minimal example (that is, doesn't involve any 3rd party libraries it that are not necessary to reproduce the problem) that just uses apscheduler to demonstrate the issue?

[likhita-8091]

Hello, I'm sure your library has an impact,

1. I call a take_ The photo function, looped 10 times, is simply called. After the program starts, it takes 230mb and is called 260mb.
2. I still loop 10 times, but I opened 10 threads to execute the take_ Photo. It will be around 330-350mb
3. If using your scheduler, add_ Job, if you put 10 photos into the job, the memory will be 500-800mb. It's amazing. Why not release it? I waited all night and won't release it.

[likhita-8091]

[agronholm]

If using your scheduler, add_ Job, if you put 10 photos into the job, the memory will be 500-800mb. It's amazing. Why not release it? I waited all night and won't release it.

What are you talking about? Why not release what? What did you wait all night on?

[likhita-8091]

My goodness, there's a problem with your library, why not solve it? As an author, you don't do anything. So lazy? They all said that memory is not released and there is a memory leak,

[agronholm]

My goodness, there's a problem with your library, why not solve it? As an author, you don't do anything. So lazy? They all said that memory is not released and there is a memory leak,

My goodness, aren't you feeling entitled? I've asked you to provide a minimum workable example, but you've done nothing. So lazy? Do you think I owe you something?

[xinyanX]

I am afraid this problem do exist. I have verified many methods to recycle memory explicitly, but memory still holds after one instance have been excuted and will accumulate a lot after several instances. However if I run the task just once without apscheduler, the memory use by that docker container will drop at once. Really hope this can be investigated. Thanks a lot as there is really no better alternative package to use.

[agronholm]

Maybe open a separate ticket where you provide a method to reproduce this (preferably without Docker) and as many details as you can. This particular issue is a bit loaded with excess negativity.