yuval9313 / FastApi-RESTful

Quicker way to develop FastApi
MIT License
180 stars 25 forks source link

[FEATURE] execute repeat tasks on a single worker #282

Open TheCodeYoda opened 8 months ago

TheCodeYoda commented 8 months ago

Is your feature request related to a problem? Please describe. execute functions under the repeat_every decorator on a single worker. (when I'm running multiple workers)

Additional context Add any other context or screenshots about the feature request here. https://github.com/dmontagu/fastapi-utils/issues/230

TheCodeYoda commented 8 months ago

Don't know if this is the correct way, but leveraged file system locks to pin a repeat task on a single worker(the first worker). locked_pid will only be set for the first worker which manages to create FILE_SYSTEM_LOCK_PATH successfully.


locked_pid=None
FILE_SYSTEM_LOCK_PATH="<path-to-dummy-lock-directory>"
@api.on_event("startup")
async def acquire_lock():
    global locked_pid
    pid = os.getpid()
    try:
        if locked_pid is None:
            os.mkdir(FILE_SYSTEM_LOCK_PATH)
            locked_pid = pid
            logger.info(f"lock acquired by pid={pid}!")
    except FileExistsError:
        logger.warn(f"lock already acquired!")

@api.on_event("startup")
@repeat_every(seconds=20 raise_exceptions=True)
async def periodic_task():
    global locked_pid
    if locked_pid:
        pid = os.getpid()
        do_work()

@api.on_event("shutdown")
async def release_lock():
    os.rmdir(FILE_SYSTEM_LOCK_PATH)
    logger.info("lock released!")
TheCodeYoda commented 8 months ago

@yuval9313 let me know if the above solution is feasible, if yes I would like to contribute on building this out. Thanks.