search

agronholm / apscheduler

Task scheduling library for Python
MIT License
6.3k stars 712 forks source link

App crash #952

Open mmmcorpsvit opened 3 months ago

mmmcorpsvit commented 3 months ago

Things to check first

Version

4.0.0a5

What happened?

work app

How can we reproduce the bug?

from asyncio import run, sleep
from datetime import datetime

from apscheduler import AsyncScheduler
from apscheduler.triggers.interval import IntervalTrigger

async def tick():
    await sleep(1000)
    print("Hello, the time is", datetime.now())

async def main():
    async with AsyncScheduler() as scheduler:
        # await scheduler.add_schedule(parser_copart_collect_csv_run2, IntervalTrigger(seconds=60))
        await scheduler.add_schedule(tick, IntervalTrigger(seconds=1))
        await scheduler.run_until_stopped()

run(main())

after some time app crash (windows 11, python 3.11.9):

mmmcorpsvit commented 3 months ago 
C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\cars-pRqyGtAI-py3.11\Scripts\python.exe D:\GoogleDrive\projects\cars-backend\worker.py 
INFO:apscheduler._schedulers.async_:Added new schedule (task='__main__:tick', trigger=IntervalTrigger(seconds=1, start_time='2024-08-10 01:49:19.465905+03:00')); next run time at 2024-08-10 01:49:19.465905+03:00
INFO:apscheduler._schedulers.async_:Cleaned up expired job results and finished schedules
INFO:apscheduler._schedulers.async_:Scheduler started
INFO:apscheduler._schedulers.async_:Cleaned up expired job results and finished schedules
Hello, the time is 2024-08-10 02:05:59.468589
INFO:apscheduler._schedulers.async_:Job 2474e627-e5ed-4994-a506-1cec5575e37e completed successfully
INFO:apscheduler._schedulers.async_:Job 0be9b4c3-101d-4382-bd2d-8dedcf6f1603 completed successfully
Hello, the time is 2024-08-10 02:06:00.479031
Hello, the time is 2024-08-10 02:06:01.476233
INFO:apscheduler._schedulers.async_:Job e38794ac-433d-4f89-845e-715ad72a6fd6 completed successfully

Hello, the time is 2024-08-10 02:06:12.497892
Hello, the time is 2024-08-10 02:06:13.480893
INFO:apscheduler._schedulers.async_:Job ca93c653-5326-49ce-853b-3ecb4b1003a7 completed successfully
INFO:apscheduler._schedulers.async_:Job 45c401da-fe3d-4f7a-b1ea-753f82edb551 completed successfully
Hello, the time is 2024-08-10 02:06:14.495734
INFO:apscheduler._schedulers.async_:Job 170ee964-f072-457d-b0fb-42314266635b completed successfully
Hello, the time is 2024-08-10 02:06:15.466828
INFO:apscheduler._schedulers.async_:Job 620ca19b-81f8-4966-aae5-3908c07d9821 completed successfully
Hello, the time is 2024-08-10 02:06:16.479123
INFO:apscheduler._schedulers.async_:Job 416c1f66-8303-4da2-ab06-d1817a5bfc11 completed successfully
Hello, the time is 2024-08-10 02:06:17.474106
INFO:apscheduler._schedulers.async_:Job c9eabd10-a76b-4172-8d31-b2947f112a09 completed successfully
Hello, the time is 2024-08-10 02:06:18.472178
Hello, the time is 2024-08-10 02:06:19.467102
INFO:apscheduler._schedulers.async_:Job bdcfa37b-4672-4dad-9d9c-0ffc0be4533f completed successfully
INFO:apscheduler._schedulers.async_:Job 706345c9-58bb-4787-a844-6a5003de1475 completed successfully
Hello, the time is 2024-08-10 02:06:20.480103
Hello, the time is 2024-08-10 02:06:21.479911
INFO:apscheduler._schedulers.async_:Job 7f6891b8-0ac5-40ec-96bc-5f30ee002c91 completed successfully
INFO:apscheduler._schedulers.async_:Job efc876d0-aa0b-4ab9-aadc-f8f18d322fcf completed successfully
Hello, the time is 2024-08-10 02:06:22.475297
...
Hello, the time is 2024-08-10 02:06:28.470972
INFO:apscheduler._schedulers.async_:Job 2474e627-e5ed-4994-a506-1cec5575e37e completed successfully
INFO:apscheduler._schedulers.async_:Job 6405a281-8415-4e32-a3ec-a91d5c188820 completed successfully
INFO:apscheduler._schedulers.async_:Job 9e4d1616-1262-4185-b12e-981ddf76eb73 was cancelled
INFO:apscheduler._schedulers.async_:Job 4236cf58-8a6e-41e1-bedf-f72d6810acfd was cancelled
INFO:apscheduler._schedulers.async_:Job eed93c7a-854c-476b-bec6-542387f7fd20 was cancelled
...
INFO:apscheduler._schedulers.async_:Job 5c428483-8908-4b83-9093-2216f5cfe78c was cancelled
INFO:apscheduler._schedulers.async_:Job 6fc1559c-9e61-475b-9621-015688db66c5 was cancelled
INFO:apscheduler._schedulers.async_:Job 3f23a3ba-a80b-4d6f-af7a-417bebb8321d was cancelled
INFO:apscheduler._schedulers.async_:Job 1fa9cb30-af0b-417f-9f79-63ae78c0ed55 was cancelled
ERROR:apscheduler._schedulers.async_:Scheduler crashed
  + Exception Group Traceback (most recent call last):
  |   File "C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\cars-pRqyGtAI-py3.11\Lib\site-packages\apscheduler\_schedulers\async_.py", line 784, in run_until_stopped
  |     async with create_task_group() as task_group:
  |   File "C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\cars-pRqyGtAI-py3.11\Lib\site-packages\anyio\_backends\_asyncio.py", line 680, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\cars-pRqyGtAI-py3.11\Lib\site-packages\apscheduler\_schedulers\async_.py", line 1018, in _process_jobs
    |     await wakeup_event.wait()
    |   File "C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\cars-pRqyGtAI-py3.11\Lib\site-packages\anyio\_backends\_asyncio.py", line 1662, in wait
    |     await self._event.wait()
    |   File "C:\Python311\Lib\asyncio\locks.py", line 213, in wait
    |     await fut
    | asyncio.exceptions.CancelledError: Cancelled by cancel scope 26e138be0d0
    | 
    | During handling of the above exception, another exception occurred:
    | 
    | Exception Group Traceback (most recent call last):
    |   File "C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\cars-pRqyGtAI-py3.11\Lib\site-packages\apscheduler\_schedulers\async_.py", line 988, in _process_jobs

.
.
.
packages\apscheduler\_schedulers\async_.py", line 1076, in _run_job
          |     self._running_jobs.remove(job.id)
          | KeyError: UUID('706345c9-58bb-4787-a844-6a5003de1475')
          +---------------- 14 ----------------
          | Traceback (most recent call last):
          |   File "C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\cars-pRqyGtAI-py3.11\Lib\site-packages\apscheduler\_schedulers\async_.py", line 1076, in _run_job
          |     self._running_jobs.remove(job.id)
          | KeyError: UUID('620ca19b-81f8-4966-aae5-3908c07d9821')
          +---------------- 15 ----------------
          | Traceback (most recent call last):
          |   File "C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\cars-pRqyGtAI-py3.11\Lib\site-packages\apscheduler\_schedulers\async_.py", line 1076, in _run_job
          |     self._running_jobs.remove(job.id)
          | KeyError: UUID('db37c1a7-46d5-41c5-9317-55aba714c9fe')
          +---------------- ... ----------------
          | and 2874 more exceptions
          +------------------------------------

Process finished with exit code 1
agronholm commented 3 months ago

Could you try again with the master branch? It has tons of fixes not yet in any release.

mmmcorpsvit commented 3 months ago

same issue, additionally check inside docker container - same(

agronholm commented 3 months ago

Ok, so the example you pasted in the description should reproduce the issue? After how much time?

agronholm commented 3 months ago

Would you mind pasting a traceback when using master? I'd like to see what exception you're getting this time.

Also, is the long sleep time of 1000 seconds truly necessary to reproduce the issue? Will it not happen with a shorter wait time? I reduced the sleep time to 30 seconds, and cannot reproduce the issue.

mmmcorpsvit commented 3 months ago 
import time
from asyncio import run, sleep
from datetime import datetime

from apscheduler import AsyncScheduler
from apscheduler.triggers.interval import IntervalTrigger

async def tick():
    await sleep(100)
    print("Hello, the time is", datetime.now())

async def main():
    start_time = time.time()

    async with AsyncScheduler() as scheduler:
        await scheduler.add_schedule(tick, IntervalTrigger(seconds=1))
        await scheduler.run_until_stopped()

    end_time = time.time()
    elapsed_time = end_time - start_time
    print(f"Scheduler ran for {elapsed_time:.2f} seconds.")

run(main())

I change code, crash after 3-5 minutes, always

in real case - yes, I have long time task - 5-10 minutes and need one instance only (if the task fails - autorepeat)

mmmcorpsvit commented 3 months ago

log.txt

agronholm commented 3 months ago

Ok, I was finally able to reproduce the issue on my end, thanks! I'll investigate and let you know when I have a fix.

agronholm commented 3 months ago

I'm seeing duplicate job IDs being added to self._running_jobs, and I can't explain why. I'm continuing to investigate.

agronholm commented 3 months ago

The data store is letting the scheduler acquire an already running job. This explains the KeyError.

agronholm commented 3 months ago

Ok, I figured it out. This happens because of the wrong looping condition in the nested function extend_job_leases() which causes it to exit before the loop runs even once.

agronholm commented 3 months ago

I have a test that reproduces the issue, but I still need to refactor it to not have to sleep(), as this introduces unwanted slowness to the test suite.

agronholm commented 3 months ago

Well, I got it working like I wanted...but the test that should fail without the patch is only failing for the memory data store. I'm investigating why it's not failing for sqlalchemy/mongodb stores.

HK-Mattew commented 3 months ago

I was able to reproduce the error using MongoDB Data Store.

from asyncio import run, sleep
from datetime import datetime

from apscheduler import AsyncScheduler
from apscheduler.triggers.interval import IntervalTrigger
from apscheduler.datastores.mongodb import MongoDBDataStore

MONGO_DB_URI = 'mongodb://127.0.0.1:27017'
MONGO_DB_NAME = 'try_bug_aps'

data_store = MongoDBDataStore(
    client_or_uri=MONGO_DB_URI,
    database=MONGO_DB_NAME
)

async def tick():
    await sleep(60)
    print("Hello, the time is", datetime.now())

async def main():
    async with AsyncScheduler(data_store=data_store) as scheduler:
        await scheduler.add_schedule(tick, IntervalTrigger(seconds=1))
        await scheduler.run_until_stopped()

run(main())
My execution logs

``` Hello, the time is 2024-08-10 15:45:34.702265 Hello, the time is 2024-08-10 15:45:34.849012 Hello, the time is 2024-08-10 15:45:34.993766 Hello, the time is 2024-08-10 15:45:35.139204 Hello, the time is 2024-08-10 15:45:35.283898 Hello, the time is 2024-08-10 15:45:35.429783 Hello, the time is 2024-08-10 15:45:35.574430 Hello, the time is 2024-08-10 15:45:35.719895 Hello, the time is 2024-08-10 15:45:35.943879 Hello, the time is 2024-08-10 15:45:36.034651 Hello, the time is 2024-08-10 15:45:36.770064 Hello, the time is 2024-08-10 15:45:36.913782 Scheduler crashed + Exception Group Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 893, in run_until_stopped | await self.event_broker.publish_local(SchedulerStarted()) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/datastores/mongodb.py", line 222, in _get_session | yield session | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/datastores/mongodb.py", line 666, in acquire_jobs | update_task_result = await to_thread.run_sync( | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/to_thread.py", line 56, in run_sync | return await get_async_backend().run_sync_in_worker_thread( | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2128, in run_sync_in_worker_thread | await cls.checkpoint() | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2050, in checkpoint | await sleep(0) | File "/usr/lib/python3.9/asyncio/tasks.py", line 641, in sleep | await __sleep0() | File "/usr/lib/python3.9/asyncio/tasks.py", line 630, in __sleep0 | yield | asyncio.exceptions.CancelledError: Cancelled by cancel scope 7b4f4bf43c70 | | During handling of the above exception, another exception occurred: | | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1143, in _process_jobs | jobs = await self.data_store.acquire_jobs( | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/datastores/mongodb.py", line 551, in acquire_jobs | async for attempt in self._retry(): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/tenacity/asyncio/__init__.py", line 166, in __anext__ | do = await self.iter(retry_state=self._retry_state) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/tenacity/asyncio/__init__.py", line 153, in iter | result = await action(retry_state) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/tenacity/_utils.py", line 99, in inner | return call(*args, **kwargs) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/tenacity/__init__.py", line 398, in | self._add_action_func(lambda rs: rs.outcome.result()) | File "/usr/lib/python3.9/concurrent/futures/_base.py", line 439, in result | return self.__get_result() | File "/usr/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result | raise self._exception | File "/usr/lib/python3.9/contextlib.py", line 653, in __aexit__ | cb_suppress = await cb(*exc_details) | File "/usr/lib/python3.9/contextlib.py", line 199, in __aexit__ | await self.gen.athrow(typ, value, traceback) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/datastores/mongodb.py", line 224, in _get_session | await to_thread.run_sync(session.end_session) | asyncio.exceptions.CancelledError: Cancelled by cancel scope 7b4f4bf43c70 | | During handling of the above exception, another exception occurred: | | Exception Group Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1162, in _process_jobs | wakeup_event = anyio.Event() | File "/usr/lib/python3.9/contextlib.py", line 670, in __aexit__ | raise exc_details[1] | File "/usr/lib/python3.9/contextlib.py", line 653, in __aexit__ | cb_suppress = await cb(*exc_details) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (16 sub-exceptions) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('03444e1b-fc1b-4282-9e3f-44695f46e7b5'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 17, 892262, tzinfo=datetime.timezone.utc)) +---------------- 2 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('face8cd4-7da8-4701-8190-9579e4ed441a'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 632025, tzinfo=datetime.timezone.utc)) +---------------- 3 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('90434ad8-7222-436e-a7eb-b3a031667c19'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 413478, tzinfo=datetime.timezone.utc)) +---------------- 4 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('d48bba75-5e31-4e4a-be75-3fa967f95dbb'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 18, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 19, 524304, tzinfo=datetime.timezone.utc)) +---------------- 5 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('600706a2-8a2d-49ac-8971-db7da6ede55a'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 18, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 781486, tzinfo=datetime.timezone.utc)) +---------------- 6 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('bd4b3289-c301-4055-9ff4-5545816d7635'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 19, 672938, tzinfo=datetime.timezone.utc)) +---------------- 7 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('b0a32ce2-d9d5-46c3-bcba-38146cf5f3a1'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 562387, tzinfo=datetime.timezone.utc)) +---------------- 8 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('b0a32ce2-d9d5-46c3-bcba-38146cf5f3a1'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 562387, tzinfo=datetime.timezone.utc)) +---------------- 9 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('600706a2-8a2d-49ac-8971-db7da6ede55a'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 18, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 781486, tzinfo=datetime.timezone.utc)) +---------------- 10 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('ec2ab9a6-4632-4af6-a41a-29c80b7b8c0d'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 21, 296291, tzinfo=datetime.timezone.utc)) +---------------- 11 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('03444e1b-fc1b-4282-9e3f-44695f46e7b5'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 17, 892262, tzinfo=datetime.timezone.utc)) +---------------- 12 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('bd4b3289-c301-4055-9ff4-5545816d7635'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 19, 672938, tzinfo=datetime.timezone.utc)) +---------------- 13 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('90434ad8-7222-436e-a7eb-b3a031667c19'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 413478, tzinfo=datetime.timezone.utc)) +---------------- 14 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('ec2ab9a6-4632-4af6-a41a-29c80b7b8c0d'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 21, 296291, tzinfo=datetime.timezone.utc)) +---------------- 15 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('face8cd4-7da8-4701-8190-9579e4ed441a'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 632025, tzinfo=datetime.timezone.utc)) +---------------- ... ---------------- | and 1 more exception +------------------------------------ + Exception Group Traceback (most recent call last): | File "/usr/lib/python3.9/contextlib.py", line 653, in __aexit__ | cb_suppress = await cb(*exc_details) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Exception Group Traceback (most recent call last): | File "/home/username/Documents/.gits/tryraisebugaps/start.py", line 23, in main | await scheduler.run_until_stopped() | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 893, in run_until_stopped | await self.event_broker.publish_local(SchedulerStarted()) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Exception Group Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1162, in _process_jobs | wakeup_event = anyio.Event() | File "/usr/lib/python3.9/contextlib.py", line 670, in __aexit__ | raise exc_details[1] | File "/usr/lib/python3.9/contextlib.py", line 653, in __aexit__ | cb_suppress = await cb(*exc_details) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (16 sub-exceptions) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('03444e1b-fc1b-4282-9e3f-44695f46e7b5'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 17, 892262, tzinfo=datetime.timezone.utc)) +---------------- 2 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('face8cd4-7da8-4701-8190-9579e4ed441a'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 632025, tzinfo=datetime.timezone.utc)) +---------------- 3 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('90434ad8-7222-436e-a7eb-b3a031667c19'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 413478, tzinfo=datetime.timezone.utc)) +---------------- 4 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('d48bba75-5e31-4e4a-be75-3fa967f95dbb'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 18, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 19, 524304, tzinfo=datetime.timezone.utc)) +---------------- 5 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('600706a2-8a2d-49ac-8971-db7da6ede55a'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 18, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 781486, tzinfo=datetime.timezone.utc)) +---------------- 6 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('bd4b3289-c301-4055-9ff4-5545816d7635'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 19, 672938, tzinfo=datetime.timezone.utc)) +---------------- 7 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('b0a32ce2-d9d5-46c3-bcba-38146cf5f3a1'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 562387, tzinfo=datetime.timezone.utc)) +---------------- 8 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('b0a32ce2-d9d5-46c3-bcba-38146cf5f3a1'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 562387, tzinfo=datetime.timezone.utc)) +---------------- 9 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('600706a2-8a2d-49ac-8971-db7da6ede55a'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 18, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 781486, tzinfo=datetime.timezone.utc)) +---------------- 10 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('ec2ab9a6-4632-4af6-a41a-29c80b7b8c0d'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 21, 296291, tzinfo=datetime.timezone.utc)) +---------------- 11 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('03444e1b-fc1b-4282-9e3f-44695f46e7b5'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 17, 892262, tzinfo=datetime.timezone.utc)) +---------------- 12 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('bd4b3289-c301-4055-9ff4-5545816d7635'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 19, 672938, tzinfo=datetime.timezone.utc)) +---------------- 13 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('90434ad8-7222-436e-a7eb-b3a031667c19'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 413478, tzinfo=datetime.timezone.utc)) +---------------- 14 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('ec2ab9a6-4632-4af6-a41a-29c80b7b8c0d'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 21, 296291, tzinfo=datetime.timezone.utc)) +---------------- 15 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('face8cd4-7da8-4701-8190-9579e4ed441a'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 632025, tzinfo=datetime.timezone.utc)) +---------------- ... ---------------- | and 1 more exception +------------------------------------ During handling of the above exception, another exception occurred: + Exception Group Traceback (most recent call last): | File "/home/username/Documents/.gits/tryraisebugaps/start.py", line 30, in | run(main()) | File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run | return loop.run_until_complete(main) | File "/usr/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete | return future.result() | File "/home/username/Documents/.gits/tryraisebugaps/start.py", line 23, in main | await scheduler.run_until_stopped() | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 195, in __aexit__ | await self._exit_stack.__aexit__(exc_type, exc_val, exc_tb) | File "/usr/lib/python3.9/contextlib.py", line 670, in __aexit__ | raise exc_details[1] | File "/usr/lib/python3.9/contextlib.py", line 653, in __aexit__ | cb_suppress = await cb(*exc_details) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Exception Group Traceback (most recent call last): | File "/home/username/Documents/.gits/tryraisebugaps/start.py", line 23, in main | await scheduler.run_until_stopped() | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 893, in run_until_stopped | await self.event_broker.publish_local(SchedulerStarted()) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Exception Group Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1162, in _process_jobs | wakeup_event = anyio.Event() | File "/usr/lib/python3.9/contextlib.py", line 670, in __aexit__ | raise exc_details[1] | File "/usr/lib/python3.9/contextlib.py", line 653, in __aexit__ | cb_suppress = await cb(*exc_details) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (16 sub-exceptions) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('03444e1b-fc1b-4282-9e3f-44695f46e7b5'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 17, 892262, tzinfo=datetime.timezone.utc)) +---------------- 2 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('face8cd4-7da8-4701-8190-9579e4ed441a'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 632025, tzinfo=datetime.timezone.utc)) +---------------- 3 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('90434ad8-7222-436e-a7eb-b3a031667c19'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 413478, tzinfo=datetime.timezone.utc)) +---------------- 4 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('d48bba75-5e31-4e4a-be75-3fa967f95dbb'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 18, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 19, 524304, tzinfo=datetime.timezone.utc)) +---------------- 5 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('600706a2-8a2d-49ac-8971-db7da6ede55a'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 18, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 781486, tzinfo=datetime.timezone.utc)) +---------------- 6 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('bd4b3289-c301-4055-9ff4-5545816d7635'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 19, 672938, tzinfo=datetime.timezone.utc)) +---------------- 7 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('b0a32ce2-d9d5-46c3-bcba-38146cf5f3a1'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 562387, tzinfo=datetime.timezone.utc)) +---------------- 8 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('b0a32ce2-d9d5-46c3-bcba-38146cf5f3a1'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 562387, tzinfo=datetime.timezone.utc)) +---------------- 9 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('600706a2-8a2d-49ac-8971-db7da6ede55a'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 18, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 781486, tzinfo=datetime.timezone.utc)) +---------------- 10 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('ec2ab9a6-4632-4af6-a41a-29c80b7b8c0d'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 21, 296291, tzinfo=datetime.timezone.utc)) +---------------- 11 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('03444e1b-fc1b-4282-9e3f-44695f46e7b5'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 17, 892262, tzinfo=datetime.timezone.utc)) +---------------- 12 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('bd4b3289-c301-4055-9ff4-5545816d7635'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 19, 672938, tzinfo=datetime.timezone.utc)) +---------------- 13 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('90434ad8-7222-436e-a7eb-b3a031667c19'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 413478, tzinfo=datetime.timezone.utc)) +---------------- 14 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('ec2ab9a6-4632-4af6-a41a-29c80b7b8c0d'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 21, 296291, tzinfo=datetime.timezone.utc)) +---------------- 15 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('face8cd4-7da8-4701-8190-9579e4ed441a'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 632025, tzinfo=datetime.timezone.utc)) +---------------- ... ---------------- | and 1 more exception +------------------------------------ | | During handling of the above exception, another exception occurred: | | Exception Group Traceback (most recent call last): | File "/usr/lib/python3.9/contextlib.py", line 653, in __aexit__ | cb_suppress = await cb(*exc_details) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Exception Group Traceback (most recent call last): | File "/home/username/Documents/.gits/tryraisebugaps/start.py", line 23, in main | await scheduler.run_until_stopped() | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 893, in run_until_stopped | await self.event_broker.publish_local(SchedulerStarted()) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/datastores/mongodb.py", line 222, in _get_session | yield session | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/datastores/mongodb.py", line 666, in acquire_jobs | update_task_result = await to_thread.run_sync( | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/to_thread.py", line 56, in run_sync | return await get_async_backend().run_sync_in_worker_thread( | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2128, in run_sync_in_worker_thread | await cls.checkpoint() | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2050, in checkpoint | await sleep(0) | File "/usr/lib/python3.9/asyncio/tasks.py", line 641, in sleep | await __sleep0() | File "/usr/lib/python3.9/asyncio/tasks.py", line 630, in __sleep0 | yield | asyncio.exceptions.CancelledError: Cancelled by cancel scope 7b4f4bf43c70 | | During handling of the above exception, another exception occurred: | | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1143, in _process_jobs | jobs = await self.data_store.acquire_jobs( | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/datastores/mongodb.py", line 551, in acquire_jobs | async for attempt in self._retry(): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/tenacity/asyncio/__init__.py", line 166, in __anext__ | do = await self.iter(retry_state=self._retry_state) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/tenacity/asyncio/__init__.py", line 153, in iter | result = await action(retry_state) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/tenacity/_utils.py", line 99, in inner | return call(*args, **kwargs) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/tenacity/__init__.py", line 398, in | self._add_action_func(lambda rs: rs.outcome.result()) | File "/usr/lib/python3.9/concurrent/futures/_base.py", line 439, in result | return self.__get_result() | File "/usr/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result | raise self._exception | File "/usr/lib/python3.9/contextlib.py", line 653, in __aexit__ | cb_suppress = await cb(*exc_details) | File "/usr/lib/python3.9/contextlib.py", line 199, in __aexit__ | await self.gen.athrow(typ, value, traceback) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/datastores/mongodb.py", line 224, in _get_session | await to_thread.run_sync(session.end_session) | asyncio.exceptions.CancelledError: Cancelled by cancel scope 7b4f4bf43c70 | | During handling of the above exception, another exception occurred: | | Exception Group Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1162, in _process_jobs | wakeup_event = anyio.Event() | File "/usr/lib/python3.9/contextlib.py", line 670, in __aexit__ | raise exc_details[1] | File "/usr/lib/python3.9/contextlib.py", line 653, in __aexit__ | cb_suppress = await cb(*exc_details) | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__ | raise BaseExceptionGroup( | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (16 sub-exceptions) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('03444e1b-fc1b-4282-9e3f-44695f46e7b5'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 17, 892262, tzinfo=datetime.timezone.utc)) +---------------- 2 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('face8cd4-7da8-4701-8190-9579e4ed441a'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 632025, tzinfo=datetime.timezone.utc)) +---------------- 3 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('90434ad8-7222-436e-a7eb-b3a031667c19'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 413478, tzinfo=datetime.timezone.utc)) +---------------- 4 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('d48bba75-5e31-4e4a-be75-3fa967f95dbb'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 18, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 19, 524304, tzinfo=datetime.timezone.utc)) +---------------- 5 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('600706a2-8a2d-49ac-8971-db7da6ede55a'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 18, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 781486, tzinfo=datetime.timezone.utc)) +---------------- 6 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('bd4b3289-c301-4055-9ff4-5545816d7635'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 19, 672938, tzinfo=datetime.timezone.utc)) +---------------- 7 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('b0a32ce2-d9d5-46c3-bcba-38146cf5f3a1'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 562387, tzinfo=datetime.timezone.utc)) +---------------- 8 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('b0a32ce2-d9d5-46c3-bcba-38146cf5f3a1'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 562387, tzinfo=datetime.timezone.utc)) +---------------- 9 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('600706a2-8a2d-49ac-8971-db7da6ede55a'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 18, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 781486, tzinfo=datetime.timezone.utc)) +---------------- 10 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('ec2ab9a6-4632-4af6-a41a-29c80b7b8c0d'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 21, 296291, tzinfo=datetime.timezone.utc)) +---------------- 11 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('03444e1b-fc1b-4282-9e3f-44695f46e7b5'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 17, 892262, tzinfo=datetime.timezone.utc)) +---------------- 12 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('bd4b3289-c301-4055-9ff4-5545816d7635'), task_id='__main__:tick', schedule_id='47c38433-f3f8-4b8f-903b-108d85d9b1df', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 66491, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 19, 672938, tzinfo=datetime.timezone.utc)) +---------------- 13 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('90434ad8-7222-436e-a7eb-b3a031667c19'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 19, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 20, 413478, tzinfo=datetime.timezone.utc)) +---------------- 14 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('ec2ab9a6-4632-4af6-a41a-29c80b7b8c0d'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 20, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 21, 296291, tzinfo=datetime.timezone.utc)) +---------------- 15 ---------------- | Traceback (most recent call last): | File "/home/username/.cache/pypoetry/virtualenvs/tryraisebugaps-YSkG1vqi-py3.9/lib/python3.9/site-packages/apscheduler/_schedulers/async_.py", line 1219, in _run_job | self._running_jobs.remove(job) | KeyError: Job(id=UUID('face8cd4-7da8-4701-8190-9579e4ed441a'), task_id='__main__:tick', schedule_id='2ec5193a-feae-4398-a14b-57197d0f6ba4', scheduled_fire_time=datetime.datetime(2024, 8, 10, 15, 36, 17, 850421, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=75600))), executor='async', metadata={}, created_at=datetime.datetime(2024, 8, 10, 18, 36, 18, 632025, tzinfo=datetime.timezone.utc)) +---------------- ... ---------------- | and 1 more exception +------------------------------------ ```

agronholm commented 3 months ago

Turns out that the bug indirectly reported in https://github.com/agronholm/apscheduler/issues/953 was causing the test failure not to be triggered.

lucasjinreal commented 2 months ago

Sorry for I posted a simillar issue but almost exact same issue I got with sqlachemdemy.

and I used time.sleep(2) in my function..

Is there any solution?

agronholm commented 2 months ago

Yes, running the master version instead of v4.0.0a5.

lucasjinreal commented 2 months ago

Hi, any time to upgrade to pypi?

agronholm commented 2 months ago

I will once I've resolved a bunch of outstanding issues with the code.

lucasjinreal commented 2 months ago

TypeError: Scheduler.add_schedule() got an unexpected keyword argument 'max_running_jobs'

Hi, just want to ask, thank u so much for the work, but can we just make the api a little bit stable?

agronholm commented 2 months ago

No, v4.0 is in alpha, meaning that the API is not stable. It will stabilize somewhat more when it hits beta, and completely so when it reaches a final release.

lucasjinreal commented 2 months ago

Hopefully we won't change all the codes again when beta and final. -.-

agronholm commented 2 months ago

You should be prepared for that. Also, the data store schemas may change while the project is in alpha. I will try to freeze the schema at least when entering the beta phase. This is an in-development version after all, not recommended for serious production use yet.

lucasjinreal commented 2 months ago

Posting numerous different versions of the API is not beneficial for your project. Currently, artificial intelligence is trending. Training AI with incorrect version corpora only leads to increased confusion among people regarding this library.

Nevertheless, scheduling is not an overly complicated matter. Do not overcomplicate things. Users simply desire a library that can handle scheduling. That is all.

agronholm commented 2 months ago

Maybe you should fork the project then, as you clearly know better than me how to design APIs and how all this should work. It's really simple, after all!

WillDaSilva commented 2 months ago

Posting numerous different versions of the API is not beneficial for your project. Currently, artificial intelligence is trending. Training AI with incorrect version corpora only leads to increased confusion among people regarding this library.

@lucasjinreal How do you suppose developers make changes to their projects without causing issues for AI? Should we all just freeze our code to suit what LLMs have been trained on? Of course not. If you really care to use an LLM to help with this, that's on you. Consider reading the docs for this project instead, as they're quite good.

Nevertheless, scheduling is not an overly complicated matter. Do not overcomplicate things. Users simply desire a library that can handle scheduling. That is all.

If it's so simple, why are you turning to this library at all? Why aren't you using the published and stable version of this library?

doluk commented 2 months ago

I can just agree to what @WillDaSilva said. It seems pretty respectless from you @lucasjinreal . Nobody forces you to use this. And the owner of a repo has the right to do whatever they want with the project😅

Falydoor commented 2 months ago

@lucasjinreal Welcome to the world of open-source, if this project doesn't suit your needs you can just fork it and maintain it yourself 😄. @agronholm is probably spending a lot of his free time working on this for free, you should be grateful that he is sharing his work with us.

Also if you really need to have the latest version on PyPi, you can just pull the master branch locally and link it with your requirements.txt using:

# APScheduler==4.0.0a5
APScheduler @ file:///LOCAL_PATH_TO_APSCHEDULER
agronholm commented 2 months ago

Better yet: apscheduler@git+https://github.com/agronholm/apscheduler.git

lucasjinreal commented 2 months ago

@doluk @WillDaSilva Please refrain from taking things personally. I did not imply that this is not a good library. In fact, I previously expressed my appreciation to @agronholm for his excellent work. I merely stated that posting numerous different API designs across multiple versions can cause confusion for users.

If you do not think this is the case, then you may disregard my opinion. Rather than posting non-constructive and emotionally venting comments.

I again, much apprecaited to @agronholm for his contribution for this lib.

I have decided to abandon the use of this library due to the disrespectful attitude of its owner and the rudeness of its followers. The scheduler is far superior.

agronholm commented 2 months ago

A good API is important for the developer experience, and so I have to change it during development as I run into snags where the previous API design doesn't work. I have pledged to uphold the Semantic Versioning 2.0 standard in my projects in order to not cause breakage between minor and patch updates, but even that standard allows for development releases to break the API arbitrarily.

This leads to my next question about your expectations: do you expect me to foresee every single design problem and design a perfect API on the first try, or do you expect me to keep a bad, non-working API just so some random user won't be upset by the breakage? I'm asking for an honest reply.

agronholm commented 2 months ago

I have decided to abandon the use of this library due to the disrespectful attitude of its owner and the rudeness of its followers. The scheduler is far superior.

As everyone else pointed out, you're the one with the disrespectful attitude. And what scheduler are you talking about?

EDIT: probably this one.

Henkhogan commented 3 weeks ago

@agronholm seems there is not much missing to complete this issue. Is there anything we can help with?

agronholm commented 2 weeks ago

@agronholm seems there is not much missing to complete this issue. Is there anything we can help with?

All the commentary has muddled the original issue; I can't even be sure if it's actually been fixed or not. So you can help by refreshing my memory :)

HK-Mattew commented 3 days ago

I just checked and the error no longer occurs for me on the master branch. It used to occur when running the sample code: https://github.com/agronholm/apscheduler/issues/952#issuecomment-2282244536

However, I just ran this same code on the master branch and the problem did not occur. So I believe this has been fixed.