AsyncIOScheduler not working with MongoDBJobStore (PicklingError)

difhel commented 7 months ago

Things to check first

[X] I have checked that my issue does not already have a solution in the FAQ
[X] I have searched the existing issues and didn't find my bug already reported there
[X] I have checked that my bug is still present in the latest release

Version

3.10.4

What happened?

I use asyncio scheduler because my project uses an asyncio based framework (aiogram). When I receive a message, I create a task that will execute after a certain number of seconds:

@router.message(Command("schedule"))
async def aps(message: Message):
    seconds_delay = int(message.text.split()[-1])
    job = SchedulerWrapper().add_job(
        my_function,
        "date",
        run_date=datetime.datetime.now() + datetime.timedelta(seconds=seconds_delay),
        args=[message]
    )
    print(job)
    await message.answer("Scheduled")

Here's my code with the scheduler (I'm using Singleton wrapper to make sure only one scheduler object is used):

import apscheduler.schedulers.asyncio
from apscheduler.jobstores.mongodb import MongoDBJobStore
from apscheduler.jobstores.memory import MemoryJobStore
from shared.db import client

jobstores = {
    "default": MongoDBJobStore("giveaways-bot", client=client)
}
# jobstores = {
#     "default": MemoryJobStore()
# }

class SchedulerWrapper:
    _instance = None

    def __new__(cls):
        if cls._instance is None:
            cls._instance = super().__new__(cls)
            cls._instance.scheduler = apscheduler.schedulers.asyncio.AsyncIOScheduler(
                jobstores={'default': MongoDBJobStore(client=client)},
                job_defaults={'misfire_grace_time': 1},
                timezone='UTC',
                serializer="json"
            )
            cls._instance.scheduler.start()
        return cls._instance

    def add_job(self, *args, **kwargs):
        return self.scheduler.add_job(*args, **kwargs)

    def shutdown(self):
        self.scheduler.shutdown()

async def my_function(message) -> None:
    await message.answer("pong")

If you uncomment and use memory jobstore instead of MongoDB, the code works perfectly. This error occurs with MongoDB:

Traceback (most recent call last):
  <erased aiogram trace>
  File "/mnt/c/Users/Mark/Desktop/Projects/kwork/giveaways/bot/routers/admin/router.py", line 110, in admin_set_winners
    job = SchedulerWrapper().add_job(
  File "/mnt/c/Users/Mark/Desktop/Projects/kwork/giveaways/bot/scheduler.py", line 53, in add_job
    return self.scheduler.add_job(*args, **kwargs)
  File "/mnt/c/Users/Mark/Desktop/Projects/kwork/giveaways/.venv/lib/python3.10/site-packages/apscheduler/schedulers/base.py", line 457, in add_job
    self._real_add_job(job, jobstore, replace_existing)
  File "/mnt/c/Users/Mark/Desktop/Projects/kwork/giveaways/.venv/lib/python3.10/site-packages/apscheduler/schedulers/base.py", line 881, in _real_add_job
    store.add_job(job)
  File "/mnt/c/Users/Mark/Desktop/Projects/kwork/giveaways/.venv/lib/python3.10/site-packages/apscheduler/jobstores/mongodb.py", line 89, in add_job
    'job_state': Binary(pickle.dumps(job.__getstate__(), self.pickle_protocol))
_pickle.PicklingError: Can't pickle <function AsyncIOScheduler.wakeup at 0x7f943470f490>: it's not the same object as apscheduler.schedulers.asyncio.AsyncIOScheduler.wakeup

How can we reproduce the bug?

See the code above

agronholm commented 7 months ago

I tried running your script but it complains about the shared.db import.

difhel commented 7 months ago

I have prepared an MRE for this problem. Don't forget to insert your token in bot/main.py:12.

Link: https://github.com/difhel/aps-mre-2.

Note bot/routers/admin/router.py:17-19. This is an empty function that never triggers and does nothing. However, if you comment out this function, the error changes to another (see #872). This is very strange, I have no idea why this happens or how to fix it. Obviously it's a bug in either aiogram, aps, or pickle, but in any of the three cases I have absolutely no idea how to solve this further. It was already very hard to find, out of 5000 lines of code, exactly which empty function breaks the code when deleted.

By the way, thank you for your great work on this lib.

agronholm commented 4 months ago

So...how do I install and run this?

agronholm commented 2 months ago

I'm closing this due to the lack of responses. You need to provide me a MINIMUM workable example – that is, not something that requires a Discord token or something else. If this cannot be reproduced without the Discord Python library, then maybe that's the issue? Either way, it's your responsibility to figure out if APScheduler is really the problem, not mine. I'll reopen the issue if you work that out.

agronholm / apscheduler