ERROR argument after ** must be a mapping, not NoneType

[Rich5]

I'm attempting to sync from postgres and I'm consistently getting this error below when running the latest code from this repo. Although I think the sync is still somewhat working because I can see the record count go up in the index and it appears to be matching the offset count of the current running query in postgres.

Looking at where this error is coming from does this mean that some some inserts into Meilisearch are working, but some batches are failing? It's hard to determine if records are missing while the sync is in progress and this database is over 20M records.

The full trace is below.

2024-01-20 20:18:43.045 | ERROR    | meilisync.main:interval:132 - meilisync.progress.file.File.set() argument after ** must be a mapping, not NoneType
Traceback (most recent call last):

  File "/usr/local/bin/meilisync", line 6, in <module>
    sys.exit(app())
    │   │    └ <typer.main.Typer object at 0x7f9815c273e0>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>
  File "/usr/local/lib/python3.12/site-packages/typer/main.py", line 311, in __call__
    return get_command(self)(*args, **kwargs)
           │           │      │       └ {}
           │           │      └ ()
           │           └ <typer.main.Typer object at 0x7f9815c273e0>
           └ <function get_command at 0x7f9817b5e340>
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           │    │     │       └ {}
           │    │     └ ()
           │    └ <function TyperGroup.main at 0x7f9817b5cc20>
           └ <TyperGroup callback>
  File "/usr/local/lib/python3.12/site-packages/typer/core.py", line 778, in main
    return _main(
           └ <function _main at 0x7f9817b53b00>
  File "/usr/local/lib/python3.12/site-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
         │    │      └ <click.core.Context object at 0x7f9815184d40>
         │    └ <function MultiCommand.invoke at 0x7f9817f84720>
         └ <TyperGroup callback>
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
           │               │       │       │      └ <click.core.Context object at 0x7f981516e7e0>
           │               │       │       └ <function Command.invoke at 0x7f9817f840e0>
           │               │       └ <TyperCommand start>
           │               └ <click.core.Context object at 0x7f981516e7e0>
           └ <function MultiCommand.invoke.<locals>._process_result at 0x7f981455bce0>
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           │   │      │    │           │   └ {}
           │   │      │    │           └ <click.core.Context object at 0x7f981516e7e0>
           │   │      │    └ <function start at 0x7f9814559bc0>
           │   │      └ <TyperCommand start>
           │   └ <function Context.invoke at 0x7f9817f62a20>
           └ <click.core.Context object at 0x7f981516e7e0>
  File "/usr/local/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
                       │       └ {}
                       └ ()
  File "/usr/local/lib/python3.12/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
           │          └ {'context': <click.core.Context object at 0x7f981516e7e0>}
           └ <function start at 0x7f98145599e0>

  File "/meilisync/meilisync/main.py", line 140, in start
    asyncio.run(run())
    │       │   └ <function start.<locals>.run at 0x7f98145354e0>
    │       └ <function run at 0x7f9818b0de40>
    └ <module 'asyncio' from '/usr/local/lib/python3.12/asyncio/__init__.py'>

  File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           │      │   └ <coroutine object start.<locals>.run at 0x7f981452aa40>
           │      └ <function Runner.run at 0x7f98180bfc40>
           └ <asyncio.runners.Runner object at 0x7f9815df0440>
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           │    │     │                  └ <Task pending name='Task-4' coro=<start.<locals>.run() running at /meilisync/meilisync/main.py:138> wait_for=<_GatheringFutur...
           │    │     └ <function BaseEventLoop.run_until_complete at 0x7f98180bd8a0>
           │    └ <_UnixSelectorEventLoop running=True closed=False debug=False>
           └ <asyncio.runners.Runner object at 0x7f9815df0440>
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 671, in run_until_complete
    self.run_forever()
    │    └ <function BaseEventLoop.run_forever at 0x7f98180bd800>
    └ <_UnixSelectorEventLoop running=True closed=False debug=False>
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 638, in run_forever
    self._run_once()
    │    └ <function BaseEventLoop._run_once at 0x7f98180bf600>
    └ <_UnixSelectorEventLoop running=True closed=False debug=False>
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1971, in _run_once
    handle._run()
    │      └ <function Handle._run at 0x7f9818219a80>
    └ <Handle Task.task_wakeup(<Future finished result=None>)>
  File "/usr/local/lib/python3.12/asyncio/events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
    │    │            │    │           │    └ <member '_args' of 'Handle' objects>
    │    │            │    │           └ <Handle Task.task_wakeup(<Future finished result=None>)>
    │    │            │    └ <member '_callback' of 'Handle' objects>
    │    │            └ <Handle Task.task_wakeup(<Future finished result=None>)>
    │    └ <member '_context' of 'Handle' objects>
    └ <Handle Task.task_wakeup(<Future finished result=None>)>

> File "/meilisync/meilisync/main.py", line 130, in interval
    await progress.set(**current_progress)
          │        │     └ None
          │        └ <function File.set at 0x7f98151249a0>
          └ <meilisync.progress.file.File object at 0x7f981490ccb0>

TypeError: meilisync.progress.file.File.set() argument after ** must be a mapping, not NoneType
2024-01-20 20:18:43.053 | ERROR    | meilisync.main:interval:133 - Error when insert data to MeiliSearch: meilisync.progress.file.File.set() argument after ** must be a mapping, not NoneTyp

[Rich5]

Looks similar to https://github.com/long2ice/meilisync/issues/17

[Rich5]

My work around (fix?) to this so far is to change line https://github.com/long2ice/meilisync/blob/dev/meilisync/main.py#L75

from:

current_progress = context.obj["current_progress"]

which appears to initialize current_progress to None, to:

current_progress = {}

My best guess is that there is a race condition where the current_progress is not initialized at https://github.com/long2ice/meilisync/blob/dev/meilisync/main.py#L105 before we try to set it https://github.com/long2ice/meilisync/blob/dev/meilisync/main.py#L130

I'm not sure if this break anything other features, but so far it's working for me.