Open kansoadi opened 9 months ago
i solved the problem related to the
"get_full_data(sync, meili_settings.insert_size)"
but i got a new one
TypeError: 'async for' received an object from aiter that does not implement anext: coroutine
meilisync start
C:\Python312\Lib\site-packages\meilisync\main.py:135: RuntimeWarning: coroutine 'Source.aiter' was never awaited
async for event in source:
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
┌─────────────────────────────── Traceback (most recent call last) ────────────────────────────────┐
│ C:\Python312\Lib\site-packages\meilisync\main.py:174 in start │
│ │
│ 171 │ │ lock = asyncio.Lock() │
│ 172 │ │ await asyncio.gather((), interval()) │
│ 173 │ │
│ > 174 │ asyncio.run(run()) │
│ 175 │
│ 176 │
│ 177 @app.command(help="Refresh all data by swap index") │
│ │
│ ┌─────────────────────────────────────────── locals ───────────────────────────────────────────┐ │
│ │ = <function start.
I wanted to provide a quick update regarding the recent issues we encountered. In light of the numerous errors and their perplexing nature, I decided to leverage Docker as a solution.
The transition to Docker has proven to be highly effective, and the application is now running smoothly without any hiccups. It appears that the challenges we faced were rooted in the operating environment, and Docker has provided a robust and reliable workaround.
I’m using Meilisync using docker compose
and I have the same issue:
meilisync-1 | TypeError: 'async for' requires an object with __aiter__ method, got coroutine
meilisync-1 exited with code 0
Edit: looks like an issue with progress.path
because when I commented-out the line the problem went away.
@bfontaine, what do you mean by "Edit: looks like an issue with progress.path because when I commented-out the line the problem went away" unfortunately, when i deploy the app on AWS i receive the same error, i try 2 methods: 1- Meilisearch and Meilisync were installed in 2 docker container 2- an EC2 was created by Meilisearch image, Meilisync installed by docker
i am stuck in the 2 methods by the error "TypeError: 'async for' requires an object with aiter method, got coroutine"
After deploying the app using two methods—utilizing Docker containers with Meilisearch and Meilisync installed separately, and creating an EC2 instance with Meilisearch image and Meilisync installed via Docker—we've hit the same roadblock. The error message we're grappling with is: "TypeError: 'async for' requires an object with aiter method, got coroutine."
@kansoadi I had a custom path
in the config, and when I removed it the issue went away.
With this config I have the error:
progress:
type: file
path: myprogress.json
# ...
With this one I don’t:
progress:
type: file
# ...
@bfontaine, but path is required if the type is set to file (type: file or redis, if set to file, another option path is required.)
_meilisync_1 | ValidationError: 1 validation error for Settings
meilisync_1 | progress
meilisync_1 | Field required [type=missing, input_value={'debug': True,
meilisync_1 | 'plugins'...ronment': 'production'}}, input_type=dict]
meilisync_1 | For further information visit https://errors.pydantic.dev/2.5/v/missing_
Please try latest source code
Sure, The latest source code was used; ' docker pull long2ice/meilisync Using default tag: latest latest: Pulling from long2ice/meilisync Digest: sha256:0916df358ccdcfeebad4d8c946c46641d2bf72a220907cde220d963638759f69 Status: Image is up to date for long2ice/meilisync:latest docker.io/long2ice/meilisync:latest "
I can't re reproduce locally.
For refresh:
For start:
I do have the same problem with the latest image, but was able to launch meilisync with older version
i got something familiar to this error :( exception=TypeError("'async for' requires an object with aiter method, got list")> i just pull the image (latest) plz HELP :)
I have the same issue when i run "meilisync refresh" Plz Help
i saw another error when i was running "meilisearch check" instead of using ret[0], i replace it with ret["count"] and it worked.
Hi, @kansoadi Have you resolved the issue? I have the same error "TypeError: 'async for' requires an object with aiter method, got coroutine" Config below
debug: true
plugins:
- meilisync.plugin.Plugin
progress:
type: file
source:
type: postgres
host: docker.for.mac.host.internal
port: 5433
database: test
user: postgres
password: <password>
meilisearch:
api_url: http://docker.for.mac.host.internal:7700
api_key: <admin_key>
insert_size: 1000
insert_interval: 10
sync:
- table: games
index: games_index
full: true
fields:
id:
Hello,
I wanted to bring to your attention some issues I have been experiencing with Meilisync. Unfortunately, I have noticed many errors while trying to integrate it into my app, which has made the process quite difficult.
It seems that Meilisync may require more thorough testing from its developer, especially when it comes to working with Postgresql. I have noticed that the replication increases in size and ultimately causes the app to crash. Additionally, it does not sync properly with the dedicated table(s), leading to discrepancies in the data.
I believe that addressing these issues and conducting further tests on Meilisync, particularly with Postgresql integration, will greatly improve its functionality and usability. I appreciate your attention to this matter and look forward to a resolution.
Thank you.
On Thu, May 23, 2024 at 1:01 PM Yurii Poliakov @.***> wrote:
Hi, @kansoadi https://github.com/kansoadi Have you resolved the issue? I have the same error "TypeError: 'async for' requires an object with aiter method, got coroutine" Config below
debug: trueplugins:
- meilisync.plugin.Pluginprogress: type: filesource: type: postgres host: docker.for.mac.host.internal port: 5433 database: test user: postgres password:
meilisearch: api_url: http://docker.for.mac.host.internal:7700 api_key: insert_size: 1000 insert_interval: 10sync: - table: games index: games_index full: true fields: id:
— Reply to this email directly, view it on GitHub https://github.com/long2ice/meilisync/issues/76#issuecomment-2126711870, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARTTFFXOTPTZ5SHBGHHV5SDZDW47BAVCNFSM6AAAAABCPTYT32VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRWG4YTCOBXGA . You are receiving this because you were mentioned.Message ID: @.***>
Hi, I am also got a similar issue. I got an error when running it for the first time with a Docker container. Then I tried running it on macOS, and it worked perfectly fine. I retried with Docker, and it worked normally. It seems this error only occurs when running it for the first time on Docker container.
hi, I got the some err.
that is my config file
when I change the param 'full: false' , that will be ok, but I found I can not sync any data in meilisearch
bump
For anyone else having this problem with the official docker image - the official image (as of about 1 week prior to this message) was using version 0.1.2 - whereas this bug was seemingly fixed in 0.1.3.
I forked the repository, built the image myself and that seems to have fixed this issue.
Hi, I still have the same error :
TypeError: 'async for' requires an object with __aiter__ method, got coroutine
With latest image. There is my config.yml :
debug: true
plugins:
- meilisync.plugin.Plugin
progress:
type: file
meilisearch:
api_url: ${MEILISEARCH_API_URL}
api_key: ${MEILISEARCH_API_KEY}
insert_size: 1000
insert_interval: 10
source:
type: postgres
host: ${POSTGRES_HOST}
port: ${POSTGRES_PORT}
database: ${POSTGRES_DB}
user: postgres
password: ${POSTGRES_PASSWORD}
sync:
- table: movie
index: movies
full: true
Btw environment variable dont work with docker-compose
Not sure if it helps anyone, but for me the error went away when I created an empty index in meilisearch for the table I was trying to sync.
Not sure if it helps anyone, but for me the error went away when I created an empty index in meilisearch for the table I was trying to sync.
Thx for ur reply, in fact creating the index first fix the issue. But now I have another issue with syncing and Im not sure meilisync can sync children of a table (with relation one-to-many, one-to-one,...)
i got the same error, when i start docker-compose.
2024-09-06 06:32:30.677 | DEBUG | meilisync.main:_:36 - plugins=['meilisync.plugin.Plugin']
progress=Progress(type=<ProgressType.file: 'file'>) debug=True source=Source
(type=<SourceType.postgres: 'postgres'>,
database='postgres', host='192.168.101.194', port=5432, user='postgres', password='postgres')
meilisearch=MeiliSearch(api_url='http://192.168.101.194:7700/', api_key='API_KEY', insert_size=1000,
insert_interval=10) sync=[Sync(plugins=[], table='document_info', pk='id', full=True,
index='document_info',
fields=None), Sync(plugins=[], table='document_content', pk='doc_id', full=True, index='document_content',
fields=None)] sentry=None
/meilisync/meilisync/main.py:89: RuntimeWarning: coroutine 'Postgres.get_full_data' was never awaited
async for items in source.get_full_data(sync, meili_settings.insert_size or 10000):
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /meilisync/meilisync/main.py:140 in start │
│ │
│ 137 │ │ lock = asyncio.Lock() │
│ 138 │ │ await asyncio.gather(_(), interval()) │
│ 139 │ │
│ ❱ 140 │ asyncio.run(run()) │
│ 141 │
│ 142 │
│ 143 @app.command(help="Refresh all data by swap index") │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ _ = <function start.<locals>._ at 0x7fc3165d28e0> │ │
│ │ collection = <meilisync.event.EventCollection object at │ │
│ │ 0x7fc3170b7050> │ │
│ │ context = <click.core.Context object at 0x7fc3174ebda0> │ │
│ │ current_progress = None │ │
│ │ interval = <function start.<locals>.interval at 0x7fc3165d2340> │ │
│ │ lock = <asyncio.locks.Lock object at 0x7fc3165b2420 │ │
│ │ [unlocked]> │ │
│ │ meili = <meilisync.meili.Meili object at 0x7fc3170b6fc0> │ │
│ │ meili_settings = MeiliSearch( │ │
│ │ │ api_url='http://192.168.101.194:7700/', │ │
│ │ │ │ │
│ │ api_key='4jy3YVpfJJW93W8g…' │ │
│ │ │ insert_size=1000, │ │
│ │ │ insert_interval=10 │ │
│ │ ) │ │
│ │ progress = <meilisync.progress.file.File object at │ │
│ │ 0x7fc317009430> │ │
│ │ run = <function start.<locals>.run at 0x7fc3165d2a20> │ │
│ │ settings = Settings( │ │
│ │ │ plugins=['meilisync.plugin.Plugin'], │ │
│ │ │ progress=Progress( │ │
│ │ │ │ type=<ProgressType.file: 'file'> │ │
│ │ │ ), │ │
│ │ │ debug=True, │ │
│ │ │ source=Source( │ │
│ │ │ │ type=<SourceType.postgres: 'postgres'>, │ │
│ │ │ │ database='postgres', │ │
│ │ │ │ host='192.168.101.194', │ │
│ │ │ │ port=5432, │ │
│ │ │ │ user='postgres', │ │
│ │ │ │ password='postgres' │ │
│ │ │ ), │ │
│ │ │ meilisearch=MeiliSearch( │ │
│ │ │ │ api_url='http://192.168.101.194:7700/', │ │
│ │ │ │ │ │
│ │ api_key='API_KEY' │ │
│ │ │ │ insert_size=1000, │ │
│ │ │ │ insert_interval=10 │ │
│ │ │ ), │ │
│ │ │ sync=[ │ │
│ │ │ │ Sync( │ │
│ │ │ │ │ plugins=[], │ │
│ │ │ │ │ table='document_info', │ │
│ │ │ │ │ pk='id', │ │
│ │ │ │ │ full=True, │ │
│ │ │ │ │ index='document_info', │ │
│ │ │ │ │ fields=None │ │
│ │ │ │ ), │ │
│ │ │ │ Sync( │ │
│ │ │ │ │ plugins=[], │ │
│ │ │ │ │ table='document_content', │ │
│ │ │ │ │ pk='doc_id', │ │
│ │ │ │ │ full=True, │ │
│ │ │ │ │ index='document_content', │ │
│ │ │ │ │ fields=None │ │
│ │ │ │ ) │ │
│ │ │ ], │ │
│ │ │ sentry=None │ │
│ │ ) │ │
│ │ source = <meilisync.source.postgres.Postgres object at │ │
│ │ 0x7fc3170b4350> │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /usr/local/lib/python3.12/asyncio/runners.py:194 in run │
│ │
│ 191 │ │ │ "asyncio.run() cannot be called from a running event loop" │
│ 192 │ │
│ 193 │ with Runner(debug=debug, loop_factory=loop_factory) as runner: │
│ ❱ 194 │ │ return runner.run(main) │
│ 195 │
│ 196 │
│ 197 def _cancel_all_tasks(loop): │
│ │
│ ╭──────────────────────────────── locals ────────────────────────────────╮ │
│ │ debug = None │ │
│ │ loop_factory = None │ │
│ │ main = <coroutine object start.<locals>.run at 0x7fc316567140> │ │
│ │ runner = <asyncio.runners.Runner object at 0x7fc3170b4590> │ │
│ ╰────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /usr/local/lib/python3.12/asyncio/runners.py:118 in run │
│ │
│ 115 │ │ │
│ 116 │ │ self._interrupt_count = 0 │
│ 117 │ │ try: │
│ ❱ 118 │ │ │ return self._loop.run_until_complete(task) │
│ 119 │ │ except exceptions.CancelledError: │
│ 120 │ │ │ if self._interrupt_count > 0: │
│ 121 │ │ │ │ uncancel = getattr(task, "uncancel", None) │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ context = <_contextvars.Context object at 0x7fc318332d40> │ │
│ │ coro = <coroutine object start.<locals>.run at 0x7fc316567140> │ │
│ │ self = <asyncio.runners.Runner object at 0x7fc3170b4590> │ │
│ │ sigint_handler = functools.partial(<bound method Runner._on_sigint of │ │
│ │ <asyncio.runners.Runner object at 0x7fc3170b4590>>, │ │
│ │ main_task=<Task finished name='Task-4' │ │
│ │ coro=<start.<locals>.run() done, defined at │ │
│ │ /meilisync/meilisync/main.py:135> │ │
│ │ exception=TypeError("'async for' requires an object │ │
│ │ with __aiter__ method, got coroutine")>) │ │
│ │ task = <Task finished name='Task-4' coro=<start.<locals>.run() │ │
│ │ done, defined at /meilisync/meilisync/main.py:135> │ │
│ │ exception=TypeError("'async for' requires an object │ │
│ │ with __aiter__ method, got coroutine")> │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /usr/local/lib/python3.12/asyncio/base_events.py:664 in run_until_complete │
│ │
│ 661 │ │ if not future.done(): │
│ 662 │ │ │ raise RuntimeError('Event loop stopped before Future comp │
│ 663 │ │ │
│ ❱ 664 │ │ return future.result() │
│ 665 │ │
│ 666 │ def stop(self): │
│ 667 │ │ """Stop running the event loop. │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ future = <Task finished name='Task-4' coro=<start.<locals>.run() done, │ │
│ │ defined at /meilisync/meilisync/main.py:135> │ │
│ │ exception=TypeError("'async for' requires an object with │ │
│ │ __aiter__ method, got coroutine")> │ │
│ │ new_task = False │ │
│ │ self = <_UnixSelectorEventLoop running=False closed=True │ │
│ │ debug=False> │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /meilisync/meilisync/main.py:138 in run │
│ │
│ 135 │ async def run(): │
│ 136 │ │ nonlocal lock │
│ 137 │ │ lock = asyncio.Lock() │
│ ❱ 138 │ │ await asyncio.gather(_(), interval()) │
│ 139 │ │
│ 140 │ asyncio.run(run()) │
│ 141 │
│ │
│ ╭────────────────────────────── locals ───────────────────────────────╮ │
│ │ _ = <function start.<locals>._ at 0x7fc3165d28e0> │ │
│ │ interval = <function start.<locals>.interval at 0x7fc3165d2340> │ │
│ │ lock = <asyncio.locks.Lock object at 0x7fc3165b2420 [unlocked]> │ │
│ ╰─────────────────────────────────────────────────────────────────────╯ │
│ │
│ /meilisync/meilisync/main.py:89 in _ │
│ │
│ 86 │ │ for sync in settings.sync: │
│ 87 │ │ │ if sync.full and not await meili.index_exists(sync.index_n │
│ 88 │ │ │ │ count = 0 │
│ ❱ 89 │ │ │ │ async for items in source.get_full_data(sync, meili_se │
│ 90 │ │ │ │ │ count += len(items) │
│ 91 │ │ │ │ │ await meili.add_full_data(sync, items) │
│ 92 │ │ │ │ if count: │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ collection = <meilisync.event.EventCollection object at │ │
│ │ 0x7fc3170b7050> │ │
│ │ count = 0 │ │
│ │ current_progress = None │ │
│ │ lock = <asyncio.locks.Lock object at 0x7fc3165b2420 │ │
│ │ [unlocked]> │ │
│ │ meili = <meilisync.meili.Meili object at 0x7fc3170b6fc0> │ │
│ │ meili_settings = MeiliSearch( │ │
│ │ │ api_url='http://192.168.101.194:7700/', │ │
│ │ │ │ │
│ │ api_key='API_KEY' │ │
│ │ │ insert_size=1000, │ │
│ │ │ insert_interval=10 │ │
│ │ ) │ │
│ │ progress = <meilisync.progress.file.File object at │ │
│ │ 0x7fc317009430> │ │
│ │ settings = Settings( │ │
│ │ │ plugins=['meilisync.plugin.Plugin'], │ │
│ │ │ progress=Progress( │ │
│ │ │ │ type=<ProgressType.file: 'file'> │ │
│ │ │ ), │ │
│ │ │ debug=True, │ │
│ │ │ source=Source( │ │
│ │ │ │ type=<SourceType.postgres: 'postgres'>, │ │
│ │ │ │ database='postgres', │ │
│ │ │ │ host='192.168.101.194', │ │
│ │ │ │ port=5432, │ │
│ │ │ │ user='postgres', │ │
│ │ │ │ password='postgres' │ │
│ │ │ ), │ │
│ │ │ meilisearch=MeiliSearch( │ │
│ │ │ │ api_url='http://192.168.101.194:7700/', │ │
│ │ │ │ │ │
│ │ api_key='4jy3YVpfJJW93WXWjZV6J0KbuLiWyVRCZG_Xl6t8l8g… │ │
│ │ │ │ insert_size=1000, │ │
│ │ │ │ insert_interval=10 │ │
│ │ │ ), │ │
│ │ │ sync=[ │ │
│ │ │ │ Sync( │ │
│ │ │ │ │ plugins=[], │ │
│ │ │ │ │ table='document_info', │ │
│ │ │ │ │ pk='id', │ │
│ │ │ │ │ full=True, │ │
│ │ │ │ │ index='document_info', │ │
│ │ │ │ │ fields=None │ │
│ │ │ │ ), │ │
│ │ │ │ Sync( │ │
│ │ │ │ │ plugins=[], │ │
│ │ │ │ │ table='document_content', │ │
│ │ │ │ │ pk='doc_id', │ │
│ │ │ │ │ full=True, │ │
│ │ │ │ │ index='document_content', │ │
│ │ │ │ │ fields=None │ │
│ │ │ │ ) │ │
│ │ │ ], │ │
│ │ │ sentry=None │ │
│ │ ) │ │
│ │ source = <meilisync.source.postgres.Postgres object at │ │
│ │ 0x7fc3170b4350> │ │
│ │ sync = Sync( │ │
│ │ │ plugins=[], │ │
│ │ │ table='document_info', │ │
│ │ │ pk='id', │ │
│ │ │ full=True, │ │
│ │ │ index='document_info', │ │
│ │ │ fields=None │ │
│ │ ) │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
TypeError: 'async for' requires an object with __aiter__ method, got coroutine
and this is my config.yml
debug: true
plugins:
- meilisync.plugin.Plugin
progress:
type: file
source:
type: postgres
host: 192.168.101.194
port: 5432
database: postgres
user: postgres
password: "postgres"
meilisearch:
api_url: http://192.168.101.194:7700/
api_key: 'api_key'
insert_size: 1000
insert_interval: 10
sync:
- table: document_info
index: document_info
pk: id
full: true
- table: document_content
index: document_content
pk: doc_id
full: true
my docker-compose.yml
services:
meilisync:
image: long2ice/meilisync:latest
volumes:
- ./meilisync/meilisync_config.yml:/meilisync/config.yml
networks:
- pub-network
restart: always
postgres_db:
image: postgres:16.4
container_name: docker_postgres
environment:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
ports:
- "5432:5432"
volumes:
- ./postgres/data:/var/lib/postgresql/data
- ./postgres/log:/var/log/postgresql
logging:
options:
max-size: "10m"
max-file: "3"
networks:
- pub-network
meilisearch_server:
image: getmeili/meilisearch:v1.10.0
container_name: meilisearch_server
ports:
- "7700:7700"
environment:
MEILI_MASTER_KEY: 'api_key'
volumes:
- ./meili_data:/meili_data
networks:
- pub-network
networks:
pub-network:
driver: bridge
For syncing databases with PostgreSQL and Meilisync,
I installed and configured Meilisync (windows 10) by following the steps in https://www.meilisearch.com/docs/learn/cookbooks/meilisync_postgresql Meilisync was well-configured: 1- 2- but for the Meilisync start, i received TypeError: 'async for' requires an object with aiter method, got generator returned from C:\Python312\Lib\site-packages\meilisync\main.py:96 in │ │ │ │ 93 │ │ for sync in settings.sync: │ │ 94 │ │ │ if sync.full and not await meili.index_exists(sync.index_name): │ │ 95 │ │ │ │ count = 0 │ │ > 96 │ │ │ │ async for items in await source.get_full_data(sync, meili_settings.inser │ │ 97 │ │ │ │ │ count += len(items) │ │ 98 │ │ │ │ │ await meili.adddata(sync, items) │ │ 99 │ │ │ │ if count:
the config.yml:
kindly, Any idea on how to solve this issue?