Open zamai opened 11 months ago
I already had the flags enabled from this article, but data wasn't replicated.
Here is a docker compose I'm using to test:
version: '3'
services:
meilisync:
platform: linux/x86_64
image: long2ice/meilisync
volumes:
- ./meilisync-config.yml:/meilisync/config.yml
restart: always
depends_on:
- meilisearch
meilisearch:
image: getmeili/meilisearch:v1.3
ports:
- "7700:7700"
environment:
- MEILI_ENV=development
- MEILI_API_KEY=some-key
Sync config:
debug: true
progress:
type: file
meilisearch:
api_url: "http://meilisearch:7700"
api_key: "API_KEY"
source:
type: postgres
host: DB_IP
port: 5432
database: inro
user: replication_user
password: "PASSWORD"
sync:
- table: PG_SCHEMA_NAME.table_name
index: table_name
full: true
Logs from containers:
infrastructure-meilisearch-1 | [2023-09-19T09:12:16Z INFO actix_server::builder] starting 6 workers
infrastructure-meilisearch-1 | [2023-09-19T09:12:16Z INFO actix_server::server] Actix runtime found; starting in Actix runtime
infrastructure-meilisync-1 | 2023-09-19 09:12:26.540 | DEBUG | meilisync.main:_:33 - plugins=None progress=Progress(type=<ProgressType.file: 'file'>) debug=True source=Source(type=<SourceType.postgres: 'postgres'>, database='inro', password='', port=5432, host='', user='replication_user') meilisearch=MeiliSearch(api_url='http://meilisearch:7700', api_key='iwBx-', insert_size=None, insert_interval=None) sync=[Sync(plugins=None, table='items', pk='id', full=True, index='', fields=None), Sync(plugins=None, table='manifest', pk='id', full=True, index='manifest', fields=None)] sentry=None
infrastructure-meilisearch-1 | [2023-09-19T09:12:27Z INFO actix_web::middleware::logger] 172.18.0.3 "POST /indexes/l/documents?primaryKey=id HTTP/1.1" 202 142 "-" "python-httpx/0.23.3" 0.007829
infrastructure-meilisearch-1 | [2023-09-19T09:12:27Z INFO actix_web::middleware::logger] 172.18.0.3 "POST /indexes/l/documents?primaryKey=id HTTP/1.1" 202 142 "-" "python-httpx/0.23.3" 0.008936
infrastructure-meilisearch-1 | [2023-09-19T09:12:27Z INFO index_scheduler] A batch of tasks was successfully completed.
infrastructure-meilisearch-1 | [2023-09-19T09:12:27Z INFO index_scheduler] A batch of tasks was successfully completed.
infrastructure-meilisync-1 | 2023-09-19 09:12:27.746 | INFO | meilisync.main:_:82 - Full data sync for table "items" done! 1378 documents added.
infrastructure-meilisearch-1 | [2023-09-19T09:12:27Z INFO actix_web::middleware::logger] 172.18.0.3 "POST /indexes//documents?primaryKey=id HTTP/1.1" 202 143 "-" "python-httpx/0.23.3" 0.004431
infrastructure-meilisearch-1 | [2023-09-19T09:12:27Z INFO index_scheduler] A batch of tasks was successfully completed.
infrastructure-meilisync-1 | 2023-09-19 09:12:27.804 | INFO | meilisync.main:_:82 - Full data sync for table "manifest" done! 409 documents added.
I'm confused by the last log, it states that X documents was added, but indexes are empty:
Hi! I'm trying to setup sync with Postgres hosted by Google Cloud, currently my sync only copies names of the tables, but the contents of index is empty. I see that meilisync requires
wal2json
extension for Postgres sync, but I don't see it as a supported extension for google cloud SQL: https://cloud.google.com/sql/docs/postgres/extensions#miscellaneous-extensionsI'm wondering is there an alternative solution, maybe some other extension will help me sync? or a setting in meilisync to use other form of decoding ...
Thanks!