Does sync work with Google Cloud SQL Postgres ?

[zamai]

Hi! I'm trying to setup sync with Postgres hosted by Google Cloud, currently my sync only copies names of the tables, but the contents of index is empty. I see that meilisync requires wal2json extension for Postgres sync, but I don't see it as a supported extension for google cloud SQL: https://cloud.google.com/sql/docs/postgres/extensions#miscellaneous-extensions

I'm wondering is there an alternative solution, maybe some other extension will help me sync? or a setting in meilisync to use other form of decoding ...

Thanks!

[long2ice]

Maybe you can check https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication

[zamai]

I already had the flags enabled from this article, but data wasn't replicated.

Here is a docker compose I'm using to test:

version: '3'
services:
  meilisync:
    platform: linux/x86_64
    image: long2ice/meilisync
    volumes:
      - ./meilisync-config.yml:/meilisync/config.yml
    restart: always
    depends_on:
      - meilisearch
  meilisearch:
    image: getmeili/meilisearch:v1.3
    ports:
      - "7700:7700"
    environment:
        - MEILI_ENV=development
        - MEILI_API_KEY=some-key

Sync config:

debug: true
progress:
  type: file
meilisearch:
  api_url: "http://meilisearch:7700"
  api_key: "API_KEY"

source:
  type: postgres
  host: DB_IP
  port: 5432
  database: inro
  user: replication_user
  password: "PASSWORD"

sync:
  - table: PG_SCHEMA_NAME.table_name
    index: table_name
    full: true

Logs from containers:

infrastructure-meilisearch-1  | [2023-09-19T09:12:16Z INFO  actix_server::builder] starting 6 workers
infrastructure-meilisearch-1  | [2023-09-19T09:12:16Z INFO  actix_server::server] Actix runtime found; starting in Actix runtime
infrastructure-meilisync-1    | 2023-09-19 09:12:26.540 | DEBUG    | meilisync.main:_:33 - plugins=None progress=Progress(type=<ProgressType.file: 'file'>) debug=True source=Source(type=<SourceType.postgres: 'postgres'>, database='inro', password='', port=5432, host='', user='replication_user') meilisearch=MeiliSearch(api_url='http://meilisearch:7700', api_key='iwBx-', insert_size=None, insert_interval=None) sync=[Sync(plugins=None, table='items', pk='id', full=True, index='', fields=None), Sync(plugins=None, table='manifest', pk='id', full=True, index='manifest', fields=None)] sentry=None
infrastructure-meilisearch-1  | [2023-09-19T09:12:27Z INFO  actix_web::middleware::logger] 172.18.0.3 "POST /indexes/l/documents?primaryKey=id HTTP/1.1" 202 142 "-" "python-httpx/0.23.3" 0.007829
infrastructure-meilisearch-1  | [2023-09-19T09:12:27Z INFO  actix_web::middleware::logger] 172.18.0.3 "POST /indexes/l/documents?primaryKey=id HTTP/1.1" 202 142 "-" "python-httpx/0.23.3" 0.008936
infrastructure-meilisearch-1  | [2023-09-19T09:12:27Z INFO  index_scheduler] A batch of tasks was successfully completed.
infrastructure-meilisearch-1  | [2023-09-19T09:12:27Z INFO  index_scheduler] A batch of tasks was successfully completed.
infrastructure-meilisync-1    | 2023-09-19 09:12:27.746 | INFO     | meilisync.main:_:82 - Full data sync for table "items" done! 1378 documents added.
infrastructure-meilisearch-1  | [2023-09-19T09:12:27Z INFO  actix_web::middleware::logger] 172.18.0.3 "POST /indexes//documents?primaryKey=id HTTP/1.1" 202 143 "-" "python-httpx/0.23.3" 0.004431
infrastructure-meilisearch-1  | [2023-09-19T09:12:27Z INFO  index_scheduler] A batch of tasks was successfully completed.
infrastructure-meilisync-1    | 2023-09-19 09:12:27.804 | INFO     | meilisync.main:_:82 - Full data sync for table "manifest" done! 409 documents added.

I'm confused by the last log, it states that X documents was added, but indexes are empty: