search

PeerDB-io / peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
https://peerdb.io
Other
2.2k stars 91 forks source link

Err initial load: DNS_ERROR #2151

Open islamyakin opened 1 week ago

islamyakin commented 1 week ago

mirroring postgres to clickhouse,

with instalation peerdb ./run-peerdb.sh.

i have a error

failed to sync records: code: 499, message: Failed to get object info: Code: 198. DB::NetException: Not found address of host: host.docker.internal. (DNS_ERROR) (version 24.9.2.42 (official build)). HTTP response code: 18446744073709551615: while reading clone_click_exa_ak_rezas_e13ff3d0_c93b_4187_9e81_7157a11ee53f/clone_click_exa_ak_rezas_e13ff3d0_c93b_4187_9e81_7157a11ee53f/f38ac276-6d49-4b92-9314-8844a7e336dc.avro.zst: While executing S3Source

islamyakin commented 1 week ago

has been config in my virtual machine

127.0.0.1 host.docker.internal 127.0.0.1 gateway.docker.internal

iamKunalGupta commented 1 week ago

Hi can you please provide some more information:

islamyakin commented 1 week ago

Hi can you please provide some more information:

  • How is ClickHouse setup? (Cloud/VM/Docker)

setup in VM

islamyakin commented 1 week ago

Clickhouse and peerdb in different host

iamKunalGupta commented 1 week ago

Minio needs to be accessible by both PeerDB and ClickHouse (via the same URL). In that case I'd recommend setting the AWS Endpoint for S3 to maybe the url of the minio VM and make the minio VM accessible to ClickHouse.

islamyakin commented 1 week ago

i see, clickhouse can access minio in docker-compose with peerdb? this is my docker-compose

name: peerdb-quickstart

x-minio-config: &minio-config PEERDB_CLICKHOUSE_AWS_CREDENTIALS_AWS_ACCESS_KEY_ID: _1 PEERDB_CLICKHOUSE_AWS_CREDENTIALS_AWS_SECRET_ACCESS_KEY: _1 PEERDB_CLICKHOUSE_AWS_CREDENTIALS_AWS_REGION: us-east-1 PEERDB_CLICKHOUSE_AWS_CREDENTIALS_AWS_ENDPOINT_URL_S3: http://minio:9000 PEERDB_CLICKHOUSE_AWS_S3_BUCKET_NAME: peerdbbucket1

iamKunalGupta commented 1 week ago

i see, clickhouse can access minio in docker-compose with peerdb?

Yes ClickHouse needs to access the object storage that PeerDB also has access to.

In this case you can set PEERDB_CLICKHOUSE_AWS_CREDENTIALS_AWS_ENDPOINT_URL_S3 to http://: and make sure your security groups allow for the CH VM to access the minio VM. Alternatively you can use an S3 Bucket and use that instead.