PeerDB-io / peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
https://peerdb.io
Other
2.17k stars 88 forks source link

Clickhouse + PeerDB OSS: check if minio bucket can be reached by Clickhouse #1976

Open Amogh-Bharadwaj opened 2 months ago

Amogh-Bharadwaj commented 2 months ago

In Clickhouse peer validation, have a step where we have a dummy avro file in a minio bucket and we try to do a select * from s3() from that file, to ensure that the user's setup will be able to enable PeerDB to take rows from bucket and insert into actual tables. Set a query timeout of 2 - 5 seconds or similar so that we get a context cancellation error and we can tell the user that 'hey, your clickhouse can't reach our bucket'