search

PeerDB-io / peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
https://peerdb.io
Other
2.15k stars 84 forks source link

Timeplus Proton sink support #1296

Open jovezhong opened 7 months ago

jovezhong commented 7 months ago

Today I noticed the announcement of low latency replication from PostgreSQL to ClickHouse. Congratulations.

It'll be great you can build a sink to Timeplus Proton, which is powered by ClickHouse, and focuses a lot on data streaming.

If realtime changes in PostgreSQL can be sent to Proton, developers can build real-time alerts and powerful streaming SQL in Proton, without using Debezium, Kafka and Flink. This will greatly simplify the deployment.

To add data to Proton, you can use the https://github.com/timeplus-io/proton-go-driver or use the streaming ingest REST API: https://docs.timeplus.com/proton-ingest-api

curl -s -X POST http://localhost:3218/proton/v1/ingest/streams/foo \
-d '{
  "columns": ["id","name"],
  "data": [
    [1,"hello"],
    [2,"world"]
  ]
}
'

It's okay to insert data row by row, or in a small batch. Internally, Proton has a streaming storage for WAL, then synced to the internal historical storage(which is mainly based on ClickHouse).

Timeplus Cloud is the managed service for Proton, with extra UI and API. The Ingest REST API https://docs.timeplus.com/ingest-api is similar to Proton but access API Key, and more flexible format, such as JSON Lines, or multi-line raw text. This could be a different ticket/feature. It'll be great if the community can support Proton sink first.

Feel free to check www.timeplus.com to learn more or join our slack at https://timeplus.com/slack

jovezhong commented 6 months ago

May I know what's the plan for this? Recently we announced the native integration of Proton and ClickHouse, https://www.timeplus.com/post/proton-clickhouse-integration. If we can use PeerDB to sync PG data to Proton (in the similar way as ClickHouse), that will be awesome.