search

PeerDB-io / peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
https://peerdb.io
Other
2.17k stars 88 forks source link

Clickhouse raw table: merge tree, different order by #2104

Closed Amogh-Bharadwaj closed 5 days ago

Amogh-Bharadwaj commented 6 days ago

Raw table DDL is now featuring merge tree with order by of batch ID and destination table name:


CREATE TABLE IF NOT EXISTS raw (
        _peerdb_uid String NOT NULL,
        _peerdb_timestamp Int64 NOT NULL,
        _peerdb_destination_table_name String NOT NULL,
        _peerdb_data String NOT NULL,
        _peerdb_record_type Int NOT NULL,
        _peerdb_match_data String,
        _peerdb_batch_id Int,
        _peerdb_unchanged_toast_columns String
) ENGINE = MergeTree() ORDER BY (_peerdb_batch_id, _peerdb_destination_table_name);

Functionally tested and normalize IIS seems to be slightly faster than what it is currently