pinax-network / substreams-sink-sql

Apache License 2.0
0 stars 0 forks source link

Flushing issue on small block ranges #20

Closed 0237h closed 2 weeks ago

0237h commented 2 weeks ago

When trying to run the sink on small block ranges, presumably smaller than the default flush intervals, the sink doesn't appear to flush any data to the database.

To reproduce:

  1. Create a ClickHouse database with the following schema.sql
    
    CREATE TABLE IF NOT EXISTS cursors
    (
    id        String,
    cursor    String,
    block_num Int64,
    block_id  String
    )
    ENGINE = ReplacingMergeTree()
        PRIMARY KEY (id)
        ORDER BY (id);

CREATE TABLE IF NOT EXISTS contract_creation ( block_hash String, block_number UInt64, block_timestamp_seconds UInt64, contract_address String, creation_bytecode String, creator_address String, creator_tx String ) ENGINE = ReplacingMergeTree() PRIMARY KEY (block_hash, contract_address) ORDER BY (block_hash, contract_address);

2. Run the sink with the default parameters on a small block range (e.g. `13042456:13042458`)

sink-sql run clickhouse://user>:<password>@<node-url:9000/ https://github.com/pinax-network/subgraph-contract-creation/releases/download/v0.1.1/evm-contract-creation-tracker-v0.1.1.spkg 13042456:13042458 -e --undo-buffer-size 1 --plaintext

3. See that no data has been flushed to the DB, no messages mentioning `flushing` with `TRACE=true`.

If you run it on a bigger range (i.e. bigger than the flush interval), you should see those messages appear:

{"severity":"DEBUG","timestamp":"2024-10-29T15:06:02.486524953Z","logger":"sink-sql","message":"flushing to database","block":"#13041000 (fe1e4293ad23436efad71f00827b7eba2e7d342176546cb99e7686b0de10e9b4)","is_live":false,"block_flush_interval_reached":true,"row_flush_interval_reached":false,"logging.googleapis.com/labels":{}} {"severity":"DEBUG","timestamp":"2024-10-29T15:06:02.495166368Z","logger":"sink-sql","message":"flushing table entries","table_name":"contract_creation","entry_count":93,"logging.googleapis.com/labels":{}}

fschoell commented 2 weeks ago

this is fixed in #25