v3 Changes

[gingerwizard]

We aren't expecting to release v3 soon, but this issue will record changes (potentially breaking) we want to make and/or new features for addition.

• [ ] Make clickhouse.Options a value not a pointer. It can't be nil and main component structs are a pointer so copy is cheap. it also means we have immutability - allowing options to be reused. Currently, we ensure this with https://github.com/ClickHouse/clickhouse-go/pull/675 but this means mixed semantics internally.
• [ ] WithExternalTable should be WithExternalData to align with ClickHouse https://clickhouse.com/docs/en/engines/table-engines/special/external-data/
• [ ] Async inserts use the ExecContext for std and a custom AsyncInsert for native. This seems unnecessary since async insert is just a parameter. No reason for it to not just use normal batch semantics - possibly with a flag.
• [ ] Increase dialtimeout to align with ClickHouse default i.e. 10s
• [ ] Decouple batches and connections to allow Retriable/reusable batches - https://github.com/ClickHouse/clickhouse-go/issues/728
• [ ] Support more binary formats - Protobuf and RowBinary https://github.com/ClickHouse/clickhouse-go/issues/737

[StarpTech]

Any updates on this? We're looking for a native batch variant with async support. Exactly:

Async inserts use the ExecContext for std and a custom AsyncInsert for native. This seems unnecessary since async insert is just a parameter. No reason for it to not just use normal batch semantics - possibly with a flag.

The README reads like this is not supported yet with native protocol in any means.

[jkaflik]

@StarpTech can you elaborate? Async insert for native protocol is supported. Either via setting or using wrapper function AsyncInsert. Unfortunately, ClickHouse does not support reading native format input in native protocol, so the only way to INSERT is inline data in SQL query.

[StarpTech]

Hi, could you reference to an async insert batch example with the native protocol? In the README states:

Using native protocol, asynchronous insert does not support batching.

[jkaflik]

@StarpTech

When I wrote:

Unfortunately, ClickHouse does not support reading native format input in native protocol, so the only way to INSERT is inline data in SQL query.

I meant it's not possible with batch INSERT, however this has changed with the newest ClickHouse release: https://clickhouse.com/docs/en/whats-new/changelog#clickhouse-release-2310-2023-11-02 (this is not yet released to Cloud) apologies, I should have been check this before.

I haven't tested it yet, but you should be able to use async_insert setting with standard batch API:

    ctx = clickhouse.Context(ctx, clickhouse.WithSettings(clickhouse.Settings{
        "async_insert": 1,
        "wait_for_async_insert": 0,
    }))

    batch, err := conn.PrepareBatch(ctx, "INSERT INTO example")
    if err != nil {
        return err
    }
    for i := 0; i < 1000; i++ {
        err := batch.Append(
            uint8(42),
            "ClickHouse",
            "Inc",
            uuid.New(),
            map[string]uint8{"key": 1},             // Map(String, UInt8)
            []string{"Q", "W", "E", "R", "T", "Y"}, // Array(String)
            []any{ // Tuple(String, UInt8, Array(Map(String, String)))
                "String Value", uint8(5), []map[string]string{
                    {"key": "value"},
                    {"key": "value"},
                    {"key": "value"},
                },
            },
            time.Now(),
        )
        if err != nil {
            return err
        }
    }

    return batch.Send()

[StarpTech]

I haven't tested it yet, but you should be able to use async_insert setting with standard batch API:

@jkaflik according to the debug logs and system.asynchronous_insert_log it doesn't work