Support HTTP headers streaming

[IvanTakarlikov-st]

Clickhouse able to send query progress in header during query execution. Documentation Example (no anchor, just find send_progress_in_http_headers

When we're making a request, we're getting all progress headers in the end of query (when query finished and we got the response) How can we read stream of headers during execution?

We can't provide a real example, but that's how you can reproduce it with curl if you have some clickhouse node to access:

echo "SOME LONG QUERY WITH PROGRESS" | curl -vvv -XPOST 'clickhouse_node:8123/?user=USER&send_progress_in_http_headers=1&query=' --data-binary @-

> POST /?user=USER&send_progress_in_http_headers=1&query= HTTP/1.1
> Host: clickhouse_node:8123
> User-Agent: curl/7.79.1
> Accept: */*
> Content-Length: 425
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Thu, 16 Nov 2023 14:10:50 GMT
< Connection: Keep-Alive
< Content-Type: text/tab-separated-values; charset=UTF-8
< Transfer-Encoding: chunked
< X-ClickHouse-Query-Id: QUERY_ID
< X-ClickHouse-Format: TabSeparated
< X-ClickHouse-Timezone: Zulu
< Keep-Alive: timeout=3
< X-ClickHouse-Progress: {"read_rows":"65661","read_bytes":"7794203","written_rows":"0","written_bytes":"0","total_rows_to_read":"47703090","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"2983003","read_bytes":"355204021","written_rows":"0","written_bytes":"0","total_rows_to_read":"5267326958","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"6136877","read_bytes":"729970296","written_rows":"0","written_bytes":"0","total_rows_to_read":"5267326958","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"9207044","read_bytes":"1094719900","written_rows":"0","written_bytes":"0","total_rows_to_read":"5267326958","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"12303523","read_bytes":"1462010911","written_rows":"0","written_bytes":"0","total_rows_to_read":"5267326958","result_rows":"0","result_bytes":"0"}
< X-ClickHouse-Progress: {"read_rows":"15088486","read_bytes":"1794558825","written_rows":"0","written_bytes":"0","total_rows_to_read":"5267326958","result_rows":"0","result_bytes":"0"}

As you can see above, curl allow to stream headers in realtime.

If it really needed, we can setup some dockerized example.

[tarcieri]

We don't currently support this, only streaming bodies.

It might be possible to add but would require careful API considerations.

[ixti]

I think this is something we want though, in order to control response consumption.