Add support for HTTP/HTTPS protocol for ClickHouse connection

QuesmaOrg / quesma

Programmable database gateway

https://quesma.com

Other

108 stars 6 forks source link

Add support for HTTP/HTTPS protocol for ClickHouse connection #938

Open avelanarius opened 3 weeks ago

avelanarius commented 3 weeks ago

This PR adds support for the HTTP (port 8123)/HTTPS (port 8443) protocol to connect to ClickHouse (in addition to the already supported clickhouse:// (port 9000) protocol).

This change is motivated by the fact that the HTTPS endpoint is the first/default endpoint shown on ClickHouse Cloud and by the fact that we saw a user try to use the HTTP endpoint.

Additionally this PR improves the "ClickHouse Connection Doctor" code and the validation of user-provided connection URLs. Related to that, RemoteLogDrainUrl and QuesmaInternalTelemetryUrl now can't be provided in the config - they were overwritten by hardcoded constants either way.

avelanarius commented 3 weeks ago

@mieciu I vaguely remember you mentioning something about http:// endpoint in Hydrolix, but from all I've seen Hydrolix also uses the clickhouse:// endpoint - just making sure that this change is also compatible with Hydrolix (I only tested ClickHouse OSS and ClickHouse Cloud).

mieciu commented 2 weeks ago

@mieciu I vaguely remember you mentioning something about http:// endpoint in Hydrolix, but from all I've seen Hydrolix also uses the clickhouse:// endpoint - just making sure that this change is also compatible with Hydrolix (I only tested ClickHouse OSS and ClickHouse Cloud).

So far we didn't really care about the protocol prefix - once connection has been established with host we relied on driver defaults, so that'd be something to check. I'm not sure if querying over http protocol is supported there.

avelanarius commented 2 weeks ago

Since there are some quirks with HTTP(S) in Hydrolix and this PR might not be mergable as-is, I extracted only the validation improvements to this (mergable) PR https://github.com/QuesmaOrg/quesma/pull/946

avelanarius commented 2 weeks ago

And the quirks are:

Hydrolix native protocol endpoint uses the https:// scheme: https://docs.hydrolix.io/docs/query-some-data

Connect to Hydrolix HTTPS API: https://{myhostname}}.hydrolix.live/query Clickhouse Native: https://{myhostname}}.hydrolix.live:9440
Hydrolix HTTPS API has a different API than ClickHouse HTTP(S) API: curl --request POST \ --url https://hostname.hydrolix.live/query \ --header 'accept: text/json' \ --header 'content-type: text/plain' \ --data 'SELECT count() FROM sample.taxi_trips' vs curl 'http://localhost:8123/?query=SELECT%201'

mieciu commented 2 weeks ago

Yea just to leave trace after our investigation:

http to ClickHouse works because we rely on https://github.com/ClickHouse/clickhouse-go library
http to Hydrolix currently is not working

This discrepancy is related to differences in how queries over http are handled in both DBs.

Relevant sources: ClickHouse / Hydrolix.