search

timeplus-io / proton

A streaming SQL engine, a fast and lightweight alternative to ksqlDB and Apache Flink, 🚀 powered by ClickHouse.
https://timeplus.com
Apache License 2.0
1.39k stars 53 forks source link

Cannot create external stream with SCRAM-SHA-256 in `proton local`. Okay in docker #626

Open jovezhong opened 4 months ago

jovezhong commented 4 months ago

Describe what's wrong

In the proton single binary, creating kakfa external stream with SCRAM-SHA-256 failed with message

Code: 36. DB::Exception: Failed to create kafka handle: No provider for SASL mechanism SCRAM-SHA-256: recompile librdkafka with libsasl2 or openssl support. Current build options: PLAIN. (BAD_ARGUMENTS)

How to reproduce

CREATE STREAM ext_stream(`raw` string)
SETTINGS type = 'kafka', brokers = 'arriving-grizzly-8171-us1-kafka.upstash.io:9092', topic = 'mar24_test_msg_key', security_protocol = 'SASL_SSL', sasl_mechanism = 'SCRAM-SHA-256', username = 'mask', password = 'mask'

Error message and/or stacktrace

Additional context

yokofly commented 4 months ago

In the proton single binary

this is proton-local, right? proton-server is okay or not?

jovezhong commented 4 months ago

I hit the issue with proton local didn't try proton server

yokofly commented 4 months ago

the problem shall be proton-local it does not support external stream. same issue https://github.com/timeplus-io/proton/issues/411 I remember proton-local use another more slim/simple/short config, like it not support create stream. https://pastebin.ubuntu.com/p/TFB3ZrFXgz/

jovezhong commented 4 months ago

I guess you can use proton local to create an external stream with PLAIN, not SCRAM-SHA-256. It seems that certain lib is missing in the binary. I will be surprised if proton local cannot be used to create normal stream. If that's the case, then proton local is too limited. Such unexpected SQL should be blocked in that mode. Or what's the use case for proton local?

yokofly commented 4 months ago

in the expected case: this tool is handy for local analysis, like selecting some info from a local CSV file. but we are far from this because this is an experimental thing.

current case: create several random streams and demo how fast proton it is:) https://www.timeplus.com/post/scary-fast

yokofly commented 2 weeks ago

cross:

435386Z Jul 01 08:23:37 -- Adding contrib module krb5 (configuring with krb5-cmake)
2024-07-01T08:23:37.4444626Z Jul 01 08:23:37 CMake Warning at contrib/krb5-cmake/CMakeLists.txt:3 (message):
2024-07-01T08:23:37.4447659Z Jul 01 08:23:37   krb5 disabled in non-Linux and non-native-Darwin environments
2024-07-01T08:23:37.4448725Z Jul 01 08:23:37 
2024-07-01T08:23:37.4449276Z Jul 01 08:23:37 
2024-07-01T08:23:37.4449935Z Jul 01 08:23:37 -- Not using krb5
2024-07-01T08:23:37.4451127Z Jul 01 08:23:37 -- Adding contrib module cyrus-sasl (configuring with cyrus-sasl-cmake)
2024-07-01T08:23:37.4452368Z Jul 01 08:23:37 -- Not using cyrus-sasl
2024-07-01T08:23:37.4453584Z Jul 01 08:23:37 -- Adding contrib module libgsasl (configuring with libgsasl-cmake)
2024-07-01T08:23:37.4456584Z Jul 01 08:23:37 -- Not using gsasl
2024-07-01T08:23:37.4457779Z Jul 01 08:23:37 -- Adding contrib module libhdfs3 (configuring with libhdfs3-cmake)
2024-07-01T08:23:37.4462465Z Jul 01 08:23:37 -- Not using hdfs
2024-07-01T08:23:37.4464210Z Jul 01 08:23:37 -- Adding contrib module librdkafka (configuring with librdkafka-cmake)
2024-07-01T08:23:37.4469920Z Jul 01 08:23:37 -- librdkafka with SSL support
2024-07-01T08:23:37.4478605Z Jul 01 08:23:37 -- Adding contrib module libpqxx (configuring with libpqxx-cmake)

native:

-- Adding contrib module krb5 (configuring with krb5-cmake)
-- Adding contrib module cyrus-sasl (configuring with cyrus-sasl-cmake)
-- Adding contrib module libgsasl (configuring with libgsasl-cmake)
-- Adding contrib module libhdfs3 (configuring with libhdfs3-cmake)
-- Not using hdfs
-- Adding contrib module librdkafka (configuring with librdkafka-cmake)
-- librdkafka with SASL support
-- librdkafka with SSL support
jovezhong commented 2 weeks ago

It seems cyrus-sasl failed to be added when we cross-platform compile it. We don't have to support gasal or hdfs

jovezhong commented 1 week ago

I will verify this in Proton 1.5.13

yokofly commented 1 week ago

please give a try to https://github.com/timeplus-io/proton/releases/tag/v1.5.13 proton-1.5.13-Darwin-arm64-native

jovezhong commented 1 week ago

I tried this 1.5.13 on mac arm. Creating external stream with SHA-256 was successful, but cannot read data

select * from car_live_data

2024.07.14 16:40:55.550236 [ 41596627 ] {ecd6a33a-88f3-49f0-8074-026a6eeaf168} <Information> executeQuery: Read 2 rows, 69.00 B in 0.00395 sec., 506 rows/sec., 17.06 KiB/sec.
2024.07.14 16:41:01.398236 [ 41596627 ] {75a5e74c-a3d2-4d9c-b833-23b81b4d198e} <Warning> KafkaExternalStream: rdkafka#consumer-1|CONFWARN buf=[thrd:app]: Configuration property enable.idempotence is a producer property and will be ignored by this consumer instance
2024.07.14 16:41:01.398271 [ 41596627 ] {75a5e74c-a3d2-4d9c-b833-23b81b4d198e} <Warning> KafkaExternalStream: rdkafka#consumer-1|CONFWARN buf=[thrd:app]: Configuration property dr_msg_cb is a producer property and will be ignored by this consumer instance
2024.07.14 16:41:01.398278 [ 41596627 ] {75a5e74c-a3d2-4d9c-b833-23b81b4d198e} <Warning> KafkaExternalStream: rdkafka#consumer-1|CONFWARN buf=[thrd:app]: Configuration property message.timeout.ms is a producer property and will be ignored by this consumer instance
2024.07.14 16:41:01.398293 [ 41596627 ] {75a5e74c-a3d2-4d9c-b833-23b81b4d198e} <Information> car_live_data.rdkafka#consumer-1: Created consumer
2024.07.14 16:41:03.804518 [ 41596627 ] {75a5e74c-a3d2-4d9c-b833-23b81b4d198e} <Information> car_live_data: Reading topic=car_live_data partitions=[0] streaming=true
2024.07.14 16:41:03.805367 [ 41596627 ] {75a5e74c-a3d2-4d9c-b833-23b81b4d198e} <Information> car_live_data: Starting reading 1 streams by seeking to  with rdkafka#consumer-1 in dedicated resource group
2024.07.14 16:41:03.806294 [ 41596944 ] {75a5e74c-a3d2-4d9c-b833-23b81b4d198e} <Information> PipelineExecutor: Using 10 threads to execute pipeline for query_id=75a5e74c-a3d2-4d9c-b833-23b81b4d198e
2024.07.14 16:41:03.806410 [ 41596942 ] {75a5e74c-a3d2-4d9c-b833-23b81b4d198e} <Information> car_live_data.rdkafka#consumer-1: Start consuming from topic=car_live_data shard=0 offset=-1 high_watermark=9223372036854775807
2024.07.14 16:41:03.806637 [ 41596941 ] {} <Information> car_live_data.rdkafka#consumer-1: Start consumer poll
2024.07.14 16:41:13.699462 [ 41596875 ] {} <Information> LogManager: Beginning checkpoint log recovery sns...
2024.07.14 16:41:13.699459 [ 41596879 ] {} <Information> LogManager: Beginning flush dirty logs...
2024.07.14 16:41:13.699495 [ 41596872 ] {} <Information> LogManager: Beginning delete logs...
2024.07.14 16:41:13.699491 [ 41596876 ] {} <Information> LogManager: Beginning checkpoint log start sns...
2024.07.14 16:41:13.699457 [ 41596873 ] {} <Information> LogManager: Beginning log cleanup...
2024.07.14 16:41:13.699585 [ 41596873 ] {} <Information> LogManager: Log cleanup completed: 0 files were deleted in 0 milliseconds
2024.07.14 16:42:13.704686 [ 41596881 ] {} <Information> LogManager: Beginning checkpoint log start sns...
2024.07.14 16:42:13.704679 [ 41596877 ] {} <Information> LogManager: Beginning flush dirty logs...
2024.07.14 16:42:13.704796 [ 41596880 ] {} <Information> LogManager: Beginning checkpoint log recovery sns...
2024.07.14 16:43:13.706381 [ 41596872 ] {} <Information> LogManager: Beginning checkpoint log recovery sns...
2024.07.14 16:43:13.706379 [ 41596879 ] {} <Information> LogManager: Beginning flush dirty logs...
2024.07.14 16:43:13.706383 [ 41596875 ] {} <Information> LogManager: Beginning checkpoint log start sns...
2024.07.14 16:44:13.707024 [ 41596876 ] {} <Information> LogManager: Beginning flush dirty logs...
2024.07.14 16:44:13.707578 [ 41596873 ] {} <Information> LogManager: Beginning checkpoint log recovery sns...
2024.07.14 16:44:13.707581 [ 41596881 ] {} <Information> LogManager: Beginning checkpoint log start sns...

select count() from table(car_live_data)


0 rows in set. Elapsed: 3.499 sec.

Received exception from server (version 1.5.13):
Code: 1002. DB::Exception: Received from localhost:8463. DB::Exception: Failed to query watermark offsets topic=car_live_data parition=0 error=Broker: Not leader for partition. (UNKNOWN_EXCEPTION)

024.07.14 16:45:27.694645 [ 41596627 ] {87389147-c2e9-439d-a574-9303848b96ba} <Warning> KafkaExternalStream: rdkafka#consumer-2|CONFWARN buf=[thrd:app]: Configuration property enable.idempotence is a producer property and will be ignored by this consumer instance
2024.07.14 16:45:27.694684 [ 41596627 ] {87389147-c2e9-439d-a574-9303848b96ba} <Warning> KafkaExternalStream: rdkafka#consumer-2|CONFWARN buf=[thrd:app]: Configuration property dr_msg_cb is a producer property and will be ignored by this consumer instance
2024.07.14 16:45:27.694691 [ 41596627 ] {87389147-c2e9-439d-a574-9303848b96ba} <Warning> KafkaExternalStream: rdkafka#consumer-2|CONFWARN buf=[thrd:app]: Configuration property message.timeout.ms is a producer property and will be ignored by this consumer instance
2024.07.14 16:45:27.694710 [ 41596627 ] {87389147-c2e9-439d-a574-9303848b96ba} <Information> car_live_data.rdkafka#consumer-2: Created consumer
2024.07.14 16:45:29.403456 [ 41596627 ] {87389147-c2e9-439d-a574-9303848b96ba} <Information> car_live_data: Counting number of messages topic=car_live_data partitions=[0]
2024.07.14 16:45:31.177972 [ 41596627 ] {87389147-c2e9-439d-a574-9303848b96ba} <Information> car_live_data.rdkafka#consumer-2: Stopped
2024.07.14 16:45:31.179659 [ 41596627 ] {87389147-c2e9-439d-a574-9303848b96ba} <Error> executeQuery: Code: 1002. DB::Exception: Failed to query watermark offsets topic=car_live_data parition=0 error=Broker: Not leader for partition. (UNKNOWN_EXCEPTION) (version 1.5.13) (from [::1]:54647) (in query: select count() from table(car_live_data))
2024.07.14 16:45:31.186693 [ 41596627 ] {87389147-c2e9-439d-a574-9303848b96ba} <Error> TCPHandler: Code: 1002. DB::Exception: Failed to query watermark offsets topic=car_live_data parition=0 error=Broker: Not leader for partition. (UNKNOWN_EXCEPTION), Stack trace (when copying this message, always include the lines below):

<Empty trace>