Error what(): unimplemented 16333

[mrAndersen]

Hi! I im using clickhouse-cpp official driver to work with clickhouse-server from CPP client, and recently I've got this error after sending ->Ping() to server

Terminate called after throwing an instance of 'clickhouse::UnimplementedError'
what(): unimplemented 16333

Can someone explain what is this? As far as I see, this is packet type = 16333 (which I can not find to be valid packet type) was sent from server to client, which successfully crashed after that)

If this is the case, what is packet type 16333 and how I can resolve that in future? Server version = 23.3.1.2823, client compiled from v2.4.0.

->Ping() is sent every several seconds to the server from client, so this error is very unusual, because production was working fine for 3+ weeks

Den Crane redirected me here https://github.com/ClickHouse/ClickHouse/issues/57570

[Enmk]

There is no packet type 16333 in the CH server: https://github.com/ClickHouse/ClickHouse/blob/master/src/Core/Protocol.h

Most likely you are trying to reach CH on an SSL port with a non-SSL connection. You may want to try doing something like this:

        ClientOptions(ClickHouseExplorerConfig)
            .SetSSLOptions(ClientOptions::SSLOptions()
                    .SetPathToCADirectory(DEFAULT_CA_DIRECTORY_PATH)),

OR

        Client(ClientOptions(ClickHouseExplorerConfig)
                .SetSSLOptions(ClientOptions::SSLOptions()
                        .SetConfiguration({
                                {"VerifyCAPath", DEFAULT_CA_DIRECTORY_PATH},
                                {"MinProtocol", "TLSv1.3"},
                                {"VerifyMode", "Peer"},
                                {"no_comp"} // shorthand for command with no value
                        })
        ))

You need to call at least ClientOptions::SetSSLOptions(ClientOptions::SSLOptions()) on the options you are trying to connect with to try to establish a client-server session over SSL.

[mrAndersen]

There is no packet type 16333

Yeah that's why I am here, cos it's very strange that somehow I've got this.

Most likely you are trying to reach CH on an SSL port with a non-SSL connection

I don't think this is the case, because this was working for ages, and nothing was done to CH settings, nor to client settings, connection is plain, non SSL, port 9000 via native protocol

Maybe backtrace could help

backtrace #1 0x00005622dade5d0a sp=0x00007f4e7e59ac00 signalHandler(int) + 0x1fb
backtrace #2 0x00007f4e814d0520 sp=0x00007f4e7e59ad40 __sigaction + 0x50
backtrace #3 0x00007f4e814b6898 sp=0x00007f4e7e59ba40 abort + 0x178
backtrace #4 0x00007f4e8183fb9e sp=0x00007f4e7e59bb80 __cxa_throw_bad_array_new_length + 0x552
backtrace #5 0x00007f4e8184b20c sp=0x00007f4e7e59bbb0 std::rethrow_exception(std::__exception_ptr::exception_ptr) + 0x7c
backtrace #6 0x00007f4e8184b277 sp=0x00007f4e7e59bbc0 std::terminate() + 0x17
backtrace #7 0x00007f4e8184b4d8 sp=0x00007f4e7e59bbd0 __cxa_throw + 0x48
backtrace #8 0x00007f4e81e601fb sp=0x00007f4e7e59bbf0 clickhouse::Client::Impl::ReceivePacket(unsigned long*) + 0x567
backtrace #9 0x00007f4e81e602d6 sp=0x00007f4e7e59bcb0 clickhouse::Client::Impl::Ping() + 0x46
backtrace #10 0x00007f4e81e603a0 sp=0x00007f4e7e59bce0 clickhouse::Client::Ping() + 0x14
backtrace #11 0x00005622dade8c6c sp=0x00007f4e7e59bcf0 Connection::isConnected() + 0x30
backtrace #12 0x00005622dadec177 sp=0x00007f4e7e59bd20 std::thread::_State_impl<std::thread::_Invoker<std::tuple<Writer::detach()::{lambda()#1}> > >::_M_run() + 0x43
backtrace #13 0x00007f4e81879253 sp=0x00007f4e7e59be40 std::error_code::default_error_condition() const + 0x33
backtrace #14 0x00007f4e81522ac3 sp=0x00007f4e7e59be60 pthread_condattr_setpshared + 0x513
backtrace #15 0x00007f4e815b3bf4 sp=0x00007f4e7e59bf00 clone + 0x44

@Enmk I've now got this error in ReadBlock()

unsupported column type: Enum8('increment' = 1, 'gauge' = 2)

Something strange is going on, because I don't have any such column types in my tables...

[mrAndersen]

Update:

Now I am getting

unsupported column type: DateTime

Something is going wrong...

[Enmk]

This is weird... can you connect to the same endpoint with clickhouse-client ?

[mrAndersen]

Yeah, I can. And also this unsupported|Unimplemented errors are random. Production is working under high load inserting a lot of rows into CH via cpp-client, like 200-500k rows every 10 seconds, also doing some rare selects from the server. Data has same structure inserting into same tables, and this errors are once a day, or twice a day, started some couple of days ago, before that it was all good...

[Enmk]

Hm... that is interesting. Any version upgrades either on clickhouse-cpp or ch side? Maybe something new in the network setup like proxy?

By any chance, do those errors happen after some other errors?

[mrAndersen]

CH version is 23.3.1.2823 has uptime of 7 month, however config.xml could have been potentially changed, however this is unlikely, because last unix edit time on file is March 2023. Also I am getting this in server logs

<Warning> TCPHandler: Using deprecated interserver protocol because the client is too old. Consider upgrading all nodes in cluster.

But all server shards have the same version

[mrAndersen]

@Enmk How it could be possible that server responded with such data, that packet_type is 16333 inside ReceivePacket function? Cos it makes no sense...

Answering my own question, it could be some network routing errors, where clickhouse-cpp driver connected successfully to 9000 port, but it was not CH server itself, but some other applicaiton that is valid and it was responding something different from expected protocol, correct?

[Enmk]

@Enmk How it could be possible that server responded with such data, that packet_type is 16333 inside ReceivePacket function? Cos it makes no sense...

I don't know, it all is pretty mysterious. So far it looks like either:

• something is corrupting the internal buffers of the clickhouse-cpp (perhaps the library code itself, perhaps some user code, idk)
• you are re-using clickhouse::Client instances after there was an exception and hence some data in the socket/buffers from the previous dialogue with the server. -- In that case, please make sure to invoke Client::ResetConnection() before issuing another query.
• clickhouse::Client is used from multiple threads, corrupting the connection state. -- use one instance per thread
• (most improbable) some weird things happening in underlying layers: kernel socket buffers, networking stack, some middleware inserting/leaving some junk in the communication channel.

[mrAndersen]

Or connection is successfully established to NON CH server at TCP\9000 port (some other network service, port 9000 is popular) and when first data packet is sent (Ping for example) clickhouse-cpp is getting invalid response and everything described earlier is happening. Can this be an issue? Because connection code itself has not been changed for 6+ month, and everything u mentioned is applied to that code

[Enmk]

Or connection is successfully established to NON CH server at TCP\9000 port (some other network service, port 9000 is popular) and when first data packet is sent (Ping for example) clickhouse-cpp is getting invalid response and everything described earlier is happening. Can this be an issue? Because connection code itself has not been changed for 6+ month, and everything u mentioned is applied to that code

Maybe, but I don't know your setup and how plausible that version is.

[mrAndersen]

I have a bit more information. Seems like this is the case where somehow driver internal buffers are corrupted, because this problem only happens when application is getting SIGTERM, and DURING the period where application is gracefully shutting down, this happens, maybe driver somehow handles signals? Or maybe you can point me somewhere I can look up for this? Cos all 4 incidents occured when application got SIGTERM, it's finishing it's jobs and during that this is happening

[Enmk]

It doesn't handle any signals explicitly

[Enmk]

Hey @mrAndersen do you still face this issue?

[mrAndersen]

Hey @mrAndersen do you still face this issue?

No idea what it was, some corrupted TCP or maybe incorrect usage of ->ResetConnection(), anyway, I am very doubtful this is actual library problem, so can be closed. THX!