search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
9.04k stars 1.08k forks source link

[YCQL][LST] Error decoding response from Cassandra. ver(4); flags(0000); stream(57); op(8); offset(9); len(37); buffer: b'\x84\x00\x009\x08\x00\x00\x00%\x00\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00\x00\x00\rks_lst_833429\x00\x04ct_0\x00\x00\x00\x00' #13946

Open def- opened 2 years ago

def- commented 2 years ago

Jira Link: DB-3442

Description

On a local macOS run of the new LST YCQL feature @vijayjoshi16 is working on I ran this error that can be reproduced easily:

$ cd ~/code/yugabyte-db
$ git checkout dbb4a340fa6cf7ad9698ca57dfce8ae8085fb636
$ ./yb_build.sh release
$ bin/yb-ctl create
$ cd ~/code/yb-long-system-test
$ git checkout a66d6d77
$ ./long_system_test.py --ycql --runtime=1 --threads=10 --seed=833429
2022-09-09 11:03:53,936 MainThread INFO
2022-09-09 11:03:53,936 MainThread INFO     --------------------------------------------------------------------------------
2022-09-09 11:03:53,936 MainThread INFO     Running Long System Test 0.1 - https://github.com/yugabyte/yb-long-system-test/
2022-09-09 11:03:53,936 MainThread INFO     --------------------------------------------------------------------------------
2022-09-09 11:03:53,936 MainThread INFO
2022-09-09 11:03:54,013 MainThread INFO     Reproduce with: git checkout a66d6d77 && ./long_system_test.py --ycql --runtime=1 --threads=10 --seed=833429
2022-09-09 11:03:54,017 MainThread WARNING  Cluster.__init__ called with contact_points specified, but no load_balancing_policy. In the next major version, this will raise an error; please specify a load-balancing policy. (contact_points = ['localhost'], lbp = None)
2022-09-09 11:03:54,022 MainThread WARNING  Downgrading core protocol version from 66 to 65 for 127.0.0.1:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
2022-09-09 11:03:54,022 MainThread WARNING  Downgrading core protocol version from 65 to 5 for 127.0.0.1:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
2022-09-09 11:03:54,023 MainThread WARNING  Downgrading core protocol version from 5 to 4 for 127.0.0.1:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
2022-09-09 11:03:54,029 MainThread INFO     Using datacenter 'datacenter1' for DCAwareRoundRobinPolicy (via host '127.0.0.1:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes
2022-09-09 11:03:54,029 MainThread INFO     Cassandra host ::1:9042 removed
2022-09-09 11:03:54,172 MainThread INFO     Starting worker_0: YCQLRandomSelectAction, YCQLSingleInsertAction, YCQLSingleUpdateAction, YCQLBulkInsertAction, YCQLBulkUpdateAction, YCQLSingleDeleteAction
2022-09-09 11:03:54,173 MainThread INFO     Starting worker_1: YCQLRandomSelectAction, YCQLSingleInsertAction, YCQLSingleUpdateAction, YCQLBulkInsertAction, YCQLBulkUpdateAction, YCQLSingleDeleteAction
2022-09-09 11:03:54,173 MainThread INFO     Starting worker_2: YCQLRandomSelectAction, YCQLSingleInsertAction, YCQLSingleUpdateAction, YCQLBulkInsertAction, YCQLBulkUpdateAction, YCQLSingleDeleteAction
2022-09-09 11:03:54,173 MainThread INFO     Starting worker_3: YCQLRandomSelectAction, YCQLSingleInsertAction, YCQLSingleUpdateAction, YCQLBulkInsertAction, YCQLBulkUpdateAction, YCQLSingleDeleteAction
2022-09-09 11:03:54,176 MainThread INFO     Starting worker_4: YCQLRandomSelectAction, YCQLSingleInsertAction, YCQLSingleUpdateAction, YCQLBulkInsertAction, YCQLBulkUpdateAction, YCQLSingleDeleteAction
2022-09-09 11:03:54,177 MainThread INFO     Starting worker_5: YCQLRandomSelectAction, YCQLSingleInsertAction, YCQLSingleUpdateAction, YCQLBulkInsertAction, YCQLBulkUpdateAction, YCQLSingleDeleteAction
2022-09-09 11:03:54,177 MainThread INFO     Starting worker_6: YCQLRandomSelectAction, YCQLSingleInsertAction, YCQLSingleUpdateAction, YCQLBulkInsertAction, YCQLBulkUpdateAction, YCQLSingleDeleteAction
2022-09-09 11:03:54,178 MainThread INFO     Starting worker_7: YCQLRandomSelectAction, YCQLSingleInsertAction, YCQLSingleUpdateAction, YCQLBulkInsertAction, YCQLBulkUpdateAction, YCQLSingleDeleteAction
2022-09-09 11:03:54,179 MainThread INFO     Starting worker_8: YCQLRandomSelectAction, YCQLSingleInsertAction, YCQLSingleUpdateAction, YCQLBulkInsertAction, YCQLBulkUpdateAction, YCQLSingleDeleteAction
2022-09-09 11:03:54,181 MainThread INFO     Starting worker_9: YCQLRandomSelectAction, YCQLSingleInsertAction, YCQLSingleUpdateAction, YCQLBulkInsertAction, YCQLBulkUpdateAction, YCQLSingleDeleteAction
2022-09-09 11:03:54,183 asyncore_cassandra_driver_event_loop ERROR    Error decoding response from Cassandra. ver(4); flags(0000); stream(57); op(8); offset(9); len(37); buffer: b'\x84\x00\x009\x08\x00\x00\x00%\x00\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00\x00\x00\rks_lst_833429\x00\x04ct_0\x00\x00\x00\x00'
Traceback (most recent call last):
  File "cassandra/connection.py", line 1229, in cassandra.connection.Connection.process_msg
  File "cassandra/protocol.py", line 1196, in cassandra.protocol._ProtocolHandler.decode_message
  File "cassandra/protocol.py", line 744, in cassandra.protocol.ResultMessage.recv_body
  File "cassandra/protocol.py", line 730, in cassandra.protocol.ResultMessage.recv
  File "cassandra/row_parser.pyx", line 34, in cassandra.row_parser.make_recv_results_rows.recv_results_rows
  File "cassandra/deserializers.pyx", line 478, in cassandra.deserializers.make_deserializers
  File "cassandra/deserializers.pyx", line 519, in cassandra.deserializers.obj_array
  File "stringsource", line 154, in View.MemoryView.array.__cinit__
ValueError: Invalid shape in axis 0: 0.

I'll try to get further information as to what's going wrong. We are not using Vector and Array. I am using yb-cassandra-driver Python driver so that should work too.

def- commented 2 years ago

This seems to occur with a wide variety of types

2022-09-09 16:34:28,098 asyncore_cassandra_driver_event_loop ERROR    Error decoding response from Cassandra. ver(4); flags(0000); stream(57); op(8); offset(9); len(37); buffer: b'\x84\x00\x009\x08\x00\x00\x00%\x00\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00\x00\x00\rks_lst_758905\x00\x04ct_1\x00\x00\x00\x00\x84\x00\x001\x08\x00\x00\x00:\x00\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00\x02\x00\rks_lst_758905\x00\x04ct_0\x00\x06c1_int\x00\t\x00\x07c0_time\x00\x12\x00\x00\x00\x00\x84\x00\x005\x08\x00\x00\x00\x04\x00\x00\x00\x01\x84\x00\x008\x08\x00\x00\x00\x04\x00\x00\x00\x01\x84\x00\x006\x08\x00\x00\x00\x04\x00\x00\x00\x01\x84\x00\x002\x08\x00\x00\x00\x04\x00\x00\x00\x01\x84\x00\x003\x08\x00\x00\x00\x04\x00\x00\x00\x01\x84\x00\x007\x08\x00\x00\x00n\x00\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00\x05\x00\rks_lst_758905\x00\x04ct_1\x00\x0bc4_set_text\x00"\x00\r\x00\x0bc0_timeuuid\x00\x0f\x00\nc1_boolean\x00\x04\x00\x07c3_text\x00\r\x00\x0cc2_timestamp\x00\x0b\x00\x00\x00\x00\x84\x00\x000\x08\x00\x00\x00\x04\x00\x00\x00\x01\x84\x00\x004\x08\x00\x00\x00\x04\x00\x00\x00\x01'
Traceback (most recent call last):
  File "cassandra/connection.py", line 1229, in cassandra.connection.Connection.process_msg
  File "cassandra/protocol.py", line 1196, in cassandra.protocol._ProtocolHandler.decode_message
  File "cassandra/protocol.py", line 744, in cassandra.protocol.ResultMessage.recv_body
  File "cassandra/protocol.py", line 730, in cassandra.protocol.ResultMessage.recv
  File "cassandra/row_parser.pyx", line 34, in cassandra.row_parser.make_recv_results_rows.recv_results_rows
  File "cassandra/deserializers.pyx", line 478, in cassandra.deserializers.make_deserializers
  File "cassandra/deserializers.pyx", line 519, in cassandra.deserializers.obj_array
  File "stringsource", line 154, in View.MemoryView.array.__cinit__
ValueError: Invalid shape in axis 0: 0.
2022-09-09 16:34:28,098 ThreadPoolExecutor-0_1 WARNING  Host 127.0.0.1:9042 has been marked down
2022-09-09 16:34:28,098 worker_5   ERROR    Unexpected query failure: ValueError
Query: SELECT c4_set_text, c1_boolean, c0_timeuuid, c3_text from ct_1 LIMIT 0 OFFSET 11
  values: None
  runtime: 2022-09-09 16:34:28.079 - 2022-09-09 16:34:28.098
  supports explain: False
  supports rollback: False
  affected rows: None
Action: YCQLRandomSelectAction
Error class: ValueError
Error message: Invalid shape in axis 0: 0.

Just removing some doesn't fix it.

def- commented 2 years ago

The common pattern is that this always happens with LIMIT 0. I'll use a minimum limit 1 for now in LST.