search

scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
56 stars 94 forks source link

Invalid partition key in a modification statement when modifying CDC tables #3949

Closed KnifeyMoloko closed 1 year ago

KnifeyMoloko commented 3 years ago

Error log:

Numerous rolling upgrade tests fail when trying to modify CDC tables, with the direct error being:

09:26:23  < t:2021-09-10 07:26:22,580 f:fill_db_data.py l:3191 c:UpgradeTest          p:INFO  > Read CDC tables for test "nonpure_function_collection_test: CASSANDRA-5795" in 0 seconds
09:26:23  < t:2021-09-10 07:26:22,610 f:fill_db_data.py l:3191 c:UpgradeTest          p:INFO  > Update CDC tables for test "nonpure_function_collection_test: CASSANDRA-5795" in 0 seconds
09:26:23  < t:2021-09-10 07:26:22,618 f:fill_db_data.py l:3255 c:sdcm.fill_db_data    p:ERROR > DELETE FROM empty_in_test1 WHERE k1 IN () < t:2021-09-10 07:26:22,618 f:fill_db_data.py l:3255 c:sdcm.fill_db_data    p:ERROR > DELETE FROM empty_in_test1 WHERE k1 IN ()
09:26:23  < t:2021-09-10 07:26:22,618 f:fill_db_data.py l:3255 c:sdcm.fill_db_data    p:ERROR > Traceback (most recent call last):
09:26:23  < t:2021-09-10 07:26:22,618 f:fill_db_data.py l:3255 c:sdcm.fill_db_data    p:ERROR >   File "/jenkins/slave/workspace/scylla-master/rolling-upgrade/rolling-upgrade-centos8-test/scylla-cluster-tests/sdcm/fill_db_data.py", line 3252, in _run_db_queries
09:26:23  < t:2021-09-10 07:26:22,618 f:fill_db_data.py l:3255 c:sdcm.fill_db_data    p:ERROR >     res = session.execute(item['queries'][i])
09:26:23  < t:2021-09-10 07:26:22,618 f:fill_db_data.py l:3255 c:sdcm.fill_db_data    p:ERROR >   File "/jenkins/slave/workspace/scylla-master/rolling-upgrade/rolling-upgrade-centos8-test/scylla-cluster-tests/sdcm/utils/common.py", line 1331, in execute_verbose
09:26:23  < t:2021-09-10 07:26:22,618 f:fill_db_data.py l:3255 c:sdcm.fill_db_data    p:ERROR >     return execute_orig(*args, **kwargs)
09:26:23  < t:2021-09-10 07:26:22,618 f:fill_db_data.py l:3255 c:sdcm.fill_db_data    p:ERROR >   File "cassandra/cluster.py", line 2611, in cassandra.cluster.Session.execute
09:26:23  < t:2021-09-10 07:26:22,618 f:fill_db_data.py l:3255 c:sdcm.fill_db_data    p:ERROR >   File "cassandra/cluster.py", line 4829, in cassandra.cluster.ResponseFuture.result
09:26:23  < t:2021-09-10 07:26:22,618 f:fill_db_data.py l:3255 c:sdcm.fill_db_data    p:ERROR > cassandra.InvalidRequest: Error from server: code=2200 [Invalid query] message="Invalid partition key in a modification statement"

This is preceded by the following error:

08:30:27  < t:2021-09-10 06:30:27,335 f:fill_db_data.py l:3191 c:UpgradeTest          p:INFO  > Ran queries for test "nan_infinity_test" in 0 seconds
08:30:27  < t:2021-09-10 06:30:27,357 f:fill_db_data.py l:3278 c:sdcm.fill_db_data    p:ERROR > content was differ cdc tables content are differes
08:30:27  < t:2021-09-10 06:30:27,357 f:fill_db_data.py l:3278 c:sdcm.fill_db_data    p:ERROR >  Initial:[Row(cdc_stream_id=b"\x9bT\x00\x00\x00\x00\x00\x00\xc0'\x04\x05\x08\x00\x07q", cdc_time=UUID('481893c0-1200-11ec-b083-59cce1710651'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b"\x9bT\x00\x00\x00\x00\x00\x00\xc0'\x04\x05\x08\x00\x07q", cdc_time=UUID('481893c0-1200-11ec-b083-59cce1710651'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b"\x9bT\x00\x00\x00\x00\x00\x00\xc0'\x04\x05\x08\x00\x07q", cdc_time=UUID('481910c0-1200-11ec-a24b-eacccfe0f6a5'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=0, cdc_ttl=None, f=nan), Row(cdc_stream_id=b"\x9bT\x00\x00\x00\x00\x00\x00\xc0'\x04\x05\x08\x00\x07q", cdc_time=UUID('481910c0-1200-11ec-a24b-eacccfe0f6a5'), cdc_batch_seq_no=1, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b"\x9bT\x00\x00\x00\x00\x00\x00\xc0'\x04\x05\x08\x00\x07q", cdc_time=UUID('481910c0-1200-11ec-a24b-eacccfe0f6a5'), cdc_batch_seq_no=2, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'U<\x00\x00\x00\x00\x00\x00Z\xd1\xafx\x9c\x006A', cdc_time=UUID('48197a60-1200-11ec-a6bb-a92e8ed9bb28'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'U<\x00\x00\x00\x00\x00\x00Z\xd1\xafx\x9c\x006A', cdc_time=UUID('48197a60-1200-11ec-a6bb-a92e8ed9bb28'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'l\xfa\x00\x00\x00\x00\x00\x00=\\>\xb3\xdc\x00;\xc1', cdc_time=UUID('4819e798-1200-11ec-69fc-518639c2d4e1'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=-inf), Row(cdc_stream_id=b'l\xfa\x00\x00\x00\x00\x00\x00=\\>\xb3\xdc\x00;\xc1', cdc_time=UUID('4819e798-1200-11ec-69fc-518639c2d4e1'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=-inf)]
08:30:27  < t:2021-09-10 06:30:27,357 f:fill_db_data.py l:3278 c:sdcm.fill_db_data    p:ERROR > New_result: [Row(cdc_stream_id=b"\x9bT\x00\x00\x00\x00\x00\x00\xc0'\x04\x05\x08\x00\x07q", cdc_time=UUID('481893c0-1200-11ec-b083-59cce1710651'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b"\x9bT\x00\x00\x00\x00\x00\x00\xc0'\x04\x05\x08\x00\x07q", cdc_time=UUID('481893c0-1200-11ec-b083-59cce1710651'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b"\x9bT\x00\x00\x00\x00\x00\x00\xc0'\x04\x05\x08\x00\x07q", cdc_time=UUID('481910c0-1200-11ec-a24b-eacccfe0f6a5'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=0, cdc_ttl=None, f=nan), Row(cdc_stream_id=b"\x9bT\x00\x00\x00\x00\x00\x00\xc0'\x04\x05\x08\x00\x07q", cdc_time=UUID('481910c0-1200-11ec-a24b-eacccfe0f6a5'), cdc_batch_seq_no=1, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b"\x9bT\x00\x00\x00\x00\x00\x00\xc0'\x04\x05\x08\x00\x07q", cdc_time=UUID('481910c0-1200-11ec-a24b-eacccfe0f6a5'), cdc_batch_seq_no=2, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'U<\x00\x00\x00\x00\x00\x00Z\xd1\xafx\x9c\x006A', cdc_time=UUID('48197a60-1200-11ec-a6bb-a92e8ed9bb28'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'U<\x00\x00\x00\x00\x00\x00Z\xd1\xafx\x9c\x006A', cdc_time=UUID('48197a60-1200-11ec-a6bb-a92e8ed9bb28'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'l\xfa\x00\x00\x00\x00\x00\x00=\\>\xb3\xdc\x00;\xc1', cdc_time=UUID('4819e798-1200-11ec-69fc-518639c2d4e1'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=-inf), Row(cdc_stream_id=b'l\xfa\x00\x00\x00\x00\x00\x00=\\>\xb3\xdc\x00;\xc1', cdc_time=UUID('4819e798-1200-11ec-69fc-518639c2d4e1'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=-inf)]

Test: upgrade_test.UpgradeTest.test_rolling_upgrade Build number: 88 Backend: gce: eu-west-1 Kernel version: 4.18.0-305.12.1.el8_4.x86_64 Test-id: bacafda7-7214-4cf1-892a-51c0908d4b07 ScyllaDB version: 4.5.rc7-0.20210906.edead1caf with build-id bcf5092fb1da0a426c8f125fe31a2e2570132a64 (-) Target ScyllaDB repo: http://downloads.scylladb.com/unstable/scylla/master/rpm/centos/2021-09-05T22:40:15Z/scylla.repo Instance type: n1-highmem-8 Number of ScyllaDB nodes: 4

Logs: grafana - https://cloudius-jenkins-test.s3.amazonaws.com/bacafda7-7214-4cf1-892a-51c0908d4b07/20210910_072636/grafana-screenshot-overview-20210910_072636-rolling-upgrade-master-centos-monitor-node-bacafda7-0-1.png grafana - https://cloudius-jenkins-test.s3.amazonaws.com/bacafda7-7214-4cf1-892a-51c0908d4b07/20210910_072636/grafana-screenshot-rolling-upgrade-centos8-test-scylla-per-server-metrics-nemesis-20210910_072859-rolling-upgrade-master-centos-monitor-node-bacafda7-0-1.png db-cluster - https://cloudius-jenkins-test.s3.amazonaws.com/bacafda7-7214-4cf1-892a-51c0908d4b07/20210910_073206/db-cluster-bacafda7.tar.gz loader-set - https://cloudius-jenkins-test.s3.amazonaws.com/bacafda7-7214-4cf1-892a-51c0908d4b07/20210910_073206/loader-set-bacafda7.tar.gz monitor-set - https://cloudius-jenkins-test.s3.amazonaws.com/bacafda7-7214-4cf1-892a-51c0908d4b07/20210910_073206/monitor-set-bacafda7.tar.gz sct-runner - https://cloudius-jenkins-test.s3.amazonaws.com/bacafda7-7214-4cf1-892a-51c0908d4b07/20210910_073206/sct-runner-bacafda7.tar.gz

Jenkins job URL: https://jenkins.scylladb.com/job/scylla-master/job/rolling-upgrade/job/rolling-upgrade-centos8-test/88/

roydahan commented 3 years ago

@aleksbykov is this something you added?

aleksbykov commented 3 years ago

Yes, but issue happened not with CDC. Next query is marked as invalid for test "empty_in_test": "DELETE FROM empty_in_test1 WHERE k1 IN ()" Failed happened, after 1st node was upgrade. Is it possible that on 4.5.rc7 this query syntax is invalid?

roydahan commented 3 years ago

We need to debug it. It may point on a scylla issue.

@KnifeyMoloko can you please try to debug it? You can try to run this query as part of ccm / dtest on master and 4.5.rc7 and prior build.

fgelcer commented 3 years ago

i'm seeing very similar issue, with rolling upgrade on master (centos7, centos8, debian10, ubuntu16, ubuntu18, ubuntu20):

Test details Test: upgrade_test.UpgradeTest.test_rolling_upgrade Build number: 196 Backend: gce: eu-west-1 Kernel version: 3.10.0-1160.42.2.el7.x86_64 Test-id: 2631e9a8-833e-49de-a161-84b203fc85fc Start time: 2021-10-10 10:39:03 End time: 2021-10-10 11:20:17 Started by user: fabio Cassandra-stress uses shared-aware driver

System under test ScyllaDB version: 4.5.rc7-0.20210906.edead1caf with build-id bcf5092fb1da0a426c8f125fe31a2e2570132a64 (-) Target ScyllaDB repo: http://downloads.scylladb.com/unstable/scylla/master/rpm/centos/2021-10-10T01:02:41Z/scylla.repo Instance type: n1-highmem-8 Number of ScyllaDB nodes: 4

< t:2021-10-10 11:07:32,005 f:fill_db_data.py l:3278 c:sdcm.fill_db_data    p:ERROR > content was differ cdc tables content are differes
< t:2021-10-10 11:07:32,005 f:fill_db_data.py l:3278 c:sdcm.fill_db_data    p:ERROR >  Initial:[Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdf3840-29b9-11ec-292d-928c9926a1af'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdf3840-29b9-11ec-292d-928c9926a1af'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=0, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=1, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=2, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'T\xbc\x0c\r\xb8\x8a\x10\xc1\xf2\x88X\x81$\x006A', cdc_time=UUID('efe01846-29b9-11ec-a11a-137a13384ff1'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'T\xbc\x0c\r\xb8\x8a\x10\xc1\xf2\x88X\x81$\x006A', cdc_time=UUID('efe01846-29b9-11ec-a11a-137a13384ff1'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'm\x9a\x00\x00\x00\x00\x00\x00\xa0(\xee>$\x00;\xe1', cdc_time=UUID('efe0877c-29b9-11ec-a56d-cc4c93edbdef'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=-inf), Row(cdc_stream_id=b'm\x9a\x00\x00\x00\x00\x00\x00\xa0(\xee>$\x00;\xe1', cdc_time=UUID('efe0877c-29b9-11ec-a56d-cc4c93edbdef'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=-inf)]
< t:2021-10-10 11:07:32,005 f:fill_db_data.py l:3278 c:sdcm.fill_db_data    p:ERROR > New_result: [Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdf3840-29b9-11ec-292d-928c9926a1af'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdf3840-29b9-11ec-292d-928c9926a1af'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=0, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=1, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=2, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'T\xbc\x0c\r\xb8\x8a\x10\xc1\xf2\x88X\x81$\x006A', cdc_time=UUID('efe01846-29b9-11ec-a11a-137a13384ff1'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'T\xbc\x0c\r\xb8\x8a\x10\xc1\xf2\x88X\x81$\x006A', cdc_time=UUID('efe01846-29b9-11ec-a11a-137a13384ff1'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'm\x9a\x00\x00\x00\x00\x00\x00\xa0(\xee>$\x00;\xe1', cdc_time=UUID('efe0877c-29b9-11ec-a56d-cc4c93edbdef'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=-inf), Row(cdc_stream_id=b'm\x9a\x00\x00\x00\x00\x00\x00\xa0(\xee>$\x00;\xe1', cdc_time=UUID('efe0877c-29b9-11ec-a56d-cc4c93edbdef'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=-inf)]
fgelcer commented 3 years ago

maybe the problem is with this 4.5 build (during the pre-test) and we should run with 4.5.0 instead?

aleksbykov commented 3 years ago 
< t:2021-10-10 11:07:32,005 f:fill_db_data.py l:3278 c:sdcm.fill_db_data    p:ERROR > content was differ cdc tables content are differes
< t:2021-10-10 11:07:32,005 f:fill_db_data.py l:3278 c:sdcm.fill_db_data    p:ERROR >  Initial:[Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdf3840-29b9-11ec-292d-928c9926a1af'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdf3840-29b9-11ec-292d-928c9926a1af'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=0, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=1, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=2, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'T\xbc\x0c\r\xb8\x8a\x10\xc1\xf2\x88X\x81$\x006A', cdc_time=UUID('efe01846-29b9-11ec-a11a-137a13384ff1'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'T\xbc\x0c\r\xb8\x8a\x10\xc1\xf2\x88X\x81$\x006A', cdc_time=UUID('efe01846-29b9-11ec-a11a-137a13384ff1'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'm\x9a\x00\x00\x00\x00\x00\x00\xa0(\xee>$\x00;\xe1', cdc_time=UUID('efe0877c-29b9-11ec-a56d-cc4c93edbdef'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=-inf), Row(cdc_stream_id=b'm\x9a\x00\x00\x00\x00\x00\x00\xa0(\xee>$\x00;\xe1', cdc_time=UUID('efe0877c-29b9-11ec-a56d-cc4c93edbdef'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=-inf)]
< t:2021-10-10 11:07:32,005 f:fill_db_data.py l:3278 c:sdcm.fill_db_data    p:ERROR > New_result: [Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdf3840-29b9-11ec-292d-928c9926a1af'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdf3840-29b9-11ec-292d-928c9926a1af'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=0, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=1, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=2, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'T\xbc\x0c\r\xb8\x8a\x10\xc1\xf2\x88X\x81$\x006A', cdc_time=UUID('efe01846-29b9-11ec-a11a-137a13384ff1'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'T\xbc\x0c\r\xb8\x8a\x10\xc1\xf2\x88X\x81$\x006A', cdc_time=UUID('efe01846-29b9-11ec-a11a-137a13384ff1'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'm\x9a\x00\x00\x00\x00\x00\x00\xa0(\xee>$\x00;\xe1', cdc_time=UUID('efe0877c-29b9-11ec-a56d-cc4c93edbdef'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=-inf), Row(cdc_stream_id=b'm\x9a\x00\x00\x00\x00\x00\x00\xa0(\xee>$\x00;\xe1', cdc_time=UUID('efe0877c-29b9-11ec-a56d-cc4c93edbdef'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=-inf)]

This error is expected and related to c-s/scylla python driver. the problem is that '-inf'/'inf' comparead as different objects for same collumn before and after upgrade. if you compare the rows, the are indentical.

the previous error was related to failed query on rc: Yes, but issue happened not with CDC. Next query is marked as invalid for test "empty_in_test": "DELETE FROM empty_in_test1 WHERE k1 IN ()" Failed happened, after 1st node was upgrade. Is it possible that on 4.5.rc7 this query syntax is invalid?

Was it investigated?

fgelcer commented 3 years ago

Was it investigated?

i don't think anyone has investigated it

fgelcer commented 2 years ago 
< t:2021-10-10 11:07:32,005 f:fill_db_data.py l:3278 c:sdcm.fill_db_data    p:ERROR > content was differ cdc tables content are differes
< t:2021-10-10 11:07:32,005 f:fill_db_data.py l:3278 c:sdcm.fill_db_data    p:ERROR >  Initial:[Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdf3840-29b9-11ec-292d-928c9926a1af'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdf3840-29b9-11ec-292d-928c9926a1af'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=0, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=1, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=2, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'T\xbc\x0c\r\xb8\x8a\x10\xc1\xf2\x88X\x81$\x006A', cdc_time=UUID('efe01846-29b9-11ec-a11a-137a13384ff1'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'T\xbc\x0c\r\xb8\x8a\x10\xc1\xf2\x88X\x81$\x006A', cdc_time=UUID('efe01846-29b9-11ec-a11a-137a13384ff1'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'm\x9a\x00\x00\x00\x00\x00\x00\xa0(\xee>$\x00;\xe1', cdc_time=UUID('efe0877c-29b9-11ec-a56d-cc4c93edbdef'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=-inf), Row(cdc_stream_id=b'm\x9a\x00\x00\x00\x00\x00\x00\xa0(\xee>$\x00;\xe1', cdc_time=UUID('efe0877c-29b9-11ec-a56d-cc4c93edbdef'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=-inf)]
< t:2021-10-10 11:07:32,005 f:fill_db_data.py l:3278 c:sdcm.fill_db_data    p:ERROR > New_result: [Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdf3840-29b9-11ec-292d-928c9926a1af'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdf3840-29b9-11ec-292d-928c9926a1af'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=0, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=1, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'\x9b\x14\x00\x00\x00\x00\x00\x00\x17lJiT\x00\x06\xd1', cdc_time=UUID('efdfb09a-29b9-11ec-53bb-531c1209f362'), cdc_batch_seq_no=2, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=nan), Row(cdc_stream_id=b'T\xbc\x0c\r\xb8\x8a\x10\xc1\xf2\x88X\x81$\x006A', cdc_time=UUID('efe01846-29b9-11ec-a11a-137a13384ff1'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'T\xbc\x0c\r\xb8\x8a\x10\xc1\xf2\x88X\x81$\x006A', cdc_time=UUID('efe01846-29b9-11ec-a11a-137a13384ff1'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=inf), Row(cdc_stream_id=b'm\x9a\x00\x00\x00\x00\x00\x00\xa0(\xee>$\x00;\xe1', cdc_time=UUID('efe0877c-29b9-11ec-a56d-cc4c93edbdef'), cdc_batch_seq_no=0, cdc_end_of_batch=None, cdc_operation=2, cdc_ttl=None, f=-inf), Row(cdc_stream_id=b'm\x9a\x00\x00\x00\x00\x00\x00\xa0(\xee>$\x00;\xe1', cdc_time=UUID('efe0877c-29b9-11ec-a56d-cc4c93edbdef'), cdc_batch_seq_no=1, cdc_end_of_batch=True, cdc_operation=9, cdc_ttl=None, f=-inf)]

This error is expected and related to c-s/scylla python driver. the problem is that '-inf'/'inf' comparead as different objects for same collumn before and after upgrade. if you compare the rows, the are indentical.

the previous error was related to failed query on rc: Yes, but issue happened not with CDC. Next query is marked as invalid for test "empty_in_test": "DELETE FROM empty_in_test1 WHERE k1 IN ()" Failed happened, after 1st node was upgrade. Is it possible that on 4.5.rc7 this query syntax is invalid?

Was it investigated?

so you suggest we need to change the way we compare the column before and after upgrade?

aleksbykov commented 2 years ago

Investigate issue in driver, and may be open the issue for it. and while it is not fixed, compare inf/-inf field's value differently. But origin issue not in this error message, the origin issue was some query failed after upgrade. And this should be investigated.

fgelcer commented 2 years ago

Investigate issue in driver, and may be open the issue for it. and while it is not fixed, compare inf/-inf field's value differently. But origin issue not in this error message, the origin issue was some query failed after upgrade. And this should be investigated.

is there an issue for the driver? what is the query that failed? if there is an issue with the driver, it could be the reason to the failure in the query, no?

KnifeyMoloko commented 1 year ago

Closing this as we haven't seen this issue reproduced in over a year.