[DRAFT] ci: run all integration tests

muzarski commented 4 months ago

Pre-review checklist

[ ] I have split my patch into logically separate commits.
[ ] All commit messages clearly explain what they change and why.
[ ] PR description sums up the changes and reasons why they should be introduced.
[ ] I have enabled appropriate tests in .github/workflows/build.yml in gtest_filter.
[ ] I have enabled appropriate tests in .github/workflows/cassandra.yml in gtest_filter.

roydahan commented 4 months ago

@muzarski seems like the change of scylla regressed the CI significantly (Previous one had 253 integration tests passing, this one has 457 tests but only 50 tests pass). Can you please send another PR that changes scylla version to 5.4.8 so we can compare tests results?

wprzytula commented 4 months ago

@muzarski seems like the change of scylla regressed the CI significantly (Previous one had 253 integration tests passing, this one has 457 tests but only 50 tests pass). Can you please send another PR that changes scylla version to 5.4.8 so we can compare tests results?

This is most probably due to tablets being the default now. As tests assume token ring, we need to alter them to explicitly disable tablets for created keyspaces.

roydahan commented 4 months ago

@muzarski seems like the change of scylla regressed the CI significantly (Previous one had 253 integration tests passing, this one has 457 tests but only 50 tests pass). Can you please send another PR that changes scylla version to 5.4.8 so we can compare tests results?

This is most probably due to tablets being the default now. As tests assume token ring, we need to alter them to explicitly disable tablets for created keyspaces.

This is my assumption as well, but let's verify it first with a 5.4.8 run.

muzarski commented 4 months ago

Done - changed scylla version to 5.4.8

muzarski commented 4 months ago

Again, only 50 tests passed. Something weird happens with ccm when we try to use authentication:

ccm create --scylla -n 1:0 -i 127.0.0. -v release:5.4.8 -b --pwd-auth cpp-driver_3-0-8_1-0-password_authenticator
/home/runner/work/cpp-rust-driver/cpp-rust-driver/tests/src/integration/integration.cpp:191: Failure
Failed
ccm> ccm updateconf authenticator:PasswordAuthenticator
Usage: ccm create [options] cluster_name

ccm: error: no such option: --pwd-auth

ccm> ccm start --wait-other-notice --wait-for-binary-proto --jvm_arg=-Dcassandra.superuser_setup_delay_ms=0 --jvm_arg=--skip-wait-for-gossip-to-settle=0
unknown file: Failure
C++ exception with description "Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.4/x64/bin/ccm", line 74, in <module>
    cmd.run()
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/ccmlib/cmds/cluster_cmds.py", line 676, in run
    if self.cluster.start(no_wait=self.options.no_wait,
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/ccmlib/scylla_cluster.py", line 174, in start
    started = self.start_nodes(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/ccmlib/scylla_cluster.py", line 139, in start_nodes
    p = node.start(update_pid=False, jvm_args=jvm_args,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/ccmlib/scylla_node.py", line 700, in start
    scylla_process = self._start_scylla(args=args, marks=marks, update_pid=update_pid,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/ccmlib/scylla_node.py", line 343, in _start_scylla
    self.wait_for_binary_interface(from_mark=from_mark, process=self._process_scylla, timeout=t)
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/ccmlib/node.py", line 540, in wait_for_binary_interface
    self.watch_log_for("Starting listening for CQL clients", **kwargs)
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/ccmlib/node.py", line 506, in watch_log_for
    raise RuntimeError(f"The process is dead, returncode={process.returncode}")
RuntimeError: The process is dead, returncode=2
" thrown in SetUp().

roydahan commented 4 months ago

what version of ccm are we using? is it master?

muzarski commented 4 months ago

what version of ccm are we using? is it master?

pip3 install https://github.com/scylladb/scylla-ccm/archive/master.zip <- this is the installation instruction in CI

muzarski commented 4 months ago

I'll try to disable auth tests and see what happens.

roydahan commented 4 months ago

I don't understand where this "pwd-auth" is coming from. I don't see it in the args of the command being used for scylla nor in ccm. @fruch any idea?

Lorak-mmk commented 4 months ago

I can see this option present in original ccm: https://github.com/riptano/ccm/blob/3d84863d766a95fe1e37d779d12de2358a35c923/ccmlib/cmds/cluster_cmds.py#L95 but not in our fork.

Lorak-mmk commented 4 months ago

It was added in this commit, in 2016: https://github.com/riptano/ccm/commit/8dc02c4182185303d0e1fe24aee1fc0f57dc405b When was our ccm forked? Did we merge with upstream since then?

wprzytula commented 4 months ago

It was added in this commit, in 2016: riptano/ccm@8dc02c4 When was our ccm forked? Did we merge with upstream since then?

@fruch Do you know this by any chance?

Lorak-mmk commented 4 months ago

The first commit in git history that is not present in riptano/ccm but present in scylla-ccm is https://github.com/scylladb/scylla-ccm/commit/eb6029181fa55ccc44f48752baff280c94047d5b That would mean we forked in June 2015 - and we didn't merge upstream improvements.

Btw was Scylla originally called Urchin???

Lorak-mmk commented 4 months ago

It looks like this. First commit that mentions Scylla: https://github.com/scylladb/scylla-ccm/commit/0966214c56c30aae98eecb9fbe8bfd92f5a69cba Later "urchin" was slowly replaced by "scylla", across multiple commits: https://github.com/scylladb/scylla-ccm/commit/4dcd81c0723adc4a653c3f1d9599e8519947a75b https://github.com/scylladb/scylla-ccm/commit/e804e298b68a3e4919ca533ec27d53c2e48fcb98 https://github.com/scylladb/scylla-ccm/commit/493c4f0a32a5a406a84b08cb482e2db13b969317 https://github.com/scylladb/scylla-ccm/commit/85c992ad19cd644bd071c5c8163e48d73ce0a764 and many others

muzarski commented 4 months ago

The stats from the run excluding auth tests for scylla 5.4.8:

[==========] 451 tests from 74 test cases ran. (1158673 ms total)
[  PASSED  ] 312 tests.
[  FAILED  ] 139 tests, listed below:

This means that 145 (139 + 6 x auth) tests still fail.

fruch commented 4 months ago

It was added in this commit, in 2016: riptano/ccm@8dc02c4 When was our ccm forked? Did we merge with upstream since then?

@fruch Do you know this by any chance?

I know it has nothing todo with scylla what so ever, here the commit introduced it in the cpp-driver: https://github.com/scylladb/cpp-driver/commit/11e61c5bb5575ff9dadbb1282a4414b09f4657d2

any test that depend on whatever it would be, isn't relevant to scylla, and should be disabled (or removed)

fruch commented 4 months ago

The first commit in git history that is not present in riptano/ccm but present in scylla-ccm is scylladb/scylla-ccm@eb60291 That would mean we forked in June 2015 - and we didn't merge upstream improvements.

we are not syncing ccm, since it was forked. we do port relevant things if they are needed

Btw was Scylla originally called Urchin???

it was a long long time ago

roydahan commented 4 months ago

was Scylla originally called Urchin???

Yes, before it had an official name :)

muzarski commented 2 months ago

Rebased on master, to see current status of ITs with new version of CI.

cc: @dkropachev

scylladb / cpp-rust-driver

[DRAFT] ci: run all integration tests #139

Pre-review checklist