search

scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
57 stars 95 forks source link

TruncateMonkeyLargePartition fails installing scylla-bench in branch 2021.1 #5604

Open yarongilor opened 1 year ago

yarongilor commented 1 year ago

Installation details Kernel version: 5.15.0-1026-aws Scylla version (or git commit hash): 2021.1.17-0.20221221.5318a7fec Cluster size: 3 nodes (i3.large) Scylla running with shards number (live nodes): gemini-with-nemesis-3h-normal-2021--db-node-38c9351a-1 (44.195.84.62 | 10.0.3.204): 2 shards gemini-with-nemesis-3h-normal-2021--db-node-38c9351a-2 (100.27.37.199 | 10.0.1.124): 2 shards gemini-with-nemesis-3h-normal-2021--db-node-38c9351a-4 (3.238.16.37 | 10.0.3.21): 2 shards Scylla running with shards number (terminated nodes): gemini-with-nemesis-3h-normal-2021--db-node-38c9351a-3 (3.235.93.186 | 10.0.0.229): 2 shards OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-0a473e61cce73366a (aws: us-east-1)

Gemini command: /$HOME/gemini -d --duration 3h --warmup 30m -c 50 -m mixed -f --non-interactive --cql-features normal --max-mutation-retries 5 --max-mutation-retries-backoff 500ms --async-objects-stabilization-attempts 5 --async-objects-stabilization-backoff 500ms --replication-strategy "{'class': 'SimpleStrategy', 'replication_factor': '3'}" --oracle-replication-strategy "{'class': 'SimpleStrategy', 'replication_factor': '1'}" --test-cluster=10.0.3.204,10.0.1.124,10.0.0.229 --outfile /home/centos/gemini_result_656bf6f4-324f-4fdd-be6a-8ccb4dcf241d.log --seed 21 --oracle-cluster=10.0.0.222 Gemini version: 1.7.5

Test: gemini-3h-with-nemesis-test Test name: gemini_test.GeminiTest.test_load_random_with_nemesis Test config file(s):

Issue description

====================================

2022-12-26 14:37:45.037: (DisruptionEvent Severity.ERROR): type=TruncateMonkeyLargePartition subtype=end node=Node gemini-with-nemesis-3h-normal-2021--db-node-38c9351a-3 [3.235.93.186 | 10.0.0.229] (seed: False) duration=40 error=Encountered a bad command exit code!
Command: 'sudo bash -cxe "rm -rf /usr/local/go\ncurl -LO [https://storage.googleapis.com/golang/go1.16.3.linux-amd64.tar.gz\ntar](https://storage.googleapis.com/golang/go1.16.3.linux-amd64.tar.gz%5Cntar) -C /usr/local -xvzf go1.16.3.linux-amd64.tar.gz\necho \'export GOPATH=$HOME/go\' >> $HOME/.bash_profile\necho \'export PATH=$PATH:/usr/local/go/bin\' >> $HOME/.bash_profile\nsource $HOME/.bash_profile\nGO111MODULE=on go get -v [github.com/scylladb/scylla-bench@v0.1.14\n](http://github.com/scylladb/scylla-bench@v0.1.14%5Cn)"'
Exit code: 1
Stdout:
go/test/uintptrescapes.go
go/test/uintptrescapes2.go
go/test/uintptrescapes3.go
go/test/undef.go
go/test/utf.go
go/test/varerr.go
go/test/varinit.go
go/test/winbatch.go
go/test/writebarrier.go
go/test/zerodivide.go
Stderr:
++ export PATH=/usr/local/bin:/usr/bin:/usr/local/go/bin:/home/centos/go/bin:/usr/local/go/bin
++ PATH=/usr/local/bin:/usr/bin:/usr/local/go/bin:/home/centos/go/bin:/usr/local/go/bin
++ export GOPATH=/home/centos/go
++ GOPATH=/home/centos/go
++ export PATH=/usr/local/bin:/usr/bin:/usr/local/go/bin:/home/centos/go/bin:/usr/local/go/bin
++ PATH=/usr/local/bin:/usr/bin:/usr/local/go/bin:/home/centos/go/bin:/usr/local/go/bin
+ GO111MODULE=on
+ go get -v [github.com/scylladb/scylla-bench@v0.1.14](http://github.com/scylladb/scylla-bench@v0.1.14)
[github.com/scylladb/scylla-bench](http://github.com/scylladb/scylla-bench) imports
[github.com/gocql/gocql/scyllacloud](http://github.com/gocql/gocql/scyllacloud): cannot find module providing package [github.com/gocql/gocql/scyllacloud](http://github.com/gocql/gocql/scyllacloud)
Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2995, in wrapper
result = method(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 3895, in disrupt
self.call_random_disrupt_method(disrupt_methods=self.disrupt_methods_list)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 1160, in call_random_disrupt_method
self.execute_disrupt_method(disrupt_method)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 1167, in execute_disrupt_method
disrupt_method()
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 1260, in disrupt_truncate_large_partition
bench_thread = self.tester.run_stress_thread(
File "/home/ubuntu/scylla-cluster-tests/sdcm/tester.py", line 1161, in run_stress_thread
return self.run_stress_thread_bench(**params)
File "/home/ubuntu/scylla-cluster-tests/sdcm/tester.py", line 1221, in run_stress_thread_bench
bench_thread.run()
File "/home/ubuntu/scylla-cluster-tests/sdcm/scylla_bench_thread.py", line 210, in run
loader.install_scylla_bench()
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 61, in inner
return func(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 588, in install_scylla_bench
self.remoter.sudo(shell_script_cmd(f"""\
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/base.py", line 121, in sudo
return self.run(cmd=cmd,
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 604, in run
result = _run()
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 61, in inner
return func(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 596, in _run
return self._run_execute(cmd, timeout, ignore_status, verbose, new_session, watchers)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 537, in _run_execute
result = connection.run(**command_kwargs)
File "<decorator-gen-3>", line 2, in run
File "/usr/local/lib/python3.9/site-packages/fabric/connection.py", line 30, in opens
return method(self, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/fabric/connection.py", line 721, in run
return self._run(self._remote_runner(), command, **kwargs)
File "/usr/local/lib/python3.9/site-packages/invoke/context.py", line 101, in _run
return runner.run(command, **kwargs)
File "/usr/local/lib/python3.9/site-packages/invoke/runners.py", line 291, in run
return self._run_body(command, **kwargs)
File "/usr/local/lib/python3.9/site-packages/invoke/runners.py", line 442, in _run_body
raise UnexpectedExit(result)
invoke.exceptions.UnexpectedExit: Encountered a bad command exit code!
Command: 'sudo bash -cxe "rm -rf /usr/local/go\ncurl -LO [https://storage.googleapis.com/golang/go1.16.3.linux-amd64.tar.gz\ntar](https://storage.googleapis.com/golang/go1.16.3.linux-amd64.tar.gz%5Cntar) -C /usr/local -xvzf go1.16.3.linux-amd64.tar.gz\necho \'export GOPATH=$HOME/go\' >> $HOME/.bash_profile\necho \'export PATH=$PATH:/usr/local/go/bin\' >> $HOME/.bash_profile\nsource $HOME/.bash_profile\nGO111MODULE=on go get -v [github.com/scylladb/scylla-bench@v0.1.14\n](http://github.com/scylladb/scylla-bench@v0.1.14%5Cn)"'
Exit code: 1
Stdout:
go/test/uintptrescapes.go
go/test/uintptrescapes2.go
go/test/uintptrescapes3.go
go/test/undef.go
go/test/utf.go
go/test/varerr.go
go/test/varinit.go
go/test/winbatch.go
go/test/writebarrier.go
go/test/zerodivide.go
Stderr:
++ export PATH=/usr/local/bin:/usr/bin:/usr/local/go/bin:/home/centos/go/bin:/usr/local/go/bin
++ PATH=/usr/local/bin:/usr/bin:/usr/local/go/bin:/home/centos/go/bin:/usr/local/go/bin
++ export GOPATH=/home/centos/go
++ GOPATH=/home/centos/go
++ export PATH=/usr/local/bin:/usr/bin:/usr/local/go/bin:/home/centos/go/bin:/usr/local/go/bin
++ PATH=/usr/local/bin:/usr/bin:/usr/local/go/bin:/home/centos/go/bin:/usr/local/go/bin
+ GO111MODULE=on
+ go get -v [github.com/scylladb/scylla-bench@v0.1.14](http://github.com/scylladb/scylla-bench@v0.1.14)
[github.com/scylladb/scylla-bench](http://github.com/scylladb/scylla-bench) imports
[github.com/gocql/gocql/scyllacloud](http://github.com/gocql/gocql/scyllacloud): cannot find module providing package [github.com/gocql/gocql/scyllacloud](http://github.com/gocql/gocql/scyllacloud)

====================================

Restore Monitor Stack command: $ hydra investigate show-monitor 38c9351a-b152-40ca-b0d0-821c4fcb6530 Restore monitor on AWS instance using Jenkins job Show all stored logs command: $ hydra investigate show-logs 38c9351a-b152-40ca-b0d0-821c4fcb6530

Test id: 38c9351a-b152-40ca-b0d0-821c4fcb6530

Logs: grafana - [https://cloudius-jenkins-test.s3.amazonaws.com/38c9351a-b152-40ca-b0d0-821c4fcb6530/20221226_174321/grafana-screenshot-gemini-3h-with-nemesis-test-scylla-per-server-metrics-nemesis-20221226_174654-gemini-with-nemesis-3h-normal-2021--monitor-node-38c9351a-1.png](https://cloudius-jenkins-test.s3.amazonaws.com/38c9351a-b152-40ca-b0d0-821c4fcb6530/20221226_174321/grafana-screenshot-gemini-3h-with-nemesis-test-scylla-per-server-metrics-nemesis-20221226_174654-gemini-with-nemesis-3h-normal-2021--monitor-node-38c9351a-1.png) grafana - [https://cloudius-jenkins-test.s3.amazonaws.com/38c9351a-b152-40ca-b0d0-821c4fcb6530/20221226_174321/grafana-screenshot-overview-20221226_174321-gemini-with-nemesis-3h-normal-2021--monitor-node-38c9351a-1.png](https://cloudius-jenkins-test.s3.amazonaws.com/38c9351a-b152-40ca-b0d0-821c4fcb6530/20221226_174321/grafana-screenshot-overview-20221226_174321-gemini-with-nemesis-3h-normal-2021--monitor-node-38c9351a-1.png) db-cluster - [https://cloudius-jenkins-test.s3.amazonaws.com/38c9351a-b152-40ca-b0d0-821c4fcb6530/20221226_175125/db-cluster-38c9351a.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/38c9351a-b152-40ca-b0d0-821c4fcb6530/20221226_175125/db-cluster-38c9351a.tar.gz) loader-set - [https://cloudius-jenkins-test.s3.amazonaws.com/38c9351a-b152-40ca-b0d0-821c4fcb6530/20221226_175125/loader-set-38c9351a.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/38c9351a-b152-40ca-b0d0-821c4fcb6530/20221226_175125/loader-set-38c9351a.tar.gz) monitor-set - [https://cloudius-jenkins-test.s3.amazonaws.com/38c9351a-b152-40ca-b0d0-821c4fcb6530/20221226_175125/monitor-set-38c9351a.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/38c9351a-b152-40ca-b0d0-821c4fcb6530/20221226_175125/monitor-set-38c9351a.tar.gz) sct-runner - [https://cloudius-jenkins-test.s3.amazonaws.com/38c9351a-b152-40ca-b0d0-821c4fcb6530/20221226_175125/sct-runner-38c9351a.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/38c9351a-b152-40ca-b0d0-821c4fcb6530/20221226_175125/sct-runner-38c9351a.tar.gz)

Jenkins job URL

yarongilor commented 1 year ago

Related to https://github.com/scylladb/scylla-cluster-tests/pull/5584