scylladb / scylla-stress-orchestrator

Apache License 2.0
5 stars 9 forks source link

Spurious failure on load balancer due to cpupower #36

Open pveentjer opened 3 years ago

pveentjer commented 3 years ago
07:04 INFO  13.59.109.152       Installing cassandra-stress: done
Traceback (most recent call last):
  File "/home/pveentjer/.local/lib/python3.10/site-packages/scyllaso/util.py", line 49, in run
    super().run()
  File "/usr/lib64/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pveentjer/.local/lib/python3.10/site-packages/scyllaso/cs.py", line 32, in __install
    ssh.set_governor("performance")
  File "/home/pveentjer/.local/lib/python3.10/site-packages/scyllaso/ssh.py", line 286, in set_governor
    self.exec(f"""
  File "/home/pveentjer/.local/lib/python3.10/site-packages/scyllaso/ssh.py", line 190, in exec
    raise Exception(f"Failed to execute [{cmd_list}], exitcode={exitcode}")
Exception: Failed to execute [['ssh', '-S', '/tmp/ubuntu@3.22.186.220.socket', '-i', 'key', '-o', 'StrictHostKeyChecking=no', '-o', 'ConnectTimeout=60', 'ubuntu@3.22.186.220', '\n            set -e\n    \n            if ! hash cpupower 2>/dev/null; then\n                echo "Installing cpupower"\n                if hash apt-get 2>/dev/null; then\n                    sudo apt-get -y -qq update\n                    version=$(uname -r)\n                    sudo apt-get -y -qq install linux-tools-$version\n                    echo "apt-get found"                           \n                elif hash yum 2>/dev/null; then\n                    sudo yum -y -q install kernel-tools                \n                else\n                    echo "Cannot install performance: yum/apt not found"                \n                    exit 1\n                fi\n            fi\n            echo "=========== frequency info before change ==========="\n            sudo cpupower frequency-info\n            echo "===================================================="\n            frequencyinfo=$(sudo cpupower frequency-info)                  \n            if [[ $frequencyinfo =~ "performance" ]]; then\n                sudo cpupower frequency-set -g performance  \n                echo "=========== frequency info after change ==========="\n                sudo cpupower frequency-info\n                echo "===================================================="                      \n            else\n                echo "Skipping set governor, [performance] is not supported"\n            fi             \n        ']], exitcode=None

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/smalldata1/tests/1_PB_test/./benchmark.py", line 33, in <module>
    cs.install()
  File "/home/pveentjer/.local/lib/python3.10/site-packages/scyllaso/cs.py", line 68, in install
    run_parallel(self.__install, [(ip,) for ip in self.load_ips])
  File "/home/pveentjer/.local/lib/python3.10/site-packages/scyllaso/util.py", line 69, in run_parallel
    raise Exception() from thread.exception
Exception
pveentjer commented 3 years ago

I believe this is the cause:

08:22:43 WARN  ....      E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 23359 (unattended-upgr)
08:22:43 WARN  ....       E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?