search

scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
57 stars 95 forks source link

[2021.1.rc2] TheCommand: 'sudo apt install -y stress-ng' #3511

Closed Orenef11 closed 1 year ago

Orenef11 commented 3 years ago

At 2021-05-06 18:24:38.168:

(DisruptionEvent Severity.ERROR): type=MemoryStress subtype=end node=Node lwt-longevity-multi-dc-24h-2021-1-db-node-b2b540b6-5 [35.167.170.190 | 10.0.2.137] (seed: False) duration=326 error=Encountered a bad command exit code!

Command: 'sudo apt install -y stress-ng'

Exit code: 100

Stdout:

Reading package lists...
Building dependency tree...
Reading state information...

No apt package "stress-ng", but there is a snap with that name.
Try "snap install stress-ng"

Stderr:

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

E: Unable to locate package stress-ng

Traceback (most recent call last):
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2955, in wrapper
    result = method(*args, **kwargs)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 3250, in disrupt
    self.call_random_disrupt_method()
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 1121, in call_random_disrupt_method
    self.execute_disrupt_method(disrupt_method)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 1128, in execute_disrupt_method
    disrupt_method()
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2799, in disrupt_memory_stress
    self.target_node.remoter.sudo('apt install -y stress-ng')
  File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/base.py", line 121, in sudo
    return self.run(cmd=cmd,
  File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 604, in run
    result = _run()
  File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 61, in inner
    return func(*args, **kwargs)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 596, in _run
    return self._run_execute(cmd, timeout, ignore_status, verbose, new_session, watchers)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_base.py", line 537, in _run_execute
    result = connection.run(**command_kwargs)
  File "", line 2, in run
  File "/usr/local/lib/python3.9/site-packages/fabric/connection.py", line 30, in opens
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/fabric/connection.py", line 721, in run
    return self._run(self._remote_runner(), command, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/invoke/context.py", line 101, in _run
    return runner.run(command, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/invoke/runners.py", line 291, in run
    return self._run_body(command, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/invoke/runners.py", line 442, in _run_body
    raise UnexpectedExit(result)
invoke.exceptions.UnexpectedExit: Encountered a bad command exit code!

Command: 'sudo apt install -y stress-ng'

Exit code: 100

Stdout:

Reading package lists...
Building dependency tree...
Reading state information...

No apt package "stress-ng", but there is a snap with that name.
Try "snap install stress-ng"

Stderr:

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

E: Unable to locate package stress-ng

test-id: b2b540b6-2689-4f5c-a81f-43f248e49108 monitor: http://34.241.224.63:3000/d/95EX8WjMz/longevity-lwt-24h-multidc-test-scylla-per-server-metrics-nemesis-master?orgId=1&from=1620325258652&to=1620325748248

Orenef11 commented 3 years ago

FYI @roydahan

amoskong commented 3 years ago

@Orenef11 does the issue occur all the time? Or you just touched once?

In my manual test, It can be successfully installed in Ubuntu AMI (ubuntu20.04).

scyllaadm@:~$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
scyllaadm@:~$ apt search stress-ng
Sorting... Done
Full Text Search... Done
s-tui/focal 1.0.0-1 all
  terminal UI for monitoring your computer

stress-ng/focal 0.11.07-1 amd64
  tool to load and stress a computer

scyllaadm@:~$ sudo apt install stress-ng
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libipsec-mb0 libjudydebian1 libsctp1
Suggested packages:
  lksctp-tools
The following NEW packages will be installed:
  libipsec-mb0 libjudydebian1 libsctp1 stress-ng
0 upgraded, 4 newly installed, 0 to remove and 11 not upgraded.
Need to get 2291 kB of archives.
After this operation, 19.8 MB of additional disk space will be used.
Do you want to continue? [Y/n]  
Get:1 http://eu-north-1.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 libipsec-mb0 amd64 0.53-1 [491 kB]
Get:2 http://eu-north-1.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 libjudydebian1 amd64 1.0.5-5 [94.6 kB]
Get:3 http://eu-north-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 libsctp1 amd64 1.0.18+dfsg-1 [7876 B]
Get:4 http://eu-north-1.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 stress-ng amd64 0.11.07-1 [1697 kB]
Fetched 2291 kB in 1s (3606 kB/s)   
Selecting previously unselected package libipsec-mb0.
(Reading database ... 91601 files and directories currently installed.)
Preparing to unpack .../libipsec-mb0_0.53-1_amd64.deb ...
Unpacking libipsec-mb0 (0.53-1) ...
Selecting previously unselected package libjudydebian1.
Preparing to unpack .../libjudydebian1_1.0.5-5_amd64.deb ...
Unpacking libjudydebian1 (1.0.5-5) ...
Selecting previously unselected package libsctp1:amd64.
Preparing to unpack .../libsctp1_1.0.18+dfsg-1_amd64.deb ...
Unpacking libsctp1:amd64 (1.0.18+dfsg-1) ...
Selecting previously unselected package stress-ng.
Preparing to unpack .../stress-ng_0.11.07-1_amd64.deb ...
Unpacking stress-ng (0.11.07-1) ...
Setting up libjudydebian1 (1.0.5-5) ...
Setting up libipsec-mb0 (0.53-1) ...
Setting up libsctp1:amd64 (1.0.18+dfsg-1) ...
Setting up stress-ng (0.11.07-1) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
scyllaadm@:~$
amoskong commented 3 years ago

try to fix by https://github.com/scylladb/scylla-cluster-tests/pull/3513

amoskong commented 3 years ago

Command: 'sudo apt install -y stress-ng' Exit code: 100

Stdout:

Reading package lists... Building dependency tree... Reading state information...

No apt package "stress-ng", but there is a snap with that name. Try "snap install stress-ng"

Stderr:

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

E: Unable to locate package stress-ng



test-id: `b2b540b6-2689-4f5c-a81f-43f248e49108`

There is no sct log of above test id:

$ hydra investigate show-logs b2b540b6-2689-4f5c-a81f-43f248e49108
New directory created: /home/amos/sct-results/20210510-052021-356308-investigate-show-logs
+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                           Log links for testrun with test id b2b540b6-2689-4f5c-a81f-43f248e49108                                            |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------+
| Date            | Log type    | Link                                                                                                                         |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------+
| 20210509_091419 | db-cluster  | https://cloudius-jenkins-test.s3.amazonaws.com/b2b540b6-2689-4f5c-a81f-43f248e49108/20210509_091419/db-cluster-b2b540b6.zip  |
| 20210509_091419 | loader-set  | https://cloudius-jenkins-test.s3.amazonaws.com/b2b540b6-2689-4f5c-a81f-43f248e49108/20210509_091419/loader-set-b2b540b6.zip  |
| 20210509_091419 | monitor-set | https://cloudius-jenkins-test.s3.amazonaws.com/b2b540b6-2689-4f5c-a81f-43f248e49108/20210509_091419/monitor-set-b2b540b6.zip |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------+

monitor: http://34.241.224.63:3000/d/95EX8WjMz/longevity-lwt-24h-multidc-test-scylla-per-server-metrics-nemesis-master?orgId=1&from=1620325258652&to=1620325748248

Orenef11 commented 3 years ago

@amoskong This is the first time I saw it so I opened a bug for it. Yes, the logic of the SCT runner was not saved, there was a problem collecting the logs.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 2 years with no activity. Remove stale label or comment or this will be closed in 2 days.

fgelcer commented 1 year ago

we now use docker based stress images