IBM / ibm-spectrum-scale-install-infra

Spectrum Scale Installation and Configuration
Apache License 2.0
64 stars 67 forks source link

explore error message suppression and only output 'mm' command failures #34

Open mrolyat opened 4 years ago

mrolyat commented 4 years ago

Looking to improve error messaging usability and problem determination by suppressing extra error messages and only posting the 'mm' command output.

Example in current Ansible run:


changed: [node-vm1] => (item={'diff': [], 'dest': '/var/tmp/StanzaFile.new.gpfs300', 'src': '/root/.ansible/tmp/ansible-tmp-1585001588.5356421-65289086087216/source', 'md5sum': '25534cf1001db2ac025b49d2c05a1332', 'checksum': 'bb2265de9b4dc8d6abbceadf832aea081427a9dd', 'changed': True, 'uid': 0, 'gid': 0, 'owner': 'root', 'group': 'root', 'mode': '0644', 'state': 'file', 'secontext': 'unconfined_u:object_r:admin_home_t:s0', 'size': 262, 'invocation': {'module_args': {'src': '/root/.ansible/tmp/ansible-tmp-1585001588.5356421-65289086087216/source', 'dest': '/var/tmp/StanzaFile.new.gpfs300', 'mode': None, 'follow': False, '_original_basename': 'StanzaFile.j2', 'checksum': 'bb2265de9b4dc8d6abbceadf832aea081427a9dd', 'backup': False, 'force': True, 'content': None, 'validate': None, 'directory_mode': None, 'remote_src': None, 'local_follow': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None, 'regexp': None, 'delimiter': None, 'unsafe_writes': None}}, 'failed': False, 'item': 'gpfs300', 'ansible_loop_var': 'item'})
failed: [node-vm1] (item={'diff': {'before': {'path': '/var/tmp/StanzaFile.new.gpfs2000'}, 'after': {'path': '/var/tmp/StanzaFile.new.gpfs2000'}}, 'path': '/var/tmp/StanzaFile.new.gpfs2000', 'changed': False, 'uid': 0, 'gid': 0, 'owner': 'root', 'group': 'root', 'mode': '0644', 'state': 'file', 'secontext': 'unconfined_u:object_r:admin_home_t:s0', 'size': 131, 'invocation': {'module_args': {'mode': None, 'follow': False, 'dest': '/var/tmp/StanzaFile.new.gpfs2000', '_original_basename': 'StanzaFile.j2', 'recurse': False, 'state': 'file', 'path': '/var/tmp/StanzaFile.new.gpfs2000', 'force': False, 'modification_time_format': '%Y%m%d%H%M.%S', 'access_time_format': '%Y%m%d%H%M.%S', '_diff_peek': None, 'src': None, 'modification_time': None, 'access_time': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None, 'content': None, 'backup': None, 'remote_src': None, 'regexp': None, 'delimiter': None, 'directory_mode': None, 'unsafe_writes': None}}, 'checksum': '7c5d84d86a6b23e28f29975eccd970850559265d', 'dest': '/var/tmp/StanzaFile.new.gpfs2000', 'failed': False, 'item': 'gpfs2000', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": true, "cmd": ["/usr/lpp/mmfs/bin/mmcrfs", "gpfs2000", "-F", "/var/tmp/StanzaFile.new.gpfs2000", "-B", "4M", "-m", "2", "-r", "2", "-n", "16", "-A", "yes", "-T", "/ibm/gpfs2000"], "delta": "0:00:02.411239", "end": "2020-03-23 15:13:26.068223", "item": {"ansible_loop_var": "item", "changed": false, "checksum": "7c5d84d86a6b23e28f29975eccd970850559265d", "dest": "/var/tmp/StanzaFile.new.gpfs2000", "diff": {"after": {"path": "/var/tmp/StanzaFile.new.gpfs2000"}, "before": {"path": "/var/tmp/StanzaFile.new.gpfs2000"}}, "failed": false, "gid": 0, "group": "root", "invocation": {"module_args": {"_diff_peek": null, "_original_basename": "StanzaFile.j2", "access_time": null, "access_time_format": "%Y%m%d%H%M.%S", "attributes": null, "backup": null, "content": null, "delimiter": null, "dest": "/var/tmp/StanzaFile.new.gpfs2000", "directory_mode": null, "follow": false, "force": false, "group": null, "mode": null, "modification_time": null, "modification_time_format": "%Y%m%d%H%M.%S", "owner": null, "path": "/var/tmp/StanzaFile.new.gpfs2000", "recurse": false, "regexp": null, "remote_src": null, "selevel": null, "serole": null, "setype": null, "seuser": null, "src": null, "state": "file", "unsafe_writes": null}}, "item": "gpfs2000", "mode": "0644", "owner": "root", "path": "/var/tmp/StanzaFile.new.gpfs2000", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 131, "state": "file", "uid": 0}, "msg": "non-zero return code", "rc": 22, "start": "2020-03-23 15:13:23.656984", "stderr": "Incompatible parameters: Unable to create file system.\nChange one or more of the following as suggested and try again:\n    increase the number of failure groups\n    decrease the value for -r\nmmcrfs: tscrfs failed.  Cannot create gpfs2000\nmmcrfs: Command failed. Examine previous error messages to determine cause.", "stderr_lines": ["Incompatible parameters: Unable to create file system.", "Change one or more of the following as suggested and try again:", "    increase the number of failure groups", "    decrease the value for -r", "mmcrfs: tscrfs failed.  Cannot create gpfs2000", "mmcrfs: Command failed. Examine previous error messages to determine cause."], "stdout": "\nThe following disks of gpfs2000 will be formatted on node node-vm2:\n    nsd_2001: size 76800 MB\nFormatting file system ...\nDisks up to size 785.99 GB can be added to storage pool system.", "stdout_lines": ["", "The following disks of gpfs2000 will be formatted on node node-vm2:", "    nsd_2001: size 76800 MB", "Formatting file system ...", "Disks up to size 785.99 GB can be added to storage pool system."]}NO MORE HOSTS LEFT ****************************************************************************************************************************************PLAY RECAP ************************************************************************************************************************************************
node-vm1               : ok=78   changed=4    unreachable=0    failed=1    skipped=54   rescued=0    ignored=0
node-vm2               : ok=56   changed=0    unreachable=0    failed=0    skipped=32   rescued=0    ignored=0
node-vm3               : ok=56   changed=0    unreachable=0    failed=0    skipped=32   rescued=0    ignored=0
node-vm4               : ok=56   changed=0    unreachable=0    failed=0    skipped=32   rescued=0    ignored=0

Example of what just the 'mm' command failure output would be:
```[root@node-vm1 group_vars]# /usr/lpp/mmfs/bin/mmcrfs gpfs2000 -F /var/tmp/StanzaFile.new.gpfs2000 -B 4M -m 2 -r 2 -n 16 -A yes -T /ibm/gpfs2000The following disks of gpfs2000 will be formatted on node node-vm2:
    nsd_2001: size 76800 MB
Formatting file system ...
Disks up to size 785.99 GB can be added to storage pool system.
Incompatible parameters: Unable to create file system.
Change one or more of the following as suggested and try again:
    increase the number of failure groups
    decrease the value for -r
mmcrfs: tscrfs failed.  Cannot create gpfs2000
mmcrfs: Command failed. Examine previous error messages to determine cause.
qiuxinyidian commented 3 months ago

when i create gpfs with spectrumscale ,got this error, how to fix it ?

[ INFO  ] failed: [node101 -> node101] (item={u'changed': True, u'uid': 0, u'dest': u'/var/mmfs/tmp/StanzaFile.new.gpfs_test', u'owner': u'root', u'diff': [], u'size': 336, u'src': u'/root/.ansible/tmp/ansible-tmp-1719455659.77-25630-66601711026952/source', u'ansible_loop_var': u'item', u'group': u'root', u'failed': False, u'checksum': u'7b2bd644c284f21382697e42f330b92c923914c2', u'md5sum': u'b9d3cb905d408898a776c9bfd7483e1d', u'item': u'gpfs_test', u'state': u'file', u'gid': 0, u'secontext': u'system_u:object_r:var_t:s0', u'mode': u'0644', u'invocation': {u'module_args': {u'directory_mode': None, u'force': True, u'remote_src': None, u'dest': u'/var/mmfs/tmp/StanzaFile.new.gpfs_test', u'selevel': None, u'_original_basename': u'StanzaFile_fs.j2', u'delimiter': None, u'regexp': None, u'owner': None, u'follow': False, u'validate': None, u'local_follow': None, u'src': u'/root/.ansible/tmp/ansible-tmp-1719455659.77-25630-66601711026952/source', u'group': None, u'unsafe_writes': False, u'checksum': u'7b2bd644c284f21382697e42f330b92c923914c2', u'seuser': None, u'serole': None, u'content': None, u'setype': None, u'mode': None, u'attributes': None, u'backup': False}}}) => {"ansible_loop_var": "item", "changed": true, "cmd": ["/usr/lpp/mmfs/bin/mmcrfs", "gpfs_test", "-F", "/var/mmfs/tmp/StanzaFile.new.gpfs_test", "-B", "4M", "-M", "3", "-m", "2", "-R", "3", "-r", "2", "-n", "32", "-A", "yes", "-T", "/gpfs"], "delta": "0:00:00.619736", "end": "2024-06-26 22:34:20.751765", "item": {"ansible_loop_var": "item", "changed": true, "checksum": "7b2bd644c284f21382697e42f330b92c923914c2", "dest": "/var/mmfs/tmp/StanzaFile.new.gpfs_test", "diff": [], "failed": false, "gid": 0, "group": "root", "invocation": {"module_args": {"_original_basename": "StanzaFile_fs.j2", "attributes": null, "backup": false, "checksum": "7b2bd644c284f21382697e42f330b92c923914c2", "content": null, "delimiter": null, "dest": "/var/mmfs/tmp/StanzaFile.new.gpfs_test", "directory_mode": null, "follow": false, "force": true, "group": null, "local_follow": null, "mode": null, "owner": null, "regexp": null, "remote_src": null, "selevel": null, "serole": null, "setype": null, "seuser": null, "src": "/root/.ansible/tmp/ansible-tmp-1719455659.77-25630-66601711026952/source", "unsafe_writes": false, "validate": null}}, "item": "gpfs_test", "md5sum": "b9d3cb905d408898a776c9bfd7483e1d", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:var_t:s0", "size": 336, "src": "/root/.ansible/tmp/ansible-tmp-1719455659.77-25630-66601711026952/source", "state": "file", "uid": 0}, "msg": "non-zero return code", "rc": 22, "start": "2024-06-26 22:34:20.132029", "stderr": "Incompatible parameters: Unable to create file system.\nChange one or more of the following as suggested and try again:\n    increase the number of failure groups\n    decrease the value for -r\nmmcrfs: tscrfs failed.  Cannot create gpfs_test\nmmcrfs: Command failed. Examine previous error messages to determine cause.", "stderr_lines": ["Incompatible parameters: Unable to create file system.", "Change one or more of the following as suggested and try again:", "    increase the number of failure groups", "    decrease the value for -r", "mmcrfs: tscrfs failed.  Cannot create gpfs_test", "mmcrfs: Command failed. Examine previous error messages to determine cause."], "stdout": "\nThe following disks of gpfs_test will be formatted on node node101:\n    nsd1: size 102400 MB\n    nsd2: size 102400 MB\n    nsd3: size 102400 MB\nFormatting file system ...\nDisks up to size 1.54 TB can be added to storage pool system.", "stdout_lines": ["", "The following disks of gpfs_test will be formatted on node node101:", "    nsd1: size 102400 MB", "    nsd2: size 102400 MB", "    nsd3: size 102400 MB", "Formatting file system ...", "Disks up to size 1.54 TB can be added to storage pool system."]}
rajan-mis commented 3 months ago

Looks like some issue with create filesystem parameter value.

"stderr_lines": ["Incompatible parameters: Unable to create file system.", "Change one or more of the following as suggested and try again:", " increase the number of failure groups", " decrease the value for -r", "mmcrfs: tscrfs failed. Cannot create gpfs_test",

qiuxinyidian commented 3 months ago

yes,don't know how to fix

rajan-mis commented 3 months ago

I see in your error, there are suggestion , please validate your stanza file

["Incompatible parameters: Unable to create file system.", "Change one or more of the following as suggested and try again:", " increase the number of failure groups", " decrease the value for -r"

KC documentation link: https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=reference-mmcrfs-command