search

StackStorm / st2ci

New and improved continuous integration actions and workflows
Apache License 2.0
3 stars 9 forks source link

Sensu client delete failing on instance clean up workflow #97

Open Mierdin opened 7 years ago

Mierdin commented 7 years ago

Whenever a CI workflow fails for some reason, the sensu client delete task in the cleanup workflow always fails as well.

oswaltm@st2cicd007:~$ st2 execution get 597127e67f769051b0bbf95f
id: 597127e67f769051b0bbf95f
action.ref: st2ci.st2_pkg_e2e_test
<truncated>
+-----------------------------+--------------------------+----------------------------+----------------------------+------------------------------+
| id                          | status                   | task                       | action                     | start_timestamp              |
+-----------------------------+--------------------------+----------------------------+----------------------------+------------------------------+
|   597127e77f769051b0bbf967  | succeeded (1s elapsed)   | get_webui_server           | core.noop                  | Thu, 20 Jul 2017 22:00:07    |
|                             |                          |                            |                            | UTC                          |
| + 597127ea7f769051b0bbf971  | succeeded (116s elapsed) | create_vm                  | st2cd.create_vm_role       | Thu, 20 Jul 2017 22:00:09    |
|                             |                          |                            |                            | UTC                          |
|    597127ed7f769051b0bbf983 | succeeded (4s elapsed)   | get_subnet_id              | st2.kv.get                 | Thu, 20 Jul 2017 22:00:13    |
|                             |                          |                            |                            | UTC                          |
|    597127f27f769051b0bbf98b | succeeded (3s elapsed)   | get_ami                    | st2.kv.get                 | Thu, 20 Jul 2017 22:00:18    |
|                             |                          |                            |                            | UTC                          |
|    597127f57f769051b0bbf993 | succeeded (4s elapsed)   | check_cname                | aws.r53_zone_get_cname     | Thu, 20 Jul 2017 22:00:21    |
|                             |                          |                            |                            | UTC                          |
|    597127fa7f769051b0bbf99d | succeeded (4s elapsed)   | run_instance               | aws.ec2_run_instances      | Thu, 20 Jul 2017 22:00:26    |
|                             |                          |                            |                            | UTC                          |
|    597127ff7f769051b0bbf9a5 | succeeded (12s elapsed)  | wait_for_instance          | aws.ec2_wait_for_state     | Thu, 20 Jul 2017 22:00:31    |
|                             |                          |                            |                            | UTC                          |
|    5971280c7f769051b0bbf9ab | succeeded (38s elapsed)  | wait_for_ssh               | linux.wait_for_ssh         | Thu, 20 Jul 2017 22:00:44    |
|                             |                          |                            |                            | UTC                          |
|    597128337f769051b0bbf9cb | succeeded (1s elapsed)   | add_tags                   | aws.ec2_create_tags        | Thu, 20 Jul 2017 22:01:23    |
|                             |                          |                            |                            | UTC                          |
|    597128357f769051b0bbf9cf | succeeded (3s elapsed)   | add_cname                  | aws.r53_zone_add_cname     | Thu, 20 Jul 2017 22:01:25    |
|                             |                          |                            |                            | UTC                          |
|    597128397f769051b0bbf9d5 | succeeded (3s elapsed)   | get_distro                 | st2cd.get_distro           | Thu, 20 Jul 2017 22:01:29    |
|                             |                          |                            |                            | UTC                          |
|    5971283d7f769051b0bbf9db | succeeded (3s elapsed)   | set_hostname               | st2cd.set_hostname         | Thu, 20 Jul 2017 22:01:33    |
|                             |                          |                            |                            | UTC                          |
|    597128417f769051b0bbf9df | succeeded (2s elapsed)   | reboot                     | core.remote_sudo           | Thu, 20 Jul 2017 22:01:37    |
|                             |                          |                            |                            | UTC                          |
|    597128437f769051b0bbf9e1 | succeeded (1s elapsed)   | wait_for_ssh_post_reboot   | linux.wait_for_ssh         | Thu, 20 Jul 2017 22:01:39    |
|                             |                          |                            |                            | UTC                          |
|   597127e97f769051b0bbf970  | succeeded (4s elapsed)   | notify_start               | slack.chat.postMessage     | Thu, 20 Jul 2017 22:00:09    |
|                             |                          |                            |                            | UTC                          |
|   5971285e7f769051b0bbf9e9  | succeeded (2s elapsed)   | patch_rhel6                | core.remote_sudo           | Thu, 20 Jul 2017 22:02:06    |
|                             |                          |                            |                            | UTC                          |
|   597128617f769051b0bbf9f1  | succeeded (19s elapsed)  | patch_rhel6_curl           | core.remote_sudo           | Thu, 20 Jul 2017 22:02:08    |
|                             |                          |                            |                            | UTC                          |
|   597128747f769051b0bbf9f3  | succeeded (3s elapsed)   | get_bootstrap_script       | core.remote_sudo           | Thu, 20 Jul 2017 22:02:27    |
|                             |                          |                            |                            | UTC                          |
|   597128767f769051b0bbf9f5  | succeeded (256s elapsed) | run_bootstrap_script       | core.remote_sudo           | Thu, 20 Jul 2017 22:02:30    |
|                             |                          |                            |                            | UTC                          |
|   597129767f769051b0bbfa29  | succeeded (66s elapsed)  | upgrade_to_enterprise      | st2cd.st2_upgrade_to_enter | Thu, 20 Jul 2017 22:06:46    |
|                             |                          |                            | prise                      | UTC                          |
| + 597129b87f769051b0bbfa41  | succeeded (31s elapsed)  | get_installed_version      | st2cd.st2_get_installed_ve | Thu, 20 Jul 2017 22:07:52    |
|                             |                          |                            | rsion                      | UTC                          |
|    597129b97f769051b0bbfa43 | succeeded (5s elapsed)   | get_distro                 | st2cd.get_distro           | Thu, 20 Jul 2017 22:07:53    |
|                             |                          |                            |                            | UTC                          |
|    597129bf7f769051b0bbfa4c | succeeded (7s elapsed)   | get_version_redhat         | core.remote_sudo           | Thu, 20 Jul 2017 22:07:58    |
|                             |                          |                            |                            | UTC                          |
|    597129bf7f769051b0bbfa4a | succeeded (9s elapsed)   | get_version_redhat         | core.remote_sudo           | Thu, 20 Jul 2017 22:07:58    |
|                             |                          |                            |                            | UTC                          |
|    597129bf7f769051b0bbfa4b | succeeded (10s elapsed)  | get_version_redhat         | core.remote_sudo           | Thu, 20 Jul 2017 22:07:58    |
|                             |                          |                            |                            | UTC                          |
|    597129bf7f769051b0bbfa4f | succeeded (7s elapsed)   | get_version_redhat         | core.remote_sudo           | Thu, 20 Jul 2017 22:07:58    |
|                             |                          |                            |                            | UTC                          |
|    597129bf7f769051b0bbfa4d | succeeded (4s elapsed)   | get_version_redhat         | core.remote_sudo           | Thu, 20 Jul 2017 22:07:58    |
|                             |                          |                            |                            | UTC                          |
|    597129bf7f769051b0bbfa4e | succeeded (7s elapsed)   | get_version_redhat         | core.remote_sudo           | Thu, 20 Jul 2017 22:07:58    |
|                             |                          |                            |                            | UTC                          |
| + 597129d77f769051b0bbfa57  | failed (160s elapsed)    | run_e2e_tests              | st2cd.st2_e2e_tests        | Thu, 20 Jul 2017 22:08:23    |
|                             |                          |                            |                            | UTC                          |
|    597129d87f769051b0bbfa59 | succeeded (2s elapsed)   | get_st2_token              | st2cd.get_st2_token        | Thu, 20 Jul 2017 22:08:24    |
|                             |                          |                            |                            | UTC                          |
|    597129db7f769051b0bbfa61 | succeeded (109s elapsed) | setup_e2e_tests            | st2cd.setup_e2e_tests      | Thu, 20 Jul 2017 22:08:27    |
|                             |                          |                            |                            | UTC                          |
|    59712a497f769051b0bbfa77 | succeeded (11s elapsed)  | test_core_remote_single_ho | core.remote                | Thu, 20 Jul 2017 22:10:16    |
|                             |                          | st                         |                            | UTC                          |
|    59712a497f769051b0bbfa79 | failed (10s elapsed)     | test_core_http             | core.remote                | Thu, 20 Jul 2017 22:10:16    |
|                             |                          |                            |                            | UTC                          |
|    59712a497f769051b0bbfa78 | succeeded (11s elapsed)  | test_core_local            | core.remote                | Thu, 20 Jul 2017 22:10:16    |
|                             |                          |                            |                            | UTC                          |
| + 59712a777f769051b0bbfa8d  | succeeded (120s elapsed) | destroy_vm_on_failure      | st2cd.destroy_vm           | Thu, 20 Jul 2017 22:11:03    |
|                             |                          |                            |                            | UTC                          |
|    59712a787f769051b0bbfa8f | succeeded (1s elapsed)   | get_instance_dns           | linux.dig                  | Thu, 20 Jul 2017 22:11:04    |
|                             |                          |                            |                            | UTC                          |
|    59712a7a7f769051b0bbfa91 | succeeded (1s elapsed)   | get_instances_by_dns       | aws.ec2_get_only_instances | Thu, 20 Jul 2017 22:11:06    |
|                             |                          |                            |                            | UTC                          |
|    59712a837f769051b0bbfa93 | failed (1s elapsed)      | deregister_monitor         | sensu.client_delete        | Thu, 20 Jul 2017 22:11:15    |
|                             |                          |                            |                            | UTC                          |
|    59712a857f769051b0bbfa96 | succeeded (2s elapsed)   | notify_deregister_monitor_ | slack.post_message         | Thu, 20 Jul 2017 22:11:16    |
|                             |                          | failure                    |                            | UTC                          |
|    59712a857f769051b0bbfa97 | succeeded (1s elapsed)   | get_volumes                | aws.ec2_get_instance_attri | Thu, 20 Jul 2017 22:11:17    |
|                             |                          |                            | bute                       | UTC                          |
|    59712a877f769051b0bbfa99 | succeeded (1s elapsed)   | destroy_instance           | aws.ec2_terminate_instance | Thu, 20 Jul 2017 22:11:19    |
|                             |                          |                            | s                          | UTC                          |
|    59712ac47f769051b0bbfaa3 | succeeded (2s elapsed)   | delete_cname               | aws.r53_zone_delete_cname  | Thu, 20 Jul 2017 22:12:20    |
|                             |                          |                            |                            | UTC                          |
|    59712ac77f769051b0bbfaa7 | succeeded (2s elapsed)   | notify_success             | slack.post_message         | Thu, 20 Jul 2017 22:12:23    |
|                             |                          |                            |                            | UTC                          |
|   59712af27f769051b0bbfab8  | succeeded (3s elapsed)   | notify_failure             | slack.chat.postMessage     | Thu, 20 Jul 2017 22:13:05    |
|                             |                          |                            |                            | UTC                          |
|   59712af27f769051b0bbfab9  | succeeded (4s elapsed)   | notify_failure             | slack.chat.postMessage     | Thu, 20 Jul 2017 22:13:06    |
|                             |                          |                            |                            | UTC                          |
+-----------------------------+--------------------------+----------------------------+----------------------------+------------------------------+
oswaltm@st2cicd007:~$ st2 execution get 59712a837f769051b0bbfa93
id: 59712a837f769051b0bbfa93
status: failed (1s elapsed)
parameters:
  client: st2-ent-pkg-unstable-el6
result:
  exit_code: 1
  result: None
  stderr: "No handlers could be found for logger "pysensu.api"
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 259, in <module>
    obj.run()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 155, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/sensu/actions/client_delete.py", line 10, in run
    return self.api.delete_client(client)
  File "/opt/stackstorm/virtualenvs/sensu/lib/python2.7/site-packages/pysensu/api.py", line 99, in delete_client
    self._request('DELETE', '/clients/{}'.format(client))
  File "/opt/stackstorm/virtualenvs/sensu/lib/python2.7/site-packages/pysensu/api.py", line 64, in _request
    raise SensuAPIException('API bad response')
pysensu.api.SensuAPIException: API bad response
"
  stdout: ''
arm4b commented 7 years ago

See existing Issue: https://github.com/StackStorm-Exchange/stackstorm-sensu/issues/3 for more info