search

StackStorm / st2

StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html
https://stackstorm.com/
Apache License 2.0
6.1k stars 746 forks source link

st2 services should log connection to [coordination] backend #4783

Open arm4b opened 5 years ago

arm4b commented 5 years ago

st2 services log error if [coordination] connection was unsuccessful, which is great:

2019-08-30 11:19:54,609 DEBUG [-] Using Python: 2.7.12 (/opt/stackstorm/st2/bin/python)
2019-08-30 11:19:54,609 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2019-08-30 11:19:54,610 DEBUG [-] Using logging config: /etc/st2/logging.api.gunicorn.conf
2019-08-30 11:19:54,622 INFO [-] Connecting to database "st2" @ "ignorant-manta-mongodb-ha-0.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-1.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-2.ignorant-manta-mongodb-ha:27017 (replica set)" as user "admin".
2019-08-30 11:19:54,797 INFO [-] Successfully connected to database "st2" @ "ignorant-manta-mongodb-ha-0.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-1.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-2.ignorant-manta-mongodb-ha:27017 (replica set)" as user "admin".
2019-08-30 11:19:56,138 INFO [-] (PID=1) ST2 API is serving on http://0.0.0.0:9101.
2019-08-30 11:19:56,138 INFO [-] Creating st2api: StackStorm v3.2dev as OpenAPI app.
2019-08-30 11:19:57,063 INFO [-] Connected to amqp://admin:**@ignorant-manta-rabbitmq-ha-discovery:5672//
2019-08-30 11:19:57,111 ERROR [-] (PID=1) ST2 API quit due to exception.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2api/cmd/api.py", line 81, in main
    return _run_server()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2api/cmd/api.py", line 70, in _run_server
    wsgi.server(sock, app.setup_app(), custom_pool=worker_pool, log=LOG, log_output=False)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2api/app.py", line 79, in setup_app
    router.add_spec(spec, transforms=transforms)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/router.py", line 211, in add_spec
    __import__(module_name)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2api/controllers/v1/keyvalue.py", line 436, in <module>
    key_value_pair_controller = KeyValuePairController()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2api/controllers/v1/keyvalue.py", line 61, in __init__
    self._coordinator = coordination.get_coordinator()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/services/coordination.py", line 226, in get_coordinator
    COORDINATOR = coordinator_setup(start_heart=start_heart)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/services/coordination.py", line 199, in coordinator_setup
    coordinator.start(start_heart=start_heart)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/tooz/coordination.py", line 426, in start
    self._start()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/tooz/drivers/etcd.py", line 252, in _start
    encodeutils.exception_to_unicode(e))
ToozConnectionError: HTTPConnectionPool(host='etcd-cluster-client', port=2379): Max retries exceeded with url: /v2/stats/self (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fca9effa850>: Failed to establish a new connection: [Errno -2] No address found',))

However it doesn't log any message if connection was successful to [coordination] backend:

$ kubectl logs ignorant-manta-st2api-fd58c5b8f-4x8gn
2019-08-30 11:22:12,899 DEBUG [-] Using Python: 2.7.12 (/opt/stackstorm/st2/bin/python)
2019-08-30 11:22:12,900 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2019-08-30 11:22:12,900 DEBUG [-] Using logging config: /etc/st2/logging.api.gunicorn.conf
2019-08-30 11:22:12,910 INFO [-] Connecting to database "st2" @ "ignorant-manta-mongodb-ha-0.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-1.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-2.ignorant-manta-mongodb-ha:27017 (replica set)" as user "admin".
2019-08-30 11:22:13,194 INFO [-] Successfully connected to database "st2" @ "ignorant-manta-mongodb-ha-0.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-1.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-2.ignorant-manta-mongodb-ha:27017 (replica set)" as user "admin".
2019-08-30 11:22:14,768 INFO [-] (PID=1) ST2 API is serving on http://0.0.0.0:9101.
2019-08-30 11:22:14,768 INFO [-] Creating st2api: StackStorm v3.2dev as OpenAPI app.
2019-08-30 11:22:15,917 INFO [-] Connected to amqp://admin:**@ignorant-manta-rabbitmq-ha-discovery:5672//
2019-08-30 11:22:16,287 INFO [-] Connected to amqp://admin:**@ignorant-manta-rabbitmq-ha-discovery:5672//

It's preferred to log this for informational, consistency and user expectation reasons, as it happens for both MongoDB and RabbitMQ in logs. This will als make it clearer which services really rely on coordination.

Rishabhraghwendra18 commented 3 years ago

Hi @armab . is this issue open?

arm4b commented 3 years ago

Yes, it's still actual and Pull Requests with improvements are welcome.

Rishabhraghwendra18 commented 3 years ago

Hi @armab . First timer this side . May you guide me how to solve this issue ?