search

StackStorm / st2

StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html
https://stackstorm.com/
Apache License 2.0
5.97k stars 745 forks source link

Eventlet causing issues in thread pools for sensors #3818

Open tonybaloney opened 6 years ago

tonybaloney commented 6 years ago

See https://github.com/eventlet/eventlet/issues/147

Observed with msexchange pack, which uses requests when watching the sensor container directly.

[root@irtyrt1 msexchange]# /opt/stackstorm/st2/bin/st2sensorcontainer --config-file=/etc/st2/st2.conf --sensor-ref=msexchange.ItemSensor
2017-10-30 03:52:44,287 DEBUG [-] Using config files: /etc/st2/st2.conf
2017-10-30 03:52:44,287 DEBUG [-] Using logging config: /etc/st2/logging.sensorcontainer.conf
2017-10-30 03:52:44,289 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2017-10-30 03:52:44,454 INFO [-] Setting up container to run 1 sensors.
2017-10-30 03:52:44,454 INFO [-]  Sensors list - [u'msexchange.ItemSensor'].
2017-10-30 03:52:44,455 INFO [-] (PID:25913) SensorContainer started.
2017-10-30 03:52:44,455 INFO [-] Running sensor msexchange.ItemSensor
2017-10-30 03:52:44,466 INFO [-] Connected to amqp://guest:**@127.0.0.1:5672//
2017-10-30 03:52:44,466 AUDIT [-] Access granted to "sensors_container" with the token set to expire at "2017-10-31T07:52:44.464372Z". (username='sensors_container',token_expiration='2017-10-31T07:52:44.464372Z')
2017-10-30 03:52:44,467 WARNING [-] "auth.api_url" configuration option is not configured
2017-10-30 03:52:44,504 INFO [-] Sensor msexchange.ItemSensor started
2017-10-30 03:52:45,401 INFO [-] Found config for sensor "ItemSensor"
2017-10-30 03:52:45,411 INFO [-] Watcher started
2017-10-30 03:52:45,411 INFO [-] Running sensor initialization code
2017-10-30 03:52:45,431 INFO [-] Connected to amqp://guest:**@127.0.0.1:5672//
2017-10-30 03:52:47,016 INFO [-] API version "Exchange2007" worked but server reports version "Exchange2016". Using "Exchange2016"
2017-10-30 03:52:47,019 ERROR [-] Exception in thread Thread-17:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 301, in _handle_workers
    while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
AttributeError: '_MainThread' object has no attribute '_state'

2017-10-30 03:52:47,101 INFO [-] Running sensor in passive mode
2017-10-30 03:52:47,101 AUDIT [-] Creating new Client object.
2017-10-30 03:52:47,106 AUDIT [-] Access granted to "sensor_service" with the token set to expire at "2017-10-31T07:52:47.104604Z". (username='sensor_service',token_expiration='2017-10-31T07:52:47.104604Z')
2017-10-30 03:52:47,106 WARNING [-] "auth.api_url" configuration option is not configured
2017-10-30 03:52:47,106 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:52:47,130 ERROR [-] /opt/stackstorm/virtualenvs/msexchange/lib/python2.7/site-packages/exchangelib/folders.py:127: UserWarning: The get_folder_by_name() method is deprecated. Use "[f for f in self.walk() if f.name == name]" or "some_folder / 'Sub Folder'" instead, to find folders by name.
  warnings.warn('The get_folder_by_name() method is deprecated. Use "[f for f in self.walk() if f.name == name]" '

2017-10-30 03:52:47,585 INFO [-] Found 0 items
2017-10-30 03:52:52,668 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:52:52,819 INFO [-] Found 0 items
2017-10-30 03:52:57,901 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:52:58,042 INFO [-] Found 0 items
2017-10-30 03:53:03,125 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:03,272 INFO [-] Found 0 items
2017-10-30 03:53:08,352 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:08,498 INFO [-] Found 0 items
2017-10-30 03:53:13,578 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:13,719 INFO [-] Found 0 items
2017-10-30 03:53:18,800 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:18,942 INFO [-] Found 0 items
2017-10-30 03:53:24,025 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:24,166 INFO [-] Found 0 items
2017-10-30 03:53:29,246 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:29,399 INFO [-] Found 0 items
2017-10-30 03:53:34,480 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:34,622 INFO [-] Found 0 items
2017-10-30 03:53:39,703 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:39,844 INFO [-] Found 0 items
2017-10-30 03:53:44,933 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:45,076 INFO [-] Found 1 items
2017-10-30 03:53:45,157 ERROR [-] Exception in thread Thread-18:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 315, in _handle_tasks
    if thread._state:
AttributeError: '_MainThread' object has no attribute '_state'
LindsayHill commented 6 years ago

There were some eventlet fixes that went into 2.7.2 - I suspect this may be fixed too https://github.com/StackStorm/st2/issues/3498#issuecomment-390951117

stale[bot] commented 5 years ago

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically marking is as stale. If this issue is not relevant or applicable anymore (problem has been fixed in a new version or similar), please close the issue or let us know so we can close it. On the contrary, if the issue is still relevant, there is nothing you need to do, but if you have any additional details or context which would help us when working on this issue, please include it as a comment to this issue.