search

clowder-framework / pyclowder

Library to assist in the development of extractors for clowder.
University of Illinois/NCSA Open Source License
10 stars 2 forks source link

pyclowder does not dectect rabbitmq shutdown #17

Closed robkooper closed 4 years ago

robkooper commented 4 years ago

When the rabbitmq pod restarts the extractor does not dectect and restart:

2020-09-24T22:21:54.178181563Z 2020-09-24 22:21:54,175 [Connector-0    ] INFO    : pika.adapters.utils.io_services_utils - Aborting transport connection: state=1; <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.42.96.8', 51062)>
2020-09-24T22:21:54.178202607Z 2020-09-24 22:21:54,175 [Connector-0    ] INFO    : pika.adapters.utils.io_services_utils - _AsyncTransportBase._initate_abort(): Initiating abrupt asynchronous transport shutdown: state=1; error=None; <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.42.96.8', 51062)>
2020-09-24T22:21:54.178213692Z 2020-09-24 22:21:54,175 [Connector-0    ] INFO    : pika.adapters.utils.io_services_utils - Deactivating transport: state=1; <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.42.96.8', 51062)>
2020-09-24T22:21:54.178222918Z 2020-09-24 22:21:54,176 [Connector-0    ] INFO    : pika.connection - AMQP stack terminated, failed to connect, or aborted: opened=True, error-arg=None; pending-error=ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2020-09-24T22:21:54.178231643Z 2020-09-24 22:21:54,176 [Connector-0    ] INFO    : pika.connection - Stack terminated due to ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2020-09-24T22:21:54.178239169Z 2020-09-24 22:21:54,176 [Connector-0    ] INFO    : pika.adapters.utils.io_services_utils - Closing transport socket and unlinking: state=3; <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.42.96.8', 51062)>
2020-09-24T22:21:54.178247197Z 2020-09-24 22:21:54,177 [Connector-0    ] ERROR   : pika.adapters.blocking_connection - Unexpected connection close detected: ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2020-09-24T22:21:54.178781491Z 2020-09-24 22:21:54,177 [Connector-0    ] ERROR   : pyclowder.connectors - Error while consuming messages.
2020-09-24T22:21:54.178795972Z Traceback (most recent call last):
2020-09-24T22:21:54.178801162Z   File "/usr/local/lib/python3.7/site-packages/pyclowder/connectors.py", line 705, in listen
2020-09-24T22:21:54.178805922Z     self.channel.connection.process_data_events(time_limit=1)  # 1 second
2020-09-24T22:21:54.178810723Z   File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 828, in process_data_events
2020-09-24T22:21:54.178815475Z     self._flush_output(timer.is_ready, common_terminator)
2020-09-24T22:21:54.178820317Z   File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 522, in _flush_output
2020-09-24T22:21:54.178825053Z     raise self._closed_result.value.error
2020-09-24T22:21:54.178829736Z pika.exceptions.ConnectionClosedByBroker: (320, "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'")
2020-09-24T22:21:54.178850007Z 2020-09-24 22:21:54,178 [Connector-0    ] INFO    : pyclowder.connectors - Stopped listening for messages.
2020-09-24T22:21:54.908505278Z 2020-09-24 22:21:54,908 [Thread-3       ] INFO    : pika.adapters.utils.io_services_utils - Aborting transport connection: state=1; <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.42.96.8', 51066)>
2020-09-24T22:21:54.908601616Z 2020-09-24 22:21:54,908 [Thread-3       ] INFO    : pika.adapters.utils.io_services_utils - _AsyncTransportBase._initate_abort(): Initiating abrupt asynchronous transport shutdown: state=1; error=None; <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.42.96.8', 51066)>
2020-09-24T22:21:54.908608557Z 2020-09-24 22:21:54,908 [Thread-3       ] INFO    : pika.adapters.utils.io_services_utils - Deactivating transport: state=1; <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.42.96.8', 51066)>
2020-09-24T22:21:54.908937577Z 2020-09-24 22:21:54,908 [Thread-3       ] INFO    : pika.connection - AMQP stack terminated, failed to connect, or aborted: opened=True, error-arg=None; pending-error=ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2020-09-24T22:21:54.909134463Z 2020-09-24 22:21:54,909 [Thread-3       ] INFO    : pika.connection - Stack terminated due to ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2020-09-24T22:21:54.909299113Z 2020-09-24 22:21:54,909 [Thread-3       ] INFO    : pika.adapters.utils.io_services_utils - Closing transport socket and unlinking: state=3; <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.42.96.8', 51066)>
2020-09-24T22:21:54.909478373Z 2020-09-24 22:21:54,909 [Thread-3       ] ERROR   : pika.adapters.blocking_connection - Unexpected connection close detected: ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2020-09-24T22:21:54.909847911Z 2020-09-24 22:21:54,909 [Thread-3       ] ERROR   : pyclowder.connectors - Error while sending heartbeat.
2020-09-24T22:21:54.909918852Z Traceback (most recent call last):
2020-09-24T22:21:54.909925702Z   File "/usr/local/lib/python3.7/site-packages/pyclowder/connectors.py", line 821, in send_heartbeat
2020-09-24T22:21:54.909929900Z     self.channel.connection.process_data_events()
2020-09-24T22:21:54.909934142Z   File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 828, in process_data_events
2020-09-24T22:21:54.909938324Z     self._flush_output(timer.is_ready, common_terminator)
2020-09-24T22:21:54.909942552Z   File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 522, in _flush_output
2020-09-24T22:21:54.909946666Z     raise self._closed_result.value.error
2020-09-24T22:21:54.909950540Z pika.exceptions.ConnectionClosedByBroker: (320, "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'")

Maybe a thread that is not marked as daemon thread keeps main process running.