apache / openwhisk-package-kafka

Apache OpenWhisk package for communicating with Kafka or Message Hub
https://openwhisk.apache.org/
Apache License 2.0
33 stars 43 forks source link

Do not update last canary everytime a database connection attempt occurs #356

Closed dubee closed 4 years ago

dubee commented 4 years ago

When the provider continuously encounters network errors from the database, the last canary time is always being set to the current time. This logic prevents monitoring from detecting that the canary has actually gone stale because of the network issues. Instead, only update the canary outside of the database reconnection loop.

dubee commented 4 years ago

To further clarify, the logs show the sequence below repeatedly. While the changes feed appeared to re-establish its connection, read timeouts occur when trying to read from the changes feed. This explains why an exception was not thrown before self.lastCanaryTime = datetime.now() was executed.

[kafkatriggers] [canary] Exception caught from changes feed. Restarting changes feed... 
[kafkatriggers] HTTPSConnectionPool(host='OMITTED', port=443): Read timed out. (read timeout=30) 
[kafkatriggers] Shutting down existing DB client 
[kafkatriggers] Starting new HTTPS connection (2): OMITTED
[kafkatriggers] Starting changes feed 
[kafkatriggers] Starting new HTTPS connection (1): OMITTED
[kafkatriggers] Database exists - connecting to it.