Closed dubee closed 5 years ago
To further clarify, the logs show the sequence below repeatedly. While the changes feed appeared to re-establish its connection, read timeouts occur when trying to read from the changes feed. This explains why an exception was not thrown before self.lastCanaryTime = datetime.now()
was executed.
[kafkatriggers] [canary] Exception caught from changes feed. Restarting changes feed...
[kafkatriggers] HTTPSConnectionPool(host='OMITTED', port=443): Read timed out. (read timeout=30)
[kafkatriggers] Shutting down existing DB client
[kafkatriggers] Starting new HTTPS connection (2): OMITTED
[kafkatriggers] Starting changes feed
[kafkatriggers] Starting new HTTPS connection (1): OMITTED
[kafkatriggers] Database exists - connecting to it.
When the provider continuously encounters network errors from the database, the last canary time is always being set to the current time. This logic prevents monitoring from detecting that the canary has actually gone stale because of the network issues. Instead, only update the canary outside of the database reconnection loop.