search

Collaborne / kubernetes-mongodb-labeler

Automatically label a MongoDB pod in kubernetes
Apache License 2.0
0 stars 0 forks source link

'Error when checking the status: topology was destroyed' after long runtime #6

Open ankon opened 7 years ago

ankon commented 7 years ago

Seen this on one of our systems, it looks from http://stackoverflow.com/questions/30909492/mongoerror-topology-was-destroyed like we need to properly handle some connection errors/states.

ankon commented 7 years ago

Another instance:

npm info it worked if it ends with ok
npm info using npm@3.10.10
npm info using node@v6.11.2
npm info lifecycle kubernetes-mongodb-labeler@1.2.0~prestart: kubernetes-mongodb-labeler@1.2.0
npm info lifecycle kubernetes-mongodb-labeler@1.2.0~start: kubernetes-mongodb-labeler@1.2.0

> kubernetes-mongodb-labeler@1.2.0 start /app
> node src/index.js "--namespace" "production-201707" "mongodb-1"

Wed, 04 Oct 2017 22:02:30 GMT kubernetes-mongodb-labeler Error when checking the status: write EPIPE
Wed, 04 Oct 2017 22:02:30 GMT kubernetes-mongodb-labeler Status for mongodb-1 changed from SECONDARY to ERROR
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: failed to reconnect after 30 attempts with interval 1000 ms
Wed, 04 Oct 2017 22:03:22 GMT kubernetes-mongodb-labeler Error when checking the status: topology was destroyed
Wed, 04 Oct 2017 22:03:23 GMT kubernetes-mongodb-labeler Error when checking the status: topology was destroyed
Wed, 04 Oct 2017 22:03:25 GMT kubernetes-mongodb-labeler Error when checking the status: topology was destroyed
[...]

Restarting the labeler worked, and it was able to connect to the topology. The MongoDB rs.status() did not indicate any issues while the labeler was complaining.

ankon commented 6 years ago

This happened again, on a pod with a runtime of 8 days.