search

Percona-Lab / mongodb_consistent_backup

A tool for performing consistent backups of MongoDB Clusters or Replica Sets
https://www.percona.com
Apache License 2.0
276 stars 80 forks source link

Tailer csReplSet/ra1svlrp4i:27019 received NetworkTimeout exception: ra1svlrp4k:27019: timed out. Attempting retry #272

Closed jpbaboo closed 5 years ago

jpbaboo commented 6 years ago

Hi during a backup of our cluster (6 serveurs) wee met this issue

[2018-08-02 13:32:10,076] [ERROR] [TailThread-7] [TailThread:run:152] Tailer csReplSet/ra1svlrp4i:27019 encountered error: Tailer host changed from (u'ra1svlrp4k', 27019) to ('ra1svlrp4i', 27019)! Process TailThread-7: Traceback (most recent call last): File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/root/.pex/install/mongodb_consistent_backup-1.3.0-py2-none-any.whl.e178d615ee15c20b6f098f1eb98bbd9412aa299f/mongodb_consistent_backup-1.3.0-py2-none-any.whl/mongodb_consistent_backup/Oplog/Tailer/TailThread.py", line 155, in run raise OperationError(e) OperationError: Tailer host changed from (u'ra1svlrp4k', 27019) to ('ra1svlrp4i', 27019)! [2018-08-02 13:32:10,197] [INFO] [TailThread-3] [TailThread:status:87] Oplog tailer rs03/ra1svlrp4k:27018 status: 201 oplog changes, ts: Timestamp(1533209508, 1) [2018-08-02 13:32:10,434] [ERROR] [MainProcess] [Mongodump:wait:90] Received backup stop event due to error(s), stopping backup! [2018-08-02 13:32:10,435] [ERROR] [MainProcess] [Stage:run:99] Stage mongodb_consistent_backup.Backup did not complete! mongo-20180802.log

here enclosed the complete log IF you need more infos don't hesitate Thks for your help

corey-hammerton commented 6 years ago

We occasionally see that too. That has happened when there was a spike in slow queries on the target secondary node. We don't know what's causing the slow queries, but this intermittent behaviour is present.

marioc-bitheads commented 5 years ago

We are also getting this "Tailer host changed" error (although we are not getting a timeout error), after 5 attempts we have yet to succeed making a backup, this seem to be fairly consistent on our environment, can this be safely ignored? is it not acceptable to have the oplog tailer change host?

jpbaboo commented 5 years ago

We have moved ou Vm' to another Host.. No more issue.. Network Traffic was not good