search

Percona-Lab / mongodb_consistent_backup

A tool for performing consistent backups of MongoDB Clusters or Replica Sets
https://www.percona.com
Apache License 2.0
276 stars 82 forks source link

[ERROR] [MainProcess] [DB:connect:117] Unable to connect to CONFIG SERVERS #239

Open patricklarochelle opened 6 years ago

patricklarochelle commented 6 years ago

Running on : CentOS Linux release 7.3.1611

With MongoDB version: 3.0.11

With mongodb-consistent-backup version: mongodb-consistent-backup version: 1.2.0, git commit hash: unknown Python version: 2.7.5 Python modules: Fabric: 1.14.0 pymongo: 3.5.1 multiprocessing: 0.70a1 yaml: 3.12 boto: 2.48.0 filechunkio: 1.8

execution failing within first few connection steps:

MongoDB 3.0.11 and mongodb-consistent-backup running on same host  or   
mongodb-consistent-backup running on a remote host with MongoDB shell version: 3.2.8 produce 
    the same error

[root@]# ./mongodb-consistent-backup --config /root/patrick/mongodb-consistent-backup.yml --verbose      

    [2017-10-27 14:08:12,645] [DEBUG] [MainProcess] [Lock:acquire:20] Acquired exclusive lock on file: /tmp/mongodb-consistent-backup.lock
    [2017-10-27 14:08:12,645] [INFO] [MainProcess] [Main:init:144] Starting mongodb-consistent-backup version 1.2.0 (git commit: unknown)
    [2017-10-27 14:08:12,646] [INFO] [MainProcess] [Main:init:145] Loaded config: {"archive": {"method": "tar", "tar": {"compression": "gzip"}, "zbackup": {"binary": "/usr/bin/zbackup", "cache_mb": 128, "compression": "lzma"}}, "authdb": "admin", "backup": {"location": "/opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe", "method": "mongodump", "mongodump": {"binary": "/usr/bin/mongodump", "compression": "auto"}, "name": "default"}, "configPath": "/root/patrick/mongodb-consistent-backup.yml", "environment": "production", "host": "host11-mongo101.ubisoft.onbe", "lock_file": "/tmp/mongodb-consistent-backup.lock", "log_dir": "/var/log/mongodb-consistent-backup", "notify": {"method": "none"}, "oplog": {"compression": "none", "flush": {"max_docs": 100, "max_secs": 1}, "tailer": {"enabled": "true", "status_interval": 30}}, "password": "******", "port": 27017, "replication": {"hidden_only": true, "max_lag_secs": 10, "max_priority": 1000}, "rotate": {"max_backups": 5}, "sharding": {"balancer": {"ping_secs": 3, "wait_secs": 300}}, "upload": {"method": "none", "retries": 5, "rsync": {"path": "/", "port": 22}, "s3": {"chunk_size_mb": 50, "region": "us-east-1", "secure": true}, "threads": 4}, "username": "admin", "verbose": true}
    [2017-10-27 14:08:12,646] [INFO] [MainProcess] [Stage:init:31] Notify stage disabled, skipping
    [2017-10-27 14:08:12,647] [DEBUG] [MainProcess] [DB:connect:111] Getting MongoDB connection to host11-mongo101.ubisoft.onbe:27017 (replicaSet=None, readPreference=secondaryPreferred, readPreferenceTags=False, ssl=False)
    [2017-10-27 14:08:12,653] [DEBUG] [MainProcess] [DB:auth_if_required:126] Authenticating connection with username: admin
    [2017-10-27 14:08:12,664] [INFO] [MainProcess] [State:init:145] Initializing root state directory /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default
    [2017-10-27 14:08:12,665] [INFO] [MainProcess] [State:load_backups:162] Found 0 existing completed backups for set
    [2017-10-27 14:08:12,666] [INFO] [MainProcess] [State:init:131] Initializing backup state directory: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/20171027_1408
    [2017-10-27 14:08:12,666] [DEBUG] [MainProcess] [Lock:acquire:20] Acquired exclusive lock on file: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/mongodb-consistent-backup_META.lock
    [2017-10-27 14:08:12,666] [DEBUG] [MainProcess] [State:write:73] Writing StateRoot state file: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/mongodb-consistent-backup_META/meta.bson
    [2017-10-27 14:08:12,666] [DEBUG] [MainProcess] [Lock:release:30] Releasing exclusive lock on file: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/mongodb-consistent-backup_META.lock
    [2017-10-27 14:08:12,666] [DEBUG] [MainProcess] [Lock:acquire:20] Acquired exclusive lock on file: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/20171027_1408/mongodb-consistent-backup_META.lock
    [2017-10-27 14:08:12,667] [DEBUG] [MainProcess] [State:write:73] Writing StateBackup state file: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/20171027_1408/mongodb-consistent-backup_META/meta.bson
    [2017-10-27 14:08:12,667] [DEBUG] [MainProcess] [Lock:release:30] Releasing exclusive lock on file: /opt/mongodb/mongo-backups/host11-mongo101.ubisoft.onbe/default/20171027_1408/mongodb-consistent-backup_META.lock
    [2017-10-27 14:08:12,668] [DEBUG] [MainProcess] [Stage:init:48] Loaded stage mongodb_consistent_backup.Archive with task Tar
    [2017-10-27 14:08:12,668] [INFO] [MainProcess] [Stage:init:31] Upload stage disabled, skipping
    [2017-10-27 14:08:12,668] [INFO] [MainProcess] [Main:run:299] Running backup in sharding mode using seed node(s): host11-mongo101.ubisoft.onbe:27017
    [2017-10-27 14:08:12,671] [INFO] [MainProcess] [Sharding:get_start_state:68] Began with balancer state running: False
    [2017-10-27 14:08:12,674] [DEBUG] [MainProcess] [DB:connect:111] Getting MongoDB connection to shard1/host11-mongo101.ubisoft.onbe:10001,host11-mongo102.ubisoft.onbe:10001 (replicaSet=None, readPreference=primaryPreferred, readPreferenceTags=False, ssl=False)
    [2017-10-27 14:08:12,678] [DEBUG] [MainProcess] [DB:auth_if_required:126] Authenticating connection with username: admin
    [2017-10-27 14:08:12,691] [DEBUG] [MainProcess] [DB:connect:111] Getting MongoDB connection to shard2/host11-mongo201.ubisoft.onbe:10001,host11-mongo202.ubisoft.onbe:10001 (replicaSet=None, readPreference=primaryPreferred, readPreferenceTags=False, ssl=False)
    [2017-10-27 14:08:12,700] [DEBUG] [MainProcess] [DB:auth_if_required:126] Authenticating connection with username: admin
    [2017-10-27 14:08:12,714] [INFO] [MainProcess] [Sharding:get_config_server:188] Found sharding config server: host11-mongo101.ubisoft.onbe:10000,host11-mongo102.ubisoft.onbe:10000,host11-mongo103.ubisoft.onbe:10000
    [2017-10-27 14:08:12,714] [DEBUG] [MainProcess] [DB:connect:111] Getting MongoDB connection to host11-mongo101.ubisoft.onbe:10000,host11-mongo102.ubisoft.onbe:10000,host11-mongo103.ubisoft.onbe:10000 (replicaSet=None, readPreference=primaryPreferred, readPreferenceTags=False, ssl=False)
    [2017-10-27 14:08:17,722] [ERROR] [MainProcess] [DB:connect:117] Unable to connect to host11-mongo101.ubisoft.onbe:10000,host11-mongo102.ubisoft.onbe:10000,host11-mongo103.ubisoft.onbe:10000! Error: No servers available
    [2017-10-27 14:08:17,723] [CRITICAL] [MainProcess] [Sharding:get_config_server:200] Unable to locate config servers using host11-mongo101.ubisoft.onbe:27017: No servers available!
    [2017-10-27 14:08:17,723] [CRITICAL] [MainProcess] [Main:exception:207] Problem getting shard/replica set info! Error: No servers available
    [2017-10-27 14:08:17,723] [INFO] [MainProcess] [Main:cleanup_and_exit:163] Starting cleanup procedure! Stopping running threads
    [2017-10-27 14:08:17,723] [INFO] [MainProcess] [Sharding:restore_balancer_state:141] Restoring balancer state to: False
    [2017-10-27 14:08:17,747] [DEBUG] [MainProcess] [Stage:close:59] Calling close on backup stage mongodb_consistent_backup.Archive with task Tar
    [2017-10-27 14:08:17,752] [DEBUG] [MainProcess] [DB:close:212] Closing connection to: host11-mongo101.ubisoft.onbe:27017
    [2017-10-27 14:08:17,753] [INFO] [MainProcess] [Main:cleanup_and_exit:193] Cleanup complete, exiting
    [2017-10-27 14:08:17,753] [INFO] [MainProcess] [Logger:rotate:81] Running rotation of log files
    [2017-10-27 14:08:17,753] [INFO] [MainProcess] [Logger:compress:60] Compressing log file: /var/log/mongodb-consistent-backup/backup.default.20171027_1332.log
    [2017-10-27 14:08:17,754] [DEBUG] [MainProcess] [Lock:release:30] Releasing exclusive lock on file: /tmp/mongodb-consistent-backup.lock
    [root@host11-mongo101 patrick]#

YML file is attached.

patricklarochelle commented 6 years ago

mongo-consistent-backup.txt

timvaillancourt commented 6 years ago

Hello @patricklarochelle,

This sounds related to issues people are seeing with non-replset-based Config Servers (SCCC mode) in 3.2 and older. There is an issue here to re-test the code with the old style config servers: https://github.com/Percona-Lab/mongodb_consistent_backup/issues/203 that I plan to action shortly.

Importantly, due to the nature of non-replset-based (SCCC mode) config servers not having any consistent changelog/oplog, the support for SCCC mode may be removed/deprecated in the future as we cannot guarantee a point of consistency without an oplog. Currently we "do our best" in SCCC mode but consistency is not guaranteed 100%. Seeing our tool is designed to create "consistent backups" this is of course confusing. See more about this here: https://github.com/Percona-Lab/mongodb_consistent_backup/issues/233.

As SCCC mode is supported as of today I will test and attempt to fix this problem, however I do suggest that you consider moving to 3.2+ with CSRS (replset-based) config servers as the support for SCCC may go away, plus there are tonnes of other benefits to using CSRS!

Stay tuned!

Tim