Percona-Lab / mongodb_consistent_backup

A tool for performing consistent backups of MongoDB Clusters or Replica Sets
https://www.percona.com
Apache License 2.0
276 stars 82 forks source link

Backup failed: Could not acquire lock on file: /var/lib/.../mongodb-consistent-backup_META.lock #283

Open snake007v opened 5 years ago

snake007v commented 5 years ago

v1.4.0 Right after s3 upload:

Backup failed: Could not acquire lock on file: /var/lib/mongodb-consistent-backup/.../20181018_2118/mongodb-consistent-backup_META.lock!
Traceback (most recent call last):
  File "/home/ubuntu/.pex/install/mongodb_consistent_backup-1.4.0-py2-none-any.whl.7a32330efda4673a2eb16ddfcf02bfeef185ac4e/mongodb_consistent_backup-1.4.0-py2-none-any.whl/mongodb_consistent_backup/__init__.py", line 16, in run
    m.run()
  File "/home/ubuntu/.pex/install/mongodb_consistent_backup-1.4.0-py2-none-any.whl.7a32330efda4673a2eb16ddfcf02bfeef185ac4e/mongodb_consistent_backup-1.4.0-py2-none-any.whl/mongodb_consistent_backup/Main.py", line 441, in run
    self.stop_timer()
  File "/home/ubuntu/.pex/install/mongodb_consistent_backup-1.4.0-py2-none-any.whl.7a32330efda4673a2eb16ddfcf02bfeef185ac4e/mongodb_consistent_backup-1.4.0-py2-none-any.whl/mongodb_consistent_backup/Main.py", line 152, in stop_timer
    self.state.set('timers', self.timer.dump())
  File "/home/ubuntu/.pex/install/mongodb_consistent_backup-1.4.0-py2-none-any.whl.7a32330efda4673a2eb16ddfcf02bfeef185ac4e/mongodb_consistent_backup-1.4.0-py2-none-any.whl/mongodb_consistent_backup/State.py", line 63, in set
    self.write(True)
  File "/home/ubuntu/.pex/install/mongodb_consistent_backup-1.4.0-py2-none-any.whl.7a32330efda4673a2eb16ddfcf02bfeef185ac4e/mongodb_consistent_backup-1.4.0-py2-none-any.whl/mongodb_consistent_backup/State.py", line 68, in write
    self.lock.acquire()
  File "/home/ubuntu/.pex/install/mongodb_consistent_backup-1.4.0-py2-none-any.whl.7a32330efda4673a2eb16ddfcf02bfeef185ac4e/mongodb_consistent_backup-1.4.0-py2-none-any.whl/mongodb_consistent_backup/Common/Lock.py", line 26, in acquire
    raise OperationError("Could not acquire lock on file: %s!" % self.lock_file)
OperationError: Could not acquire lock on file: /var/lib/mongodb-consistent-backup/.../20181018_2118/mongodb-consistent-backup_META.lock!
root@mng:~# stat /var/lib/mongodb-consistent-backup/.../20181018_2118/mongodb-consistent-backup_META.lock
stat: cannot stat '/var/lib/mongodb-consistent-backup/.../20181018_2118/mongodb-consistent-backup_META.lock': No such file or directory

Configuration:

production:
  host: mongo-01.com,mongo-02.com,mongo-03.com
  port: 27017
  username: ...
  password: ...
  authdb: admin
  log_dir: /var/log/mongodb-consistent-backup
  backup:
    method: mongodump
    name: ...
    location: /var/lib/mongodb-consistent-backup
  rotate:
    max_backups: 1
    max_days: 0.15
  replication:
    max_lag_secs: 35 
  oplog:
    resolver:
      threads: 4 
    tailer:
      enabled: true
      status_interval: 30
  archive:
    method: tar
    tar:
      compression: gzip
      threads: 2
  notify:
    method: none
  upload:
    method: s3
    remove_uploaded: true
    s3:
      region: us-west-2
      access_key: ...
      secret_key: ...
      bucket_name: ...
      secure: false

Any ideas?

timvaillancourt commented 5 years ago

Hi @snake007v. No immediate ideas, could you upload a full debug/--verbose log? Also --version output is useful (git commit/branch).

You may want to find/replace any hostnames you don't want on GH. Thanks!

ttwthomas commented 5 years ago

I have the same issue. I think it is due to the remove_uploaded: true option. (with upload to s3) When I don't use this option the error is gone. I think the local folder is deleted before the lock check or something like that.