search

wamdam / backy2

backy2: Deduplicating block based backup software for ceph/rbd, image files and devices
http://backy2.com/
Other
195 stars 39 forks source link

Backy2 cleanup fails #55

Closed ghost closed 4 years ago

ghost commented 4 years ago

I'm running backy2 to backup ceph rbd images to a dedicated ceph storage radoswg.

After deleting backups older than 15 days, I ran "backy2 cleanup" to delete unused objects on radosgw.

Unfortunately the cleanup fails, returning the following error message:

root@osb1:~# backy2 cleanup INFO: $ /usr/bin/backy2 cleanup INFO: Cleanup-fast: 230 false positives, 769 data deletions. ERROR: Unexpected exception ERROR: timed out Traceback (most recent call last): File "/usr/lib/python3/dist-packages/backy2/scripts/backy.py", line 740, in main func(**func_args) File "/usr/lib/python3/dist-packages/backy2/scripts/backy.py", line 299, in cleanup backy.cleanup_fast() File "/usr/lib/python3/dist-packages/backy2/backy.py", line 674, in cleanup_fast no_del_uids = self.data_backend.rm_many(uid_list) File "/usr/lib/python3/dist-packages/backy2/data_backends/s3.py", line 198, in rm_many errors = self.bucket.delete_keys(uids, quiet=True) File "/usr/local/lib/python3.6/dist-packages/boto/s3/bucket.py", line 728, in delete_keys while delete_keys2(headers): File "/usr/local/lib/python3.6/dist-packages/boto/s3/bucket.py", line 717, in delete_keys2 body = response.read() File "/usr/local/lib/python3.6/dist-packages/boto/connection.py", line 410, in read self._cached_response = http_client.HTTPResponse.read(self) File "/usr/lib/python3.6/http/client.py", line 466, in read return self._readall_chunked() File "/usr/lib/python3.6/http/client.py", line 573, in _readall_chunked chunk_left = self._get_chunk_left() File "/usr/lib/python3.6/http/client.py", line 556, in _get_chunk_left chunk_left = self._read_next_chunk_size() File "/usr/lib/python3.6/http/client.py", line 516, in _read_next_chunk_size line = self.fp.readline(_MAXLINE + 1) File "/usr/lib/python3.6/socket.py", line 586, in readinto return self._sock.recv_into(b) socket.timeout: timed out INFO: Backy failed.

I'm running the latest version of backy2 (2.10.5) on Ubuntu Bionic.

Does anybody else have that issue?

ghost commented 4 years ago

Forgot: backup target runs on latest Ceph nautilus.

wamdam commented 4 years ago

The timeout happens on your s3 data target. Just run cleanup again when s3 is working.