Open otraupe opened 8 years ago
ThomasWaldmann
commented
8 years ago librelist seems to have problems now and then (I recently moved another mailing list away from there due to that).
So, try again later, contact support there, or just create a ticket about it here.
otraupe
commented
8 years ago Thank you for clarifying this, Thomas. So I will try here.
I have installed Attic on CentOS 6.7. CentOS requires Python 2.6 for its packet manager "yum". So I installed Python 3.4.3 with "make altinstall". Further I installed it as a shared library and put it in /usr/local/Python3/bin. Python 3.4.3 is on the path as python3 and python3.4. I took some of this advice from here: http://stackoverflow.com/questions/8087184/installing-python3-on-rhel
Further, Selinux is off, libacl-devel is installed, OpenSSL is > 1.0, python-msgpack is installed via "pip3.4 install msgpack-python", and finally Attic via "pip3.4 install Attic".
Initially I had forgotten about libacl-devel and selinux. I might have done those after the installation of Attic.
Then Attic hung on the initialization of my first backup repo. I then ran the test suite by "python3 -m attic.testsuite.run", which gave me the below output.
Did I do something wrong? Is the coexistance of python (Python 2.6) and python3 (Python 3.4) on the path a problem for attic?
Some of the errors below are supposedly due to permission problems, but this makes no sense to me, as I run the test suite as root (in the temp dir, obviously). In particular:
PermissionError: [Errno 1] b'/tmp/tmp3ml_3ov1/input/link1'
The dir "/tmp/tmp3ml_3ov1" doesn't exist, but other similar ones are there.
Thanks for any ideas! Ole
[root@backuper ole]# python3 -m attic.testsuite.run ...........EEE.....ss.....EE..EEE.Error reading segment 2 Index object count mismatch. 87 != 0 ....ss......EE........................sKey file "/tmp/tmplt2gfk6i/tmp_tmpmnlwzgia" created. Keep this file safe. Your data will be inaccessible without it. ..Remember your passphrase. Your data will be inaccessible without it. .Encryption NOT enabled. Use the "--encryption=passphrase|keyfile" to enable encryption. ...s....Error reading segment 1 Index object count mismatch. 6 != 3 Error reading segment 1 Index object count mismatch. 6 != 3 Error reading segment 1 attempting to recover /tmp/tmpllpnu5fj/repository/data/0/1 .....Error reading segment 0 Error reading segment 0 Error reading segment 0 attempting to recover /tmp/tmp3t4ltrwa/repository/data/0/0 Adding commit tag to segment 0 ..........Error reading segment 1 Index object count mismatch. 6 != 3 Error reading segment 1 Index object count mismatch. 6 != 3 Error reading segment 1 attempting to recover /tmp/tmp8sfnusbq/repository/data/0/1 .....Error reading segment 0 Error reading segment 0 Error reading segment 0 attempting to recover /tmp/tmpyc3nm4pp/repository/data/0/0 Adding commit tag to segment 0
Traceback (most recent call last): File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 435, in test_aes_counter_uniqueness_keyfile self.verify_aes_counter_uniqueness('keyfile') File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 422, in verify_aes_counter_uniqueness self.create_test_files() File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr _check(func(path, name, value, len(value) if value else 0, 0), path) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check raise OSError(get_errno(), path) PermissionError: [Errno 1] b'/tmp/tmp2hlwh3ja/input/link1'
Traceback (most recent call last): File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 438, in test_aes_counter_uniqueness_passphrase self.verify_aes_counter_uniqueness('passphrase') File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 422, in verify_aes_counter_uniqueness self.create_test_files() File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr _check(func(path, name, value, len(value) if value else 0, 0), path) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check raise OSError(get_errno(), path) PermissionError: [Errno 1] b'/tmp/tmp125a_5wt/input/link1'
Traceback (most recent call last): File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 172, in test_basic_functionality self.create_test_files() File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr _check(func(path, name, value, len(value) if value else 0, 0), path) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check raise OSError(get_errno(), path) PermissionError: [Errno 1] b'/tmp/tmp0x7tqa35/input/link1'
Traceback (most recent call last): File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 202, in test_repository_swap_detection self.create_test_files() File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr _check(func(path, name, value, len(value) if value else 0, 0), path) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check raise OSError(get_errno(), path) PermissionError: [Errno 1] b'/tmp/tmpe1f8m7bp/input/link1'
Traceback (most recent call last): File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 214, in test_repository_swap_detection2 self.create_test_files() File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr _check(func(path, name, value, len(value) if value else 0, 0), path) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check raise OSError(get_errno(), path) PermissionError: [Errno 1] b'/tmp/tmp52df34rs/input/link1'
Traceback (most recent call last): File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 435, in test_aes_counter_uniqueness_keyfile self.verify_aes_counter_uniqueness('keyfile') File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 422, in verify_aes_counter_uniqueness self.create_test_files() File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr _check(func(path, name, value, len(value) if value else 0, 0), path) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check raise OSError(get_errno(), path) PermissionError: [Errno 1] b'/tmp/tmp0zfss0ve/input/link1'
Traceback (most recent call last): File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 438, in test_aes_counter_uniqueness_passphrase self.verify_aes_counter_uniqueness('passphrase') File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 422, in verify_aes_counter_uniqueness self.create_test_files() File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr _check(func(path, name, value, len(value) if value else 0, 0), path) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check raise OSError(get_errno(), path) PermissionError: [Errno 1] b'/tmp/tmpx4aq34j5/input/link1'
Traceback (most recent call last): File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 172, in test_basic_functionality self.create_test_files() File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr _check(func(path, name, value, len(value) if value else 0, 0), path) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check raise OSError(get_errno(), path) PermissionError: [Errno 1] b'/tmp/tmp7m7dpqc1/input/link1'
Traceback (most recent call last): File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 202, in test_repository_swap_detection self.create_test_files() File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr _check(func(path, name, value, len(value) if value else 0, 0), path) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check raise OSError(get_errno(), path) PermissionError: [Errno 1] b'/tmp/tmpur53mpbb/input/link1'
Traceback (most recent call last): File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 214, in test_repository_swap_detection2 self.create_test_files() File "/usr/local/Python3/lib/python3.4/site-packages/attic/testsuite/archiver.py", line 165, in create_test_files xattr.setxattr(os.path.join(self.input_path, 'link1'), 'user.foo_symlink', b'bar_symlink', follow_symlinks=False) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 101, in setxattr _check(func(path, name, value, len(value) if value else 0, 0), path) File "/usr/local/Python3/lib/python3.4/site-packages/attic/xattr.py", line 36, in _check raise OSError(get_errno(), path) PermissionError: [Errno 1] b'/tmp/tmp3ml_3ov1/input/link1'
Ran 118 tests in 27.728s
FAILED (errors=10, skipped=6)
otraupe
commented
8 years ago Found this: https://github.com/jborg/attic/issues/201 The init is working now, the repo is on a cifs mount.
Backups seem to work, too. A lot of 5 M size files are created.
ThomasWaldmann
commented
8 years ago The tests should be run with fakeroot -u. Not sure if that has to do with your issue though.
It is no problem to have both python 2.x and 3.x installed, you just need to use python 3.x for attic.
otraupe
commented
8 years ago With fakeroot, it takes much longer, but the result seems to be the same. Very similar to what Whissi posted two weeks ago. Makes me worry, honestly.
use python 3.x for attic
This should be ensured by "pip3.4 install attic", right? The "attic" executable resides in "/usr/local/Python3/bin/".
otraupe
commented
8 years ago All detailed error reports seem to be related to the "xattr.py". Another dependency missing?
ThomasWaldmann
commented
8 years ago I assume you have libattr and -dev?
otraupe
commented
8 years ago [root@backuper ole]# yum list installed | grep attr attr.x86_64 2.4.44-7.el6 @anaconda-CentOS-201508042137.x86_64/6.7 libattr.x86_64 2.4.44-7.el6 @anaconda-CentOS-201508042137.x86_64/6.7 libattr-devel.x86_64 2.4.44-7.el6 @base
otraupe
commented
8 years ago I would like to ask the following questions:
Which tests are failing exactly, what do they do? Is this critical, and in what way? What might be the reason? What can I do?
My first backup attempt over night finally "failed", although it seems that a lot of data was backuped: "attic: Exiting with failure status due to previous errors" Does it have to do with the failed tests?
otraupe
commented
8 years ago And why is it so slow? Ok, that might have many reasons, but 16 GB took about 45 minutes. I intend to backup multiple TB of data. That's probably not working in the long run.
ThomasWaldmann
commented
8 years ago Tests: it seems like they can not set extended attributes (xattrs) on the files in /tmp. Maybe this is because your /tmp filesystem does not support xattrs or is mounted without enabling that support. It is not the test itself failing, but the test setup.
"attic: Exiting with failure status due to previous errors" is a warning, you have to read the log output above that to see what exactly did not work. Likely it could not access some file(s) due to permissions or so. That is often not a fatal error, your backup has reached its normal end.
ThomasWaldmann
commented
8 years ago Aside from all hardware, setup and data related factors, some of the slowness may be caused by attic's usage of zlib compression at level 6, by encryption (if you do not have hw accelerated AES) and by internal scheduling. The first two can slow down if your CPU is relatively slow and/or old.
If you intend to go multiple TBs, you likely will run into the issue that the target chunksize of attic is ~64KB, so multiple TBs create a lot of chunks, a huge chunk index and huge RAM consumption. I made the chunker configurable in my fork "borgbackup".
otraupe
commented
8 years ago Thanks for your help, Thomas!
@xattrs: that's what I thought first. But the respective dirs in /tmp don't even exist - probably DUE to the failed tests. Also, I tested for xattr support following this procedure: https://wiki.samba.org/index.php/Operating_system_requirements/File_system_support#Testing_your_filesystem No problems here.
@the log file: where do I find it? The output in the console is cut to several hundreds (?) of lines. No errors there as far as I can see.
ThomasWaldmann
commented
8 years ago Well, it seems rather clear after analyzing the traceback that there are xattr problems, because the setxattr call in the test setup fails all the time. So, did you check xattr support inside /tmp/?
About the log: redirect stdout and stderr to files so you are not limited by the scrollback buffer. The warning/error msg will be in there then.
otraupe
commented
8 years ago Yes, just checked it again. And again: the setxattr must fail on files and folders that aren't present. Right?
@Logs: I will do that.
ThomasWaldmann
commented
8 years ago You should test whether setting xattrs on existing files works.
Of course the tests clean up their test files after running, that's why you do not see them existing afterwards.
otraupe
commented
8 years ago This is the output of the test for working xattrs according to the Samba wiki page I quoted above. Never had any problems with that.
[root@backuper tmp]# touch test.txt [root@backuper tmp]# setfattr -n user.test -v test test.txt [root@backuper tmp]# setfattr -n security.test -v test2 test.txt [root@backuper tmp]# getfattr -d test.txt
user.test="test"
[root@backuper tmp]# getfattr -n security.test -d test.txt
security.test="test2"
otraupe
commented
8 years ago (Can I somehow disable this annoying auto-format stuff, here?)
otraupe
commented
8 years ago And: only the faild tests clean-up their files and folders? Because there are a lot of tmp... folders in /tmp:
drwx------. 3 root root 4096 Nov 25 16:59 tmp072yfnc5 drwx------ 3 root root 4096 Nov 25 19:31 tmp0nm7p64u -rw------- 1 root root 0 Nov 25 18:20 tmp262xfzi5 drwx------ 2 ole ole 4096 Nov 25 18:56 tmp2_cnwz2d drwx------ 2 root root 4096 Nov 25 19:31 tmp2vre6xyg drwx------ 2 root root 4096 Nov 25 17:08 tmp4fo1tks drwx------ 3 root root 4096 Nov 25 18:14 tmp6suxymd0 drwx------ 3 root root 4096 Nov 25 17:05 tmp781unq5 drwx------ 3 root root 4096 Nov 25 17:08 tmp8pum50n3 -rw------- 1 ole ole 0 Nov 25 19:11 tmp96x_sw0m drwx------. 2 root root 4096 Nov 25 16:59 tmpagsc7y8d drwx------ 2 root root 4096 Nov 25 18:21 tmpcpkj4svc -rw------- 1 root root 0 Nov 25 17:04 tmpg5586lhr -rw------- 1 root root 0 Nov 25 17:08 tmphp7ohcws drwx------ 2 root root 4096 Nov 25 17:05 tmphu3bnjqi drwx------ 3 ole ole 4096 Nov 25 18:56 tmpigk28jy4 drwx------ 2 ole ole 4096 Nov 25 19:12 tmpluke3721 -rw------- 1 root root 0 Nov 25 18:14 tmpmnlwzgia drwx------. 3 root root 4096 Nov 25 16:49 tmpq1_xoc1s -rw-------. 1 root root 0 Nov 25 16:49 tmpqva6ej30 -rw------- 1 root root 0 Nov 25 19:30 tmps7e926cm drwx------ 2 root root 4096 Nov 25 18:14 tmpsx2t2bkj drwx------ 3 root root 4096 Nov 25 18:21 tmpu3x99ax9 -rw-------. 1 root root 0 Nov 25 16:58 tmpv3_ykfjj drwx------. 2 root root 4096 Nov 25 16:49 tmpvtq0aco4 -rw------- 1 ole ole 0 Nov 25 18:55 tmpyv3u853s drwx------ 3 ole ole 4096 Nov 25 19:12 tmpyvypfzgi
otraupe
commented
8 years ago Thank you also for your explanations and tips regarding speed.
Attic currently runs on a VM (qemu-kvm) with 2 cores of a Xeon E5-2620 v2 @ 2.10GHz, 2 GB RAM, and a separate physical network inferface.
HW accelerated AES is out of the question, I guess, but encryption is not selected, if I remember the Attic init correctly. What do you mean by the internal scheduler?
However, currently I backup from a Samba share on the host machine, and to a Samba share on a NAS. So this might contribute to the slowness. On the other hand, doing the same on a Windows machine (having backup source and target on Samba shares from different machines; using some arbitrary copy software like the Total Commancer), I achieve a sufficient throughput of ~60 MB/s (on a GB network). So the network architecture shouldn't be much of an issue.
Anyways, I plan to migrate the Attic installation to the host system (file server hosting the zfs pool being backuped) once everything is ok (initlaly was concerned about the alternative python3 installation).
ThomasWaldmann
commented
8 years ago Internal scheduling: attic is single threaded. reading, processing, writing, syncing is done in-order.
Thus, I/O and I/O wait times are not used otherwise, no parallel processing. Maybe this is not the main cause of slowness for some people (see other factors), but if everything else is fast, this is a limiting factor.
otraupe
commented
8 years ago Ok, will see more when attic runs directly on the file server. Curently, I am down to <6 MB/s, which I can't explain by any of these factors.
Any more ideas regarding the failed test setup?
jdchristensen
commented
8 years ago You should really keep separate topics in separate issues, but about the speed: are you mounting a remote file system locally and then backing up to or from it? If so, that's almost certainly your problem. To be efficient, you need two copies of attic running, one as close as possible to the source files and one as close as possible to the backup location.
otraupe
commented
8 years ago You are totally right, I am sorry. I wasn't paying attention to that as not posting to the mailing list made me feel already out of place.
---------- SPEED DISCUSSION CLOSED ----------
So, how do I find out whether the failing tests pose any thread to my intended use of attic? And I don't actually mean some trial-and-error approach.
otraupe
commented
8 years ago Followed the advice here to comment out the culprit line about setting xattrs: https://github.com/jborg/attic/issues/361
Looks better now. Here is my new test output ("Ok", 8 tests skipped, due to encription not enabled? but with actual root rights, it is only 6 skipped):
[ole@backuper ~]$ fakeroot python3 -m attic.testsuite.run ...................ss.............Error reading segment 2 Index object count mismatch. 87 != 0 ....ss................................sKey file "/tmp/tmpd90ckw_j/tmp_tmpzrrbsr4q" created. Keep this file safe. Your data will be inaccessible without it. ..Remember your passphrase. Your data will be inaccessible without it. .Encryption NOT enabled. Use the "--encryption=passphrase|keyfile" to enable encryption. ...sss..Error reading segment 1 Index object count mismatch. 6 != 3 Error reading segment 1 Index object count mismatch. 6 != 3 Error reading segment 1 attempting to recover /tmp/tmpby8gc0xk/repository/data/0/1 .....Error reading segment 0 Error reading segment 0 Error reading segment 0 attempting to recover /tmp/tmpgavmpuuu/repository/data/0/0 Adding commit tag to segment 0 ..........Error reading segment 1 Index object count mismatch. 6 != 3 Error reading segment 1 Index object count mismatch. 6 != 3 Error reading segment 1 attempting to recover /tmp/tmp5ansxv5w/repository/data/0/1 .....Error reading segment 0 Error reading segment 0 Error reading segment 0 attempting to recover /tmp/tmpnuj33p4u/repository/data/0/0 Adding commit tag to segment 0
Ran 118 tests in 157.289s
OK (skipped=8)
I take it the 8x 's' in the '.......' output are the skipped ones?
What's the rest about?
otraupe
commented
8 years ago Thomas, as you are responding here and not the original developer of attic (you are talking about your fork), would you say that attic isn't actively developed at the moment?
Whissi
commented
8 years ago . are passed tests, E are failed tests and s are skipped tests. The output regarding "Error reading segment" shouldn't be a problem because theses tests are testing recovering from errors (="attempting to recover" succeed).
@otraupe: Read https://github.com/jborg/attic/issues/217. Compare commit log between https://github.com/jborg/attic/commits/master and https://github.com/borgbackup/borg/commits/master
Now decide on your own which project you would call more active.
otraupe
commented
8 years ago I see, thank you! Ole
Am 30.11.2015 um 14:22 schrieb Thomas D.:
|.| are passed tests, |E| are failed tests and |s| are skipped tests. The output regarding "Error reading segment" shouldn't be a problem because theses tests are testing recovering from errors (="attempting to recover" succeed).
@otraupe https://github.com/otraupe: Read #217 https://github.com/jborg/attic/issues/217. Compare commit log between https://github.com/jborg/attic/commits/master and https://github.com/borgbackup/borg/commits/master
Now decide on your own which project you would call more active.
— Reply to this email directly or view it on GitHub https://github.com/jborg/attic/issues/364#issuecomment-160625536.
unode
commented
8 years ago If you use anaconda I've packaged attic 0.16 at https://anaconda.org/unode/attic. Also had problems setting it up due to a broken compilation toolkit. Packaging via anaconda solved it for me.
Hi, I have problems with the installation and would like to post to the list "attic@librelist.com". I have sent multiple messages to it, but I don't get a response. Ole