ThomasWaldmann
commented
6 years ago Closed ThomasWaldmann closed 6 years ago
ThomasWaldmann
commented
6 years ago ThomasWaldmann
commented
6 years ago i did a local test with the stretch64 machine in master branch and a pyenv-built 3.7.0rc1 and still see some of these errors. either something in py37 is broken or borg and py37 are somehow not compatible.
eli-schwartz
commented
6 years ago Arch Linux is currently rebuilding the world for the final python 3.7 release. Here's a log of the testsuite: https://paste.xinu.at/wEIaO/
Errors excerpt:
=================================== FAILURES ===================================
_____________________ ArchiverTestCase.test_common_options _____________________
self = <borg.testsuite.archiver.ArchiverTestCase testMethod=test_common_options>
def test_common_options(self):
self.create_test_files()
self.cmd('init', '--encryption=repokey', self.repository_location)
log = self.cmd('--debug', 'create', self.repository_location + '::test', 'input')
> assert 'security: read previous location' in log
E AssertionError: assert 'security: read previous location' in 'using builtin fallback logging configuration\n'
borg/testsuite/archiver.py:2025: AssertionError
__________________ ArchiverTestCase.test_corrupted_repository __________________
self = <borg.testsuite.archiver.ArchiverTestCase testMethod=test_corrupted_repository>
def test_corrupted_repository(self):
self.cmd('init', '--encryption=repokey', self.repository_location)
self.create_src_archive('test')
self.cmd('extract', '--dry-run', self.repository_location + '::test')
output = self.cmd('check', '--show-version', self.repository_location)
self.assert_in('borgbackup version', output) # implied output even without --info given
self.assert_not_in('Starting repository check', output) # --info not given for root logger
name = sorted(os.listdir(os.path.join(self.tmpdir, 'repository', 'data', '0')), reverse=True)[1]
with open(os.path.join(self.tmpdir, 'repository', 'data', '0', name), 'r+b') as fd:
fd.seek(100)
fd.write(b'XXXX')
output = self.cmd('check', '--info', self.repository_location, exit_code=1)
> self.assert_in('Starting repository check', output) # --info given for root logger
E AssertionError: 'Starting repository check' not found in 'Data integrity error: Segment entry checksum mismatch [segment 4, offset 8]\nIndex object count mismatch.\ncommitted index: 126 objects\nrebuilt index: 0 objects\nID rebuilt index committed index \n0000000000000000000000000000000000000000000000000000000000000000 <not found> (4, 3717805) \n345046f023114df709fefdf9ed3e8171859de0afe762597ce3c4fb4dbf01860d <not found> (4, 1003843) \n13b660a93fc3d3332d17776fce7257aca9cadf7a6b0cac39f08ce04be5c6f5c6 <not found> (4, 8) \n14c8b73bba2c42cddc7f9cdcebfb98e989078e66714a3ee7a13d34933e794f21 <not found> (4, 1522395) \nefab4a77155c2f3f89933c441f6594d39c46ed4ca258643fe69377310f6328af <not found> (4, 3144932) \n28b1a3e69d22e7d12110e445982bf7df3d8a5777f61c57fd8346bd94eb46f989 <not found> (4, 1057530) \nf6d8da58ed51ffcc77f74a9fa12fb7687d8eb1752f0145e275e84e411ff0b03f <not found> (4, 3400689) \n7aa73fe24b3145478a80bb3c17b3772008e7a4251bd4f6c5c8335fee1a5bc103 <not found> (4, 1080821) \n7dbd4eb812f75d7200fc5958a8d59a046cbee0971fdeaf04095f22b2b9f83116 <not found> (4, 1316624) \na38fb0861f3ed036a7e9f7ccc77a10da6521a792c46576d1fe787cab651a053a <not found> (4, 464428) \ne78d1b98d83fcddb0b10cdeda5b10b12baa72aacc12e0599380ee51c1951462d <not found> (4, 997343) \n1c5e783b62e099c8454a488d31990641dc3431a9cf2f0954e2d6f633a446472b <not found> (4, 1146808) \n49ce3de4acf057025c904a0f2132d482b1d2807e3170411bc37ba0252789c6a9 <not found> (4, 3431277) \n1f051a0c6b9c7b117137ab4af0ed1ef165f39b176caaa14ecc00fb0e575e1bcb <not found> (4, 3536091) \n0a811b4c43439b8e13c9d7839f5a4ca6ca63465bb5fdf7b06e43da81124a2f93 <not found> (4, 2445536) \n3a71a8a7330b8999b6eb24bc0ca7c467a08ee487c9b083fc0b9f3b46eb8f719c <not found> (4, 1498651) \n811120405b238eb7746fca504f0355a18f2c6ac04f0fb955bc4a8e634f28c320 <not found> (4, 1053721) \n29d6021ea444635e189c4904bf2ecac7193aed2c9952446ac6a19c849f303433 <not found> (4, 3402532) \nf038cfead9be72dab5077c23a0ebdc567e5cd75c844c0d75c34b92af2f84db21 <not found> (4, 113629) \n4325fc3d8d388d06b4791a8520522a0bb2c63aad27f068bb4ec0148a2bff41a4 <not found> (4, 110733) \n32705f06c23cf20fe419d0f90f4140474c4b4ba849b879d12f2465eba2b77d03 <not found> (4, 1085588) \n6cd100fe9bd1b77cc5e8bbecbe94141c825463c9073623df695fe1f692b3777d <not found> (4, 1039965) \n471d61abe915a5c20dfa5e053e49037c4cf6a71be6b439e231b3c4ee2c7e149f <not found> (4, 1337181) \n6fc4b1dc2990e952038c9841bf7dfdb82603d446346f69a8a6d88dac1e0681ff <not found> (4, 1018630) \n1e00ba8e989ffbdc2cb651aaac8a501777b745f1b70e09ce054e4b175d2d7426 <not found> (4, 3707280) \nd7f12f06854e7808b72846708615996c9b1fd80f4e2f50a4490340f504633885 <not found> (4, 3544419) \n13981c0a37fce34935251ba7dafdfc3d9d8835b224612309de22869a94a3f112 <not found> (4, 3692810) \nb00390823ef45ffa452b499a507e80297d94d0f1a6f8a419ff9cae2007671907 <not found> (4, 3553966) \n1962e377a766717db33624c1082ffb22550b0049a5d3f3a3c44a2218367c0c48 <not found> (4, 1162566) \n8c55e6b8842da27674ea725881d5a3e1cbf1bb9963a182f9f55e8dddaf46481a <not found> (4, 3696924) \nd233c6b6b20b836e5294570c8014abbdc5a8af52dbfd3a322d62887818033daf <not found> (4, 1910781) \nfbb96bf4168337d43ff31d69cb05cb9ee5a53cc598c4b71fd4e8e29b33ece6dd <not found> (4, 1854140) \n224bc2b43835dc1facb56fcd985e3ba5884ed5d2b9374c7d7a054b18310c6980 <not found> (4, 83996) \n96b6b3b945819a64f20272cd0400132878a44def35cb2860351ef0aa6199d528 <not found> (4, 108075) \n9f9ed84401f5a0c8ae08a3624627e3a21b74d6fa964c520975b2e273a68ced55 <not found> (4, 1049337) \nf38644dc3d47a16bf256b8b97ce7bd5a34cc42ff48149e71bcbc6220a5ade8c4 <not found> (4, 182376) \nc676c50b130b02b184d4e14009e780f9cebe182a4bc2543e06ba2a6d55d3c61a <not found> (4, 1406109) \ne48be54b957e764f495fe4a011fab3eca63065d918e6031081633b874c09add5 <not found> (4, 469453) \n2fe99459a4c38c0cd3cc0687fb72a54bc95acedfaf60bee7e8e8ed4102bb61ae <not found> (4, 193541) \nd0304f17f0fc68b02546527bdfa20198632bc81ab95d6bf709f205322c38f1a1 <not found> (4, 1412319) \n4dc4259c9f8c55ed0564f0ace955b974b6aef5be9682baefe3fc516db74e20ba <not found> (4, 1033114) \n43d1de059905392951a9f58e46bbdc316490305c1e27aa96c33f157694b27443 <not found> (4, 1087461) \nb5042b1ed21a8c6c1cbd564f723eca42bd582f40b88af48dda88bb6b2db519aa <not found> (4, 1482474) \n86084d6e3542609acf578042721747694d0e09107544e598ea11677cece736e8 <not found> (4, 286880) \nc1e5b558372c5ef107b835d14ccfcebc346b06361d920bd2d296f67589759117 <not found> (4, 1479845) \nafc4b7ab74c14524b00d2975aaa1aec61b866848b1433170b84f0b854f7c4ba3 <not found> (4, 1496589) \nf0188f35743fc61299e4d08a7ee422d2f0c5fb5b607ffae28086cbf86c25ed8e <not found> (4, 1284809) \n25ada8cbb76fe3eacf55940b47412b21cdfc2c2c024590359aa9da05010e13dc <not found> (4, 1149568) \n25106324cff4d5064455d67e44e773285eaec0d096d8545515f0dce180195b2f <not found> (4, 996441) \n782d5a4209d1eb025f0a09a2af9fae3a84a02d460b477e06dea2b324f1173d85 <not found> (4, 106716) \ne6643840f546249bd4982e72080c04cf1ba234d116330b855e44dbc404d9ef83 <not found> (4, 1141266) \n8ef68d5a9cb979f69f80fd8e5cf2ea5ba8367ad52c6eb396f22770bf5a3b33c4 <not found> (4, 3547564) \nd68929ee01e6c974ac9ba81898407ccee33f58b1ce0de947e2f6633170c7bd6b <not found> (4, 1516492) \ndc6cc3e16871309e05e97e77262e4350208c4bf4cf559b09ef30e705209b621c <not found> (4, 102777) \nae84355e0bfc31411749800dfe8f14a20df92759b47980172cb278ef6a8a1232 <not found> (4, 1016579) \ndc6b9f5deea7f4648c69034da57d3e396884615ea4e5c8ea7e43782bb5c148b8 <not found> (4, 3400936) \n4b734bfd3009c7f62fafeda4a855927637d58aeaa79fed8a97f5447798810c08 <not found> (4, 1059673) \nb32b3952ef250390da75d0836cd9d0d298b7306173dab9feeef34d79c5684767 <not found> (4, 1038599) \n908ec843f0c435681959238248ca2997e4276a3a19617a9c7fbc5dbb79f03347 <not found> (4, 63013) \nfb98a84584a20fd6cbeaa25f6a7bf93124bb848dcc674a3d9d63b51176d47a65 <not found> (4, 195221) \ne7ff1b6ec36421d088d5f82a94574f22fb581470a840c1bedc4d96394f1fd29f <not found> (4, 105934) \nd957cfb1ba437cb0fc892800d0ae328cceb660ca3f1ae4ab599d58f3c13fe9d8 <not found> (4, 1492393) \ncc9c6588bc4ddcb63071c1f039e8027a07dbfd9bdff45041d8433310d9bd0b4a <not found> (4, 1361971) \n8c528d5ff5540f5ededa1dc194e99ad019e27f0adb78c0c0e0fb0d41d421eff1 <not found> (4, 1418555) \ne529e4d02de908c5ebb07499cf640f55b3a37671e91d2e59720c70f30e9b9b21 <not found> (4, 1879689) \n5d9214e2f8dabae680d891000f60c5bda0312260e293db49113f2e63a62a84e2 <not found> (4, 1445914) \n3966080a8bf2090b97cd05c96a2d04f151103bdba3c86d6131d53c97f8843d0e <not found> (4, 3717266) \nc726855984e16e878d9810addeee51835113c55a5507153536dfc6ae7975160c <not found> (4, 1871577) \nf02250caf5861d5452ec7f5e3901570c54ae1a782651c53216fdceba513a1129 <not found> (4, 1438019) \ndae8a4155aab0ddbe98e2e40530b1806cb0934ee9b207ee3dda4ca701532ee5c <not found> (4, 287237) \n1c5d70af0c9fcb5d0267a4f7db9195dd0c4c917f4e98df2f55bb948b76c16d6f <not found> (4, 3600051) \n5aa4042cafeba7724ab1d49985d4a12341492f620fffa6090c933bb0be162ac3 <not found> (4, 280616) \n1e958bff28639832561ffd5fff99c372571a866db1e60deeac45f1e2e4257daf <not found> (4, 40803) \nf4ca6b557b171304ba54826f6b007f289368dba3c378846270b108061b3db78a <not found> (4, 187348) \n86be3b9bb7da793afef3e2212e9767474ba06e03abebac48fda6d835fe42dc2b <not found> (4, 1478003) \n27b786dc79330cbb3eeae9ed6d2f4f1fd00e485e78ffc5cb1d7f023c28a51383 <not found> (4, 108885) \ncedc79351e25e913f1c2c58f768ae7de210d97e05e0d6d9b04b8b3b933b8d825 <not found> (4, 1884794) \n8b78fdc61b9e3694324d202ac0871f8eecb71b2a1c975fca52f28b0b5dbb3adf <not found> (4, 3482960) \n5c54cd5d1986742de1c6d3af7756604db25d05b0f5a853486f83718e066488ce <not found> (4, 1063370) \nd04c64157070f79bbd4d7aa98a0466323c6fac2ff458d102dacaf1eb71f0351a <not found> (4, 1076062) \n26b216b836940c9b014959dfacd13f60b4a147c4f9114922304d9654edc9fe5d <not found> (4, 1512333) \nddb2d4f5ab6a964d5dd97a57cec1dc0c5dfd31fde2c1912b52eaff5fa64fa667 <not found> (4, 1182953) \nda70fbe79bfcfeb1fed1beb75772eedb64d768db4e84079be549e5529894b835 <not found> (4, 3699539) \nc7b52bd09522659de1b0da746b66cb376efde533a4b7d90f3c9fd136c961d060 <not found> (4, 1340450) \n66c78170a106eeff0961e551c66970c491560f4781c886201ca8618f42823591 <not found> (4, 3481381) \n4fe55f75f8cbdeaa4d616d22fc19a1a551a661684a58e11d4cda241a58492ec2 <not found> (4, 1030347) \n471fab706454aef36fad3dff4c4d77b8bfc060b0eb531885229e2ed59fecb0c5 <not found> (4, 1079986) \n0ac2751cae5823585de40c98a0afdd47d1b7de57a83bb6bd4dc985c3eccda650 <not found> (4, 1015299) \nc6b267c0f31460257b2de2f836469de3ec5870211e26d141196a0bdf83a06a58 <not found> (4, 1011462) \n6aaed8a2f5b6c848cba6e3bdf66213d5daab445822eeacf2878da80704a53995 <not found> (4, 1414591) \n684dbdad333da537a5c2600827f1f624e4f1689f47c61da77fdda0d51eda8183 <not found> (4, 1354095) \n33b864cb2b312d495c7fac826f2c291a57dad7b777f758db084f0129887f140e <not found> (4, 1475260) \n6aa06f7d407acad6bc65b0fbe638dbe4993fd4d9834fb18ea56d0c392c3f54d2 <not found> (4, 1173672) \n651e4ec0e6c4b110f73cddf9195e94d19fb275d6f8432f0cb6b07fd32847f8a6 <not found> (4, 1874048) \n520e1096b411645aa07f4dc04621ba20904338ffa3fe456f337082dd3e15ac79 <not found> (4, 1420771) \n19250790aaea149398a6becc43ebb5875073b5c78311441b69e676e12e3ac8a9 <not found> (4, 463911) \nd0803dbbc450af7ed3a61371a3a275b10e7def88424a47b33de54d727ff5b1ba <not found> (4, 1047146) \n27df4cdc10bebdd21fdaa34db561804cd8517cb11d2e3fc3e1b7212870fb38a3 <not found> (4, 1058964) \ne83e56a1e7a14358d56cd8ed383c5650748aa62f65397707739b11e821fa70dd <not found> (4, 1426519) \n0cf2f899b1e26d74215b5e1bceb153e0c434aa7ca430b46d5bc80287daba9831 <not found> (4, 1301160) \n0b6b12ead5f42f93a6f4bed3ca876100a4683ab4a99b0490b4096d8061a2e15c <not found> (4, 1046545) \n9d550583c5670c07e3fff03377b45461fad266983495d83cca81a2b15fb86fdf <not found> (4, 195027) \nf5a0cf724df50d45efa1a795d60cdd86a9819d9b5c4721d963002161f00c9985 <not found> (4, 2782486) \n21e020752f1ddc377dc15e16054be8e46e7caa7039082202427d23aec301f4f9 <not found> (4, 1084100) \n1c04f416bb5bb77435710a4e29f7b587bd6209086f8cb3bb2c534fa99f4dccd6 <not found> (4, 149187) \nb70409beef1e57b05e395043ff863bbb6a2d644cd894fd22b49eaa09cb2b677f <not found> (4, 1518894) \n89076990c9d5ec24c6d0cfa64b020be85a1db6cb545af142dee75c0793574784 <not found> (4, 3591867) \n04ff36ea1303848a2f3c51a74e01eecaa4f70823f001cfe1d197df3178af1fa0 <not found> (4, 132) \n6342a43806a6ada5c7b3e568adf3bbab4e6e8be154e273d0fad3ef45b4c72d61 <not found> (4, 556584) \nc5878e6520f6714a157a1acab6185d3c4ce49544a15b11f3f72f379c64bb0c69 <not found> (4, 3426260) \nb7484f210535d5d2da0d640329e5ab2d83485c8f3d43f0a6821601a48976c514 <not found> (4, 3399936) \na045f466857cbc6d661e9370cdf5e7d681e61a2c6f815748d4af1b47efc1e3ef <not found> (4, 89526) \nc93bd5ad56d2ed43f70a78f8ce780208086df9d0bd7a2236d0e23dc380d9de71 <not found> (4, 468852) \nd2e0ddaf58566ad355ca4dd01e1d90295988815791c6f28481b2bb7618651a75 <not found> (4, 1350597) \nbdc4e2bfce51b077af33fa8e64fe70eb831380e55e1d06ee49b65ca419a0d7a3 <not found> (4, 1155935) \n43b24e7b6c5f06c9507b54727f05d201e1143c0322b9d56413259b86c4d71459 <not found> (4, 1056885) \nd8a67aa989db13a0c520f59b3536e7da987d5484f7dcb82cb6d853a5c746e609 <not found> (4, 3229004) \n8addd383212cce732ab27a6c78c4f0339f6ed4b15bf7c08fa4a9532f84bd7b50 <not found> (4, 119992) \n7ae4b4a1e8d2e408ee7fd8ee11d4f3b05a30e2611a0d1134ea088741fb943832 <not found> (4, 3506226) \n8670356437aab40620aaedffff02179744328dd0a400ad62a294c9f7c056197d <not found> (4, 1853904) \n6b9bf5973f0e0485f17ae9e1e5049f6aa8e97b1472a91a13cc00dacd98f7f153 <not found> (4, 3545913) \ne4a8592bd584c4a08425ea6e67ba587334db6aa384a25102f6d77bb4b11f5f90 <not found> (4, 1002572) \nc2a12fed23464b145ff7310747a6c8551d6eb33649364043a15000d2ac4c6dcb <not found> (4, 1368030) \n60352646630ac8090c926c79b823f31dce20807115de1c404a06b9d3c251352f <not found> (4, 1307199) \n5217fe062a21d5487cbbfb3585f694f6ebd0e1ba7da647ba72cd38c3ca397f14 <not found> (4, 1309683) \n999770f264f934910f8e0e756fb5b322b097179142ac7b48aab0da7756a47639 <not found> (4, 1036562) \nCompleted repository check, errors found.\n'
borg/testsuite/archiver.py:1502: AssertionError
____________________ ArchiverCheckTestCase.test_check_usage ____________________
self = <borg.testsuite.archiver.ArchiverCheckTestCase testMethod=test_check_usage>
def test_check_usage(self):
output = self.cmd('check', '-v', '--progress', self.repository_location, exit_code=0)
> self.assert_in('Starting repository check', output)
E AssertionError: 'Starting repository check' not found in 'Checking segments 0.0%\rChecking segments 16.7%\rChecking segments 33.3%\rChecking segments 50.0%\rChecking segments 66.7%\rChecking segments 83.3%\r\r'
borg/testsuite/archiver.py:2944: AssertionError
________________ ArchiverCheckTestCase.test_corrupted_manifest _________________
self = <borg.testsuite.archiver.ArchiverCheckTestCase testMethod=test_corrupted_manifest>
def test_corrupted_manifest(self):
archive, repository = self.open_archive('archive1')
with repository:
manifest = repository.get(Manifest.MANIFEST_ID)
corrupted_manifest = manifest + b'corrupted!'
repository.put(Manifest.MANIFEST_ID, corrupted_manifest)
repository.commit()
self.cmd('check', self.repository_location, exit_code=1)
output = self.cmd('check', '-v', '--repair', self.repository_location, exit_code=0)
> self.assert_in('archive1', output)
E AssertionError: 'archive1' not found in "'check --repair' is an experimental feature that might result in data loss.\nType 'YES' if you understand this and want to continue: YES (from BORG_CHECK_I_KNOW_WHAT_I_AM_DOING)\nRepository manifest is corrupted: Data integrity error: Chunk (unknown): Encryption envelope checksum mismatch\nArchive consistency check complete, problems found.\n"
borg/testsuite/archiver.py:3053: AssertionError
_________ ArchiverCheckTestCase.test_manifest_rebuild_corrupted_chunk __________
self = <borg.testsuite.archiver.ArchiverCheckTestCase testMethod=test_manifest_rebuild_corrupted_chunk>
def test_manifest_rebuild_corrupted_chunk(self):
archive, repository = self.open_archive('archive1')
with repository:
manifest = repository.get(Manifest.MANIFEST_ID)
corrupted_manifest = manifest + b'corrupted!'
repository.put(Manifest.MANIFEST_ID, corrupted_manifest)
chunk = repository.get(archive.id)
corrupted_chunk = chunk + b'corrupted!'
repository.put(archive.id, corrupted_chunk)
repository.commit()
self.cmd('check', self.repository_location, exit_code=1)
output = self.cmd('check', '-v', '--repair', self.repository_location, exit_code=0)
> self.assert_in('archive2', output)
E AssertionError: 'archive2' not found in "'check --repair' is an experimental feature that might result in data loss.\nType 'YES' if you understand this and want to continue: YES (from BORG_CHECK_I_KNOW_WHAT_I_AM_DOING)\nRepository manifest is corrupted: Data integrity error: Chunk (unknown): Encryption envelope checksum mismatch\nSkipping corrupted chunk: Data integrity error: Chunk 25da34b42c0017747af714518ed27845ffccd90826bf8e6d7494da6d2e089b9a: Encryption envelope checksum mismatch\n1 orphaned objects found!\nArchive consistency check complete, problems found.\n"
borg/testsuite/archiver.py:3070: AssertionError
________________ ArchiverCheckTestCase.test_missing_file_chunk _________________
self = <borg.testsuite.archiver.ArchiverCheckTestCase testMethod=test_missing_file_chunk>
def test_missing_file_chunk(self):
archive, repository = self.open_archive('archive1')
with repository:
for item in archive.iter_items():
if item.path.endswith('testsuite/archiver.py'):
valid_chunks = item.chunks
killed_chunk = valid_chunks[-1]
repository.delete(killed_chunk.id)
break
else:
self.fail('should not happen')
repository.commit()
self.cmd('check', self.repository_location, exit_code=1)
output = self.cmd('check', '--repair', self.repository_location, exit_code=0)
self.assert_in('New missing file chunk detected', output)
self.cmd('check', self.repository_location, exit_code=0)
output = self.cmd('list', '--format={health}#{path}{LF}', self.repository_location + '::archive1', exit_code=0)
self.assert_in('broken#', output)
# check that the file in the old archives has now a different chunk list without the killed chunk
for archive_name in ('archive1', 'archive2'):
archive, repository = self.open_archive(archive_name)
with repository:
for item in archive.iter_items():
if item.path.endswith('testsuite/archiver.py'):
self.assert_not_equal(valid_chunks, item.chunks)
self.assert_not_in(killed_chunk, item.chunks)
break
else:
self.fail('should not happen')
# do a fresh backup (that will include the killed chunk)
with patch.object(ChunkBuffer, 'BUFFER_SIZE', 10):
self.create_src_archive('archive3')
# check should be able to heal the file now:
output = self.cmd('check', '-v', '--repair', self.repository_location, exit_code=0)
> self.assert_in('Healed previously missing file chunk', output)
E AssertionError: 'Healed previously missing file chunk' not found in "'check --repair' is an experimental feature that might result in data loss.\nType 'YES' if you understand this and want to continue: YES (from BORG_CHECK_I_KNOW_WHAT_I_AM_DOING)\n"
borg/testsuite/archiver.py:2999: AssertionError
_________________ ArchiverCheckTestCase.test_missing_manifest __________________
self = <borg.testsuite.archiver.ArchiverCheckTestCase testMethod=test_missing_manifest>
def test_missing_manifest(self):
archive, repository = self.open_archive('archive1')
with repository:
repository.delete(Manifest.MANIFEST_ID)
repository.commit()
self.cmd('check', self.repository_location, exit_code=1)
output = self.cmd('check', '-v', '--repair', self.repository_location, exit_code=0)
> self.assert_in('archive1', output)
E AssertionError: 'archive1' not found in "'check --repair' is an experimental feature that might result in data loss.\nType 'YES' if you understand this and want to continue: YES (from BORG_CHECK_I_KNOW_WHAT_I_AM_DOING)\nRepository manifest not found!\nArchive consistency check complete, problems found.\n"
borg/testsuite/archiver.py:3040: AssertionError
_________________ ManifestAuthenticationTest.test_not_required _________________
self = <borg.testsuite.archiver.ManifestAuthenticationTest testMethod=test_not_required>
def test_not_required(self):
self.cmd('init', '--encryption=repokey', self.repository_location)
self.create_src_archive('archive1234')
repository = Repository(self.repository_path, exclusive=True)
with repository:
shutil.rmtree(get_security_dir(bin_to_hex(repository.id)))
_, key = Manifest.load(repository, Manifest.NO_OPERATION_CHECK)
key.tam_required = False
key.change_passphrase(key._passphrase)
manifest = msgpack.unpackb(key.decrypt(None, repository.get(Manifest.MANIFEST_ID)))
del manifest[b'tam']
repository.put(Manifest.MANIFEST_ID, key.encrypt(msgpack.packb(manifest)))
repository.commit()
output = self.cmd('list', '--debug', self.repository_location)
assert 'archive1234' in output
> assert 'TAM not found and not required' in output
E AssertionError: assert 'TAM not found and not required' in 'using builtin fallback logging configuration\narchive1234 Wed, 2018-07-04 14:01:40 [111123cc3edf41dc9041e76cf6c5ffa53d2253ab6a65b7553ff0cf147532956a]\n'
borg/testsuite/archiver.py:3218: AssertionError
__________________ RemoteArchiverTestCase.test_common_options __________________
self = <borg.testsuite.archiver.RemoteArchiverTestCase testMethod=test_common_options>
def test_common_options(self):
self.create_test_files()
self.cmd('init', '--encryption=repokey', self.repository_location)
log = self.cmd('--debug', 'create', self.repository_location + '::test', 'input')
> assert 'security: read previous location' in log
E AssertionError: assert 'security: read previous location' in 'using builtin fallback logging configuration\nRemote: using builtin fallback logging configuration\nRemote: 35 self t...lated working space for compact as -33 bytes\nRemote: check_free_space: required bytes 169135, free bytes 8312791040\n'
borg/testsuite/archiver.py:2025: AssertionError
_________________ ArchiverCorruptionTestCase.test_cache_files __________________
self = <borg.testsuite.archiver.ArchiverCorruptionTestCase testMethod=test_cache_files>
def test_cache_files(self):
self.cmd('create', self.repository_location + '::test', 'input')
self.corrupt(os.path.join(self.cache_path, 'files'))
out = self.cmd('create', self.repository_location + '::test1', 'input')
# borg warns about the corrupt files cache, but then continues without files cache.
> assert 'files cache is corrupted' in out
E AssertionError: assert 'files cache is corrupted' in ''
borg/testsuite/archiver.py:3345: AssertionError
________________ ArchiverCorruptionTestCase.test_chunks_archive ________________
self = <borg.testsuite.archiver.ArchiverCorruptionTestCase testMethod=test_chunks_archive>
def test_chunks_archive(self):
self.cmd('create', self.repository_location + '::test1', 'input')
# Find ID of test1 so we can corrupt it later :)
target_id = self.cmd('list', self.repository_location, '--format={id}{LF}').strip()
self.cmd('create', self.repository_location + '::test2', 'input')
# Force cache sync, creating archive chunks of test1 and test2 in chunks.archive.d
self.cmd('delete', '--cache-only', self.repository_location)
self.cmd('info', self.repository_location, '--json')
chunks_archive = os.path.join(self.cache_path, 'chunks.archive.d')
assert len(os.listdir(chunks_archive)) == 4 # two archives, one chunks cache and one .integrity file each
self.corrupt(os.path.join(chunks_archive, target_id + '.compact'))
# Trigger cache sync by changing the manifest ID in the cache config
config_path = os.path.join(self.cache_path, 'config')
config = ConfigParser(interpolation=None)
config.read(config_path)
config.set('cache', 'manifest', bin_to_hex(bytes(32)))
with open(config_path, 'w') as fd:
config.write(fd)
# Cache sync notices corrupted archive chunks, but automatically recovers.
out = self.cmd('create', '-v', self.repository_location + '::test3', 'input', exit_code=1)
> assert 'Reading cached archive chunk index for test1' in out
E AssertionError: assert 'Reading cached archive chunk index for test1' in 'Cached archive chunk index of test1 is corrupted: File failed integrity check: /tmp/tmp6pnmxa1f/cache/30a6c025b272935...c98ca2517321ebaced30dbb6fd/chunks.archive.d/65f7a4569f6f41db53008e0af6c787fabd39c744cfc9522c53675ddedd7be277.compact\n'
borg/testsuite/archiver.py:3372: AssertionError
____________ ArchiverCorruptionTestCase.test_old_version_interfered ____________
self = <borg.testsuite.archiver.ArchiverCorruptionTestCase testMethod=test_old_version_interfered>
def test_old_version_interfered(self):
# Modify the main manifest ID without touching the manifest ID in the integrity section.
# This happens if a version without integrity checking modifies the cache.
config_path = os.path.join(self.cache_path, 'config')
config = ConfigParser(interpolation=None)
config.read(config_path)
config.set('cache', 'manifest', bin_to_hex(bytes(32)))
with open(config_path, 'w') as fd:
config.write(fd)
out = self.cmd('info', self.repository_location)
> assert 'Cache integrity data not available: old Borg version modified the cache.' in out
E AssertionError: assert 'Cache integrity data not available: old Borg version modified the cache.' in 'Repository ID: 8cf3479a67f4dd5ab4f8cbd675c1db32da964d4a74ad5a227a20fb2a20bbd061\nLocation: /tmp/tmpju4hp9l1/repositor... Unique chunks Total chunks\nChunk index: 0 0\n'
borg/testsuite/archiver.py:3387: AssertionError
----------- coverage: platform linux, python 3.7.0-final-0 -----------
Name Stmts Miss Cover
------------------------------------------------------
borg/__init__.py 4 0 100%
borg/__main__.py 2 2 0%
borg/_version.py 1 0 100%
borg/algorithms/__init__.py 0 0 100%
borg/archive.py 1296 191 85%
borg/archiver.py 1924 245 87%
borg/cache.py 765 73 90%
borg/constants.py 33 0 100%
borg/crypto/__init__.py 0 0 100%
borg/crypto/file_integrity.py 145 2 99%
borg/crypto/key.py 560 52 91%
borg/crypto/keymanager.py 170 9 95%
borg/crypto/nonces.py 56 1 98%
borg/fuse.py 418 418 0%
borg/helpers.py 1393 163 88%
borg/locking.py 268 23 91%
borg/logger.py 118 27 77%
borg/lrucache.py 39 0 100%
borg/nanorst.py 159 13 92%
borg/patterns.py 197 12 94%
borg/platform/__init__.py 23 8 65%
borg/platform/base.py 106 22 79%
borg/remote.py 606 78 87%
borg/repository.py 978 83 92%
borg/selftest.py 42 7 83%
borg/shellpattern.py 36 0 100%
borg/testsuite/__init__.py 215 54 75%
borg/testsuite/archive.py 171 2 99%
borg/testsuite/archiver.py 2859 414 86%
borg/testsuite/benchmark.py 61 32 48%
borg/testsuite/cache.py 146 0 100%
borg/testsuite/checksums.py 26 0 100%
borg/testsuite/chunker.py 33 0 100%
borg/testsuite/compress.py 131 3 98%
borg/testsuite/crypto.py 103 0 100%
borg/testsuite/file_integrity.py 100 2 98%
borg/testsuite/hashindex.py 390 0 100%
borg/testsuite/helpers.py 659 10 98%
borg/testsuite/item.py 116 0 100%
borg/testsuite/key.py 297 0 100%
borg/testsuite/locking.py 199 0 100%
borg/testsuite/logger.py 40 0 100%
borg/testsuite/lrucache.py 51 0 100%
borg/testsuite/nanorst.py 21 0 100%
borg/testsuite/nonces.py 166 2 99%
borg/testsuite/patterns.py 134 0 100%
borg/testsuite/platform.py 146 43 71%
borg/testsuite/remote.py 137 0 100%
borg/testsuite/repository.py 766 0 100%
borg/testsuite/shellpattern.py 20 0 100%
borg/testsuite/upgrader.py 85 0 100%
borg/testsuite/version.py 15 0 100%
borg/testsuite/xattr.py 50 34 32%
borg/upgrader.py 169 54 68%
borg/version.py 26 0 100%
borg/xattr.py 252 150 40%
------------------------------------------------------
TOTAL 16923 2229 87%
12 failed, 1104 passed, 225 skipped, 1 deselected, 2 xfailed in 578.25 seconds it seems to be a bug in python 3.7 due to caching in the logging module.
i reviewed the code, but could not find an obvious bug in caching or invalidation of the cache, but some must be there. i reverted the caching change locally and without caching our tests work as they used to work in 3.6.
ThomasWaldmann
commented
6 years ago Hmm, maybe part of the problem is also our test teardown method is clearing the internal loggerDict of logging before setting up logging from scratch.
I am currently testing whether this is needed at all, because without that, it seems to work on 3.6 and on 3.7.
ThomasWaldmann
commented
6 years ago changeset that introduced the problematic line: 53d0140bd523b404890dead6f42bffa30e6d4181
https://travis-ci.org/borgbackup/borg/jobs/360736109
note: 3.7.0a4+ might be rather old, like Jan 2018, currently python 3.7 is at b4 (soon rc1).
https://docs.python.org/3.7/whatsnew/3.7.html