search

fair-research / bdbag

Big Data Bag Utilities
https://fair-research.org
Apache License 2.0
49 stars 23 forks source link

checksum tests fails on Windows 10 #11

Closed stain closed 6 years ago

stain commented 6 years ago

test_validate_complete_bag_full, test_resolve_fetch_ark, test_resolve_fetch_http and test_validate fail on Windows 10.

Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

with a fresh virtualenv enviroment and fresh pip installs.

Stacktrace

C:\Users\stain\src\bdbag (master)
(venv-master) λ python setup.py test
running test
running egg_info
writing requirements to bdbag.egg-info\requires.txt
writing bdbag.egg-info\PKG-INFO
writing top-level names to bdbag.egg-info\top_level.txt
writing dependency_links to bdbag.egg-info\dependency_links.txt
writing entry points to bdbag.egg-info\entry_points.txt
reading manifest file 'bdbag.egg-info\SOURCES.txt'
writing manifest file 'bdbag.egg-info\SOURCES.txt'
running build_ext
test_archive_bag_tar (test.test_api.TestAPI) ... ok
test_archive_bag_tgz (test.test_api.TestAPI) ... ok
test_archive_bag_zip (test.test_api.TestAPI) ... ok
test_create_bag (test.test_api.TestAPI) ... ok
test_create_bag_with_config (test.test_api.TestAPI) ... ok
test_extract_bag_archive_tar (test.test_api.TestAPI) ... ok
test_extract_bag_archive_tgz (test.test_api.TestAPI) ... ok
test_extract_bag_archive_zip (test.test_api.TestAPI) ... ok
test_resolve_fetch_ark (test.test_api.TestAPI) ... FAIL
test_resolve_fetch_http (test.test_api.TestAPI) ... FAIL
test_revert_bag (test.test_api.TestAPI) ... ok
test_update_bag_add_file (test.test_api.TestAPI) ... ok
test_update_bag_change_file (test.test_api.TestAPI) ... ok
test_update_bag_change_metadata (test.test_api.TestAPI) ... ok
test_update_bag_change_metadata_only (test.test_api.TestAPI) ... ok
test_update_bag_prune (test.test_api.TestAPI) ... ok
test_update_bag_remote (test.test_api.TestAPI) ... ok
test_update_bag_remove_file (test.test_api.TestAPI) ... ok
test_validate_complete_bag_fast (test.test_api.TestAPI) ... ok
test_validate_complete_bag_full (test.test_api.TestAPI) ... FAIL
test_validate_incomplete_bag_fast (test.test_api.TestAPI) ... ok
test_validate_incomplete_bag_full (test.test_api.TestAPI) ... ok
test_validate_profile (test.test_api.TestAPI) ... ok
test_validate_profile_serialization (test.test_api.TestAPI) ... ok
test_archive (test.test_cli.TestCli) ... ok
test_create (test.test_cli.TestCli) ... ok
test_extract (test.test_cli.TestCli) ... ok
test_resolve_fetch (test.test_cli.TestCli) ... ok
test_update (test.test_cli.TestCli) ... ok
test_validate (test.test_cli.TestCli) ... FAIL
test_validate_profile (test.test_cli.TestCli) ... ok
test_checksum_without_update (test.test_cli.TestCliArgParsing) ... ok
test_create_bag_already_exists (test.test_cli.TestCliArgParsing) ... ok
test_create_bag_archive_from_existing_archive (test.test_cli.TestCliArgParsing) ... ok
test_create_bag_bad_path (test.test_cli.TestCliArgParsing) ... ok
test_metadata_file_without_update (test.test_cli.TestCliArgParsing) ... ok
test_prune_manifests_without_update (test.test_cli.TestCliArgParsing) ... ok
test_remote_file_manifest_without_update (test.test_cli.TestCliArgParsing) ... ok
test_remote_manifest_with_resolve_fetch (test.test_cli.TestCliArgParsing) ... ok
test_set_checksum_on_existing_archive (test.test_cli.TestCliArgParsing) ... ok
test_skip_manifests_without_update (test.test_cli.TestCliArgParsing) ... ok
test_update_existing_archive (test.test_cli.TestCliArgParsing) ... ok
test_update_with_resolve_fetch (test.test_cli.TestCliArgParsing) ... ok

======================================================================
FAIL: test_resolve_fetch_ark (test.test_api.TestAPI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\users\stain\src\bdbag\test\test_api.py", line 309, in test_resolve_fetch_ark
    self.fail(bdbag.get_named_exception(e))
AssertionError: [BagValidationError] [ChecksumMismatch] fetch.txt checksum validation failed (alg=sha256 expected=7ae3fed7e3ab3ff4b61dc28caef60b0bbfd33d8d02e3a3872f85763db001e21f found=95fe857ab41c6a0b98bdc3d1885db88e5cb6ef1809f5aff1a0a4086514dd6663)
Error: [ChecksumMismatch] fetch.txt checksum validation failed (alg=md5 expected=ef7652679d206141f0e92b3fc6535b01 found=b77763223c065da6927cdfdfc6111812)
Error: [ChecksumMismatch] bagit.txt checksum validation failed (alg=sha256 expected=e91f941be5973ff71f1dccbdd1a32d598881893a7f21be516aca743da38b1689 found=4ec18508e945726d1bdbdbdfef3e8973ede85fef1f4839bc166883dc958fbe93)
Error: [ChecksumMismatch] bagit.txt checksum validation failed (alg=md5 expected=9e5ad981e0d29adc278f6a294b8c2aca found=defc71b28593bb73c7c94a8332f85da8)
Error: [ChecksumMismatch] manifest-sha256.txt checksum validation failed (alg=sha256 expected=d875ab56b58fa26efabfac9b88df76c744e9345ce3db5917e73d6848f62a4bda found=0eca8bd7f6ba6307bf2d3d345005671c8ca143497d18dac46df727c8b4210550)
Error: [ChecksumMismatch] manifest-sha256.txt checksum validation failed (alg=md5 expected=333cab21170b2a11e0312668c959e836 found=4d236dc55ade78d6d4cabd095e546470)
Error: [ChecksumMismatch] bag-info.txt checksum validation failed (alg=sha256 expected=9bfdaf2684806d20f77dd53e06d84a61793749e824917460ec8d0f68dd143825 found=80b473f5afc7ae3fb629b61fe759d514125c33db963803a20bc76fb4b42001ed)
Error: [ChecksumMismatch] bag-info.txt checksum validation failed (alg=md5 expected=999aff45e9bb52eb58117faa3615fdaf found=674367d5c1b92d7274575941e989f9ea)
Error: [ChecksumMismatch] manifest-md5.txt checksum validation failed (alg=sha256 expected=3bc03d737c6e0c7432ca7a5ac06c3f411292ff8ddd884c52f6b1df1abba50763 found=16f38a4b542c0202a86940c8f5e61a3cbed0e0bd9343cc0478a49c395df256f5)
Error: [ChecksumMismatch] manifest-md5.txt checksum validation failed (alg=md5 expected=d0c7f2742808e0f0007f197339110959 found=9e7a6a0af32684e614348513557c2ddb)

======================================================================
FAIL: test_resolve_fetch_http (test.test_api.TestAPI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\users\stain\src\bdbag\test\test_api.py", line 296, in test_resolve_fetch_http
    self.fail(bdbag.get_named_exception(e))
AssertionError: [BagValidationError] [ChecksumMismatch] fetch.txt checksum validation failed (alg=sha256 expected=57b6d06ac0fe19897b830bb35bd99212174add6a5e744079678328c884cd3e6e found=daa654d998274bc5f00e04c8e57058cb56cb377e0ad534031d873a3285473c80)
Error: [ChecksumMismatch] fetch.txt checksum validation failed (alg=md5 expected=7c184a183f96fb2c7f7d3dae1cb1572b found=e3a826326018fb91255bd07a6edbebf7)
Error: [ChecksumMismatch] manifest-sha256.txt checksum validation failed (alg=sha256 expected=f3b404ad84a6fa1dac8347fbb4fca948506e838ff50f54910df4a34107bff786 found=baa9be0ce515e3c6c37374d0c4bdc041ba958f6ce8b48e7f1d62332f2cc69617)
Error: [ChecksumMismatch] manifest-sha256.txt checksum validation failed (alg=md5 expected=512b703d68b25f51d00a43e58b7ee7cf found=d881efe76f1196f4c93281d30cf43cc9)
Error: [ChecksumMismatch] bagit.txt checksum validation failed (alg=sha256 expected=e91f941be5973ff71f1dccbdd1a32d598881893a7f21be516aca743da38b1689 found=4ec18508e945726d1bdbdbdfef3e8973ede85fef1f4839bc166883dc958fbe93)
Error: [ChecksumMismatch] bagit.txt checksum validation failed (alg=md5 expected=9e5ad981e0d29adc278f6a294b8c2aca found=defc71b28593bb73c7c94a8332f85da8)
Error: [ChecksumMismatch] bag-info.txt checksum validation failed (alg=sha256 expected=7d9112f67d8a7211e9791c328c0ac11fdd8ae01917f30a87b6601c50fef089ea found=97a19e048ce27415a75f84c9dff2279e35ecfdd29ac4aeba051b7fc67f806e48)
Error: [ChecksumMismatch] bag-info.txt checksum validation failed (alg=md5 expected=784dbd330810579366a5f2da7349fbcb found=055f3f8886765f3b4e700f4a25d93411)
Error: [ChecksumMismatch] manifest-md5.txt checksum validation failed (alg=sha256 expected=c505e17748f4c20d778c4bc12751ad5568a9316ff3fddbf0b6258286b9c50e5c found=d9ac1e83d4fb1500fa184581349fe4249068e8cc3c1253bdf031e5a00c6ecf2b)
Error: [ChecksumMismatch] manifest-md5.txt checksum validation failed (alg=md5 expected=696c1a8accc36fa88e29369623937b8c found=8de2883c4a6520f33e44ffea0ebbefba)

======================================================================
FAIL: test_validate_complete_bag_full (test.test_api.TestAPI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\users\stain\src\bdbag\test\test_api.py", line 245, in test_validate_complete_bag_full
    self.fail(bdbag.get_named_exception(e))
AssertionError: [BagValidationError] [ChecksumMismatch] manifest-sha1.txt checksum validation failed (alg=sha1 expected=c392c06018b4dcb5a20db90cb0e33164bd3e2bd6 found=eefc792db843b2905055bd576fd0b195d8843e0c)
Error: [ChecksumMismatch] manifest-sha1.txt checksum validation failed (alg=sha256 expected=70615718807d67bb652bfd1a3bdc35a7f99f8eda7458b073f5808db3f5c3b365 found=36ddc0470ad84298ac80116adcf7cf3bd387831824eafb947c6f0651529a8a02)
Error: [ChecksumMismatch] manifest-sha1.txt checksum validation failed (alg=sha512 expected=7444259042079d10f4ea436d598a7f3485d24bf7e790ff20c775213f22558cb08d92cae1bc7aa15dd83e8bb00873fae2963d8e6204adf172a8b1c147204f89b5 found=71cda655a23310684b39d0128b0a8aa095535daaadffa7e58d0943633cda81e0a497397fb02827f323e04beffccfaeb23bc6f639bb46d38e5f1f3146ded87277)
Error: [ChecksumMismatch] manifest-sha1.txt checksum validation failed (alg=md5 expected=beb3e836fcab267224d2fc8d87e667db found=d27f874d3e56a00eae70160540dc9436)
Error: [ChecksumMismatch] manifest-sha256.txt checksum validation failed (alg=sha1 expected=a7ef096226ef78e49be795835fe4d817f653cc00 found=2a93adaf969ab04464447241587d68aa12812a6a)
Error: [ChecksumMismatch] manifest-sha256.txt checksum validation failed (alg=sha256 expected=9603969a98d63e51b77e29b14d730a0041734eb953d3e626712f481103944b94 found=a4e21b29989a5e9da230120abe1fdd05d1568d8451dd01f67614e64a5c7b1a47)
Error: [ChecksumMismatch] manifest-sha256.txt checksum validation failed (alg=sha512 expected=ad20e473a5f535021ca0bcaf7a19aa5ee045a162681cdf8b04ca5945cca6d4257204d694a78c00347c79f851e978fdad45d0dadb77fe881a11bb7b45b1f600e5 found=08b9dacad30e49dd0eebe4615e837cb0ce94668cf6418086a8b3bb686ceeb0a3eccf2d187c4dc2ae461f908520542c9de7b64852c0224a6176dae77b2d30a6be)
Error: [ChecksumMismatch] manifest-sha256.txt checksum validation failed (alg=md5 expected=8f98a0b3cc9a383c5a2fb77b034300e9 found=10a01c3caa44496e97feb4cf7bbe007b)
Error: [ChecksumMismatch] manifest-sha512.txt checksum validation failed (alg=sha1 expected=5690cc83030704f99c6fc118e57b3c8e7de67938 found=ae2cf8d65a8182736a564ba365dbc79f104b5dd1)
Error: [ChecksumMismatch] manifest-sha512.txt checksum validation failed (alg=sha256 expected=4f70c03f907a390e1b2c70e264c7997f49f9da3f550c664eb37b2afeac729b43 found=4c61f506a75a9b076bd33e5e222d4e26e5d5c25c969fde7554547eb6cdefb77e)
Error: [ChecksumMismatch] manifest-sha512.txt checksum validation failed (alg=sha512 expected=3655d3335949ceaec3b64b8dcdf3874951e3d7efc34db23b79a7e85273e887e600264e0a1b849b277c8aec3bf93c29d7f7c53eb743384de05ab7e92a6ae125cf found=74de55fde3c316e113d0362046d48845208793ae5bb5485f457889befc5de3d809554435e58a10ecb444274452e353d27bb3d9ac830132aaa9942f1467bb6a0b)
Error: [ChecksumMismatch] manifest-sha512.txt checksum validation failed (alg=md5 expected=ef1ab1a06c8466b4a0bb1af49c0c716d found=6d2c02c74edab7f3aa7249273b0155f5)
Error: [ChecksumMismatch] bagit.txt checksum validation failed (alg=sha1 expected=e2924b081506bac23f5fffe650ad1848a1c8ac1d found=a7679d9bd2d9211ba88af47e52390187076cc6e4)
Error: [ChecksumMismatch] bagit.txt checksum validation failed (alg=sha256 expected=e91f941be5973ff71f1dccbdd1a32d598881893a7f21be516aca743da38b1689 found=4ec18508e945726d1bdbdbdfef3e8973ede85fef1f4839bc166883dc958fbe93)
Error: [ChecksumMismatch] bagit.txt checksum validation failed (alg=sha512 expected=418dcfbe17d5f4b454b18630be795462cf7da4ceb6313afa49451aa2568e41f7ca3d34cf0280c7d056dc5681a70c37586aa1755620520b9198eede905ba2d0f6 found=2eacbf477915683268fa941f92a29d903627486193b4c27496f8dd12bcc856593a435ab7ddb5b9ba8143e9fac71abc32069df9104247189cee14333ed39cb13b)
Error: [ChecksumMismatch] bagit.txt checksum validation failed (alg=md5 expected=9e5ad981e0d29adc278f6a294b8c2aca found=defc71b28593bb73c7c94a8332f85da8)
Error: [ChecksumMismatch] bag-info.txt checksum validation failed (alg=sha1 expected=daf2d826ddf0e416fdf54550b25a3cab40406c07 found=1135868f3ec8ffea4f0e27eccadb912dd9bca378)
Error: [ChecksumMismatch] bag-info.txt checksum validation failed (alg=sha256 expected=3dd5ab365661c7fd3f0f92ec0a1ac66e5b764136373c322e7254ec7e0adbe43d found=07fb60b620007bc6b962c63da8584c92cd7845976dd737011809715140ccc735)
Error: [ChecksumMismatch] bag-info.txt checksum validation failed (alg=sha512 expected=02532ad2019ddaad798e66431d2817c75891cebe27a44a001985503612e642bb28deceaee48fb46302e8af8ab0486f99b11a970396d35737c73cc885e609d978 found=257c9cac9b79d2442b61e4130bf7d5b5f44497e116349d704cee0e607f191ef71523562d2eedea309e547a20c2eca33c0823746d74c5893c243acb59563b4e36)
Error: [ChecksumMismatch] bag-info.txt checksum validation failed (alg=md5 expected=8cd3f7dd4bbf53cdb57aa23f1657667c found=038c434c36e41b435b2dab600f31dade)
Error: [ChecksumMismatch] manifest-md5.txt checksum validation failed (alg=sha1 expected=00f00590ea8c78c27fd5866f337fd900eeeb9d17 found=3a20bb8d35339fbda6db28eca9ba2d3baa6d7db0)
Error: [ChecksumMismatch] manifest-md5.txt checksum validation failed (alg=sha256 expected=3bc03d737c6e0c7432ca7a5ac06c3f411292ff8ddd884c52f6b1df1abba50763 found=16f38a4b542c0202a86940c8f5e61a3cbed0e0bd9343cc0478a49c395df256f5)
Error: [ChecksumMismatch] manifest-md5.txt checksum validation failed (alg=sha512 expected=0eecbffb32b44584197afe7e983ebb8c3c321167e916a64f23c3ec8170c4940395da9820bb2074defadd77d9aa59c6aa890c1d652888c8ca12e46f403c540c98 found=ed8aa23aa81ede8ef2ed8f6a9fb7bb66a5a3d0ec14252383b610166a6c774e5c96c5e86e4084663f4c5f343e26c279f516061729e3008064664c73a57417b74d)
Error: [ChecksumMismatch] manifest-md5.txt checksum validation failed (alg=md5 expected=d0c7f2742808e0f0007f197339110959 found=9e7a6a0af32684e614348513557c2ddb)

======================================================================
FAIL: test_validate (test.test_cli.TestCli)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\users\stain\src\bdbag\test\test_cli.py", line 61, in test_validate
    self._test_successful_invocation(args, ["test-bag is valid"])
  File "c:\users\stain\src\bdbag\test\test_cli.py", line 31, in _test_successful_invocation
    self.assertExpectedMessages(expected, output)
  File "c:\users\stain\src\bdbag\test\test_common.py", line 37, in assertExpectedMessages
    self.assertIn(expected, output, "Expected \'%s\' in output string." % expected)
AssertionError: Expected 'test-bag is valid' in output string.

----------------------------------------------------------------------
Ran 43 tests in 62.007s

FAILED (failures=4)
Test failed: <unittest.runner.TextTestResult run=43 errors=0 failures=4>
error: Test failed: <unittest.runner.TextTestResult run=43 errors=0 failures=4>

It seems errors are all in tag files - are the .txt files written with CR+LF on Windows but the tests experting checksums based on LF?

mikedarcy commented 6 years ago

This is most likely due to your git client converting the existing LF terminators in the test bag tag files to CRLF on checkout. I just validated the tests do complete on my Win10 system provided the line terminators don't get modified.

See this thread for some more insight: https://stackoverflow.com/questions/2517190/how-do-i-force-git-to-use-lf-instead-of-crlf-under-windows

I'm going to add a .gitattributes file to the repo with the setting * text=auto eol=lf, hopefully that works for most people doing checkouts on Windows without having to modify their local git config.

mikedarcy commented 6 years ago

I just tested this with the latest code and a fresh virtualenv on Win10 and could not reproduce, so it looks like the .gitattributes file with the * text=auto eol=lf setting is doing the right thing. @stain if you have a spare bit of time and could confirm, that would great.

stain commented 6 years ago

Thanks, @mikedarcy - I can confirm that in a fresh checkout python setup.py test works well in my Windows 10, also for the 1.2.4 release.

However in the old git checkout the same tests fails, even after git reset --hard -- so a good trick for anyone affected is to do a clean checkout.