ross-spencer
commented
5 years ago Thanks @mjaddis.
Seeing this I was concerned about what was happening, and whether there was a possibility that spurious results could be generated, or not all errors may be reported when using the md5deep hash-check, e.g. could files/hashes be missed if not listed correctly?
I modified the script archivematicaCheckMD5NoGUI.sh to generate a lot of additional output, here:
#!/bin/bash
#
# This file is part of Archivematica.
#
# Copyright 2010-2013 Artefactual Systems Inc. <http://artefactual.com>
#
# Archivematica is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Archivematica is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Archivematica. If not, see <http://www.gnu.org/licenses/>.
# @package Archivematica
# @subpackage archivematicaClientScript
# @author Joseph Perry <joseph@artefactual.com>
# @version svn: $Id: 09716783f2866a77f1240067a610b19103f69c1b $
#create temp report files
UUID=`uuid`
failTmp=/tmp/fail-$UUID
passTmp=/tmp/pass-$UUID
reportTmp=/tmp/report-$UUID
checkFolder="$1"
md5Digest="$2"
integrityReport="$3"
checksumTool="$4"
tmpDir=`pwd`
ret=0
cd "$checkFolder"
echo "^^^"
echo "-- Current directory listing:"
echo
echo "$(ls)"
echo
echo "^^^"
echo "-- Provided MD5 digest:"
echo
echo "$(< $md5Digest)"
echo
echo "^^^^^"
#check for passing checksums
"${checksumTool}" -w -l -r -M "$md5Digest" . > $passTmp
rx="$?"
echo
echo "-- relative exit code for -m flag"
echo $rx
echo
echo "-- passTmp output"
echo
echo "$(< $passTmp)"
echo
ret+=$rx
#check for failing checksums
"${checksumTool}" -w -l -r -X "$md5Digest" . > $failTmp
ry="$?"
echo
echo "-- relative exit code for -x flag"
echo $ry
echo
echo "-- failTmp output"
echo
echo "$(< $failTmp)"
echo
ret+=$ry
# ------------
#check for passing checksums
"${checksumTool}" -w -r -M "$md5Digest" . > $passTmp
rx="$?"
echo
echo "-- non-relative exit code for -m flag"
echo $rx
echo
echo "-- passTmp output"
echo
echo "$(< $passTmp)"
echo
ret+=$rx
echo "---"
#check for failing checksums
"${checksumTool}" -w -r -X "$md5Digest" . > $failTmp
ry="$?"
echo
echo "-- non-relative exit code for -x flag"
echo $ry
echo
echo "-- failTmp output"
echo
echo "$(< $failTmp)"
echo
ret+=$ry
echo "---"
# Change directory to temporary directory to read results...
cd $tmpDir
#Count number of Passed/Failed
numberPass=`wc -l $passTmp| cut -d" " -f1`
numberFail=`wc -l $failTmp| cut -d" " -f1`
#Create report
echo "PASSED" >> $reportTmp
cat $passTmp >> $reportTmp
echo " " >> $reportTmp
echo $numberPass "items passed integrity checking" >> $reportTmp
echo " " >> $reportTmp
echo " " >> $reportTmp
echo "FAILED" >> $reportTmp
cat $failTmp >> $reportTmp
echo " " >> $reportTmp
echo $numberFail "items failed integrity checking" >> $reportTmp
#copy pasta
cp $reportTmp "$integrityReport"
cat $failTmp 1>&2
#cleanup
rm $failTmp $passTmp $reportTmp
ret+="$numberFail"
exit ${ret}It would seem that part of the problem lies between the use of relative and absolute paths. Looking at the man pages, it's not entirely clear why, but it doesn't seem that hashdeep can do much very clever using either method.
In the above script, I do manage to engineer a pass using relative paths (output (and comparison) using -l and amending ./ to each line. I match using:
"${checksumTool}" -w -l -r -M "$md5Digest" . > $passTmp
And I also see something more like the failure that you're seeing using:
"${checksumTool}" -w -r -M "$md5Digest" . > $passTmp
You can see the results in a simple transfer here:
^^^
-- Current directory listing:
file-one.a
file-two.b
^^^
-- Provided MD5 digest:
ebe44f549cb0406915a9239986f2c72f ./file-one.a
ebe44f549cb0406915a9239986f2c72f ./file-two.b
^^^^^
-- relative exit code for -m flag
0
-- passTmp output
ebe44f549cb0406915a9239986f2c72f ./file-one.a matched ./file-one.a
ebe44f549cb0406915a9239986f2c72f ./file-two.b matched ./file-two.b
-- relative exit code for -x flag
0
-- failTmp output
-- non-relative exit code for -m flag
1
-- passTmp output
ebe44f549cb0406915a9239986f2c72f /var/archivematica/sharedDirectory/currentlyProcessing/md5-example-b7900671-fbae-4287-a2db-4ceed3afe57b/objects/file-two.b matched ./file-one.a
ebe44f549cb0406915a9239986f2c72f /var/archivematica/sharedDirectory/currentlyProcessing/md5-example-b7900671-fbae-4287-a2db-4ceed3afe57b/objects/file-one.a matched ./file-one.a
---
-- non-relative exit code for -x flag
1
-- failTmp output
---
File Does not exist: /var/archivematica/sharedDirectory/currentlyProcessing/md5-example-b7900671-fbae-4287-a2db-4ceed3afe57b/metadata/checksum.sha1
File Does not exist: /var/archivematica/sharedDirectory/currentlyProcessing/md5-example-b7900671-fbae-4287-a2db-4ceed3afe57b/metadata/checksum.sha256And a more complex transfer here:
^^^
-- Current directory listing:
beihai.tif
bird.mp3
ocr-image.png
piiTestDataCreditCardNumbers.txt
piiTestDataSocialSecurityNumbers.txt
View_from_lookout_over_Queenstown_towards_the_Remarkables_in_spring.jpg
^^^
-- Provided MD5 digest:
2121dca88ad7f701d3f3e2d041004a56 ./beihai.tif
2121dca88ad7f701d3f3e2d041004a56 ./bird.mp3
6dc1519418859ea5c20fd708e89d7254 ./ocr-image.png
75388a532283b988f79206d63f65e9a2 ./piiTestDataCreditCardNumbers.txt
1d7193ea3b2193c79f55ea7e645503a9 ./piiTestDataSocialSecurityNumbers.txt
4737e4dacfc9510915ea58cf12e51712 ./View_from_lookout_over_Queenstown_towards_the_Remarkables_in_spring.jpg
^^^^^
-- relative exit code for -m flag
0
-- passTmp output
1d7193ea3b2193c79f55ea7e645503a9 ./piiTestDataSocialSecurityNumbers.txt matched ./piiTestDataSocialSecurityNumbers.txt
75388a532283b988f79206d63f65e9a2 ./piiTestDataCreditCardNumbers.txt matched ./piiTestDataCreditCardNumbers.txt
6dc1519418859ea5c20fd708e89d7254 ./ocr-image.png matched ./ocr-image.png
4737e4dacfc9510915ea58cf12e51712 ./View_from_lookout_over_Queenstown_towards_the_Remarkables_in_spring.jpg matched ./View_from_lookout_over_Queenstown_towards_the_Remarkables_in_spring.jpg
2121dca88ad7f701d3f3e2d041004a56 ./bird.mp3 matched ./bird.mp3
2121dca88ad7f701d3f3e2d041004a56 ./beihai.tif matched ./beihai.tif
-- relative exit code for -x flag
0
-- failTmp output
-- non-relative exit code for -m flag
1
-- passTmp output
75388a532283b988f79206d63f65e9a2 /var/archivematica/sharedDirectory/currentlyProcessing/dupe-example-5f8a8094-9134-4333-a625-519bacf594d1/objects/piiTestDataCreditCardNumbers.txt matched ./piiTestDataCreditCardNumbers.txt
1d7193ea3b2193c79f55ea7e645503a9 /var/archivematica/sharedDirectory/currentlyProcessing/dupe-example-5f8a8094-9134-4333-a625-519bacf594d1/objects/piiTestDataSocialSecurityNumbers.txt matched ./piiTestDataSocialSecurityNumbers.txt
6dc1519418859ea5c20fd708e89d7254 /var/archivematica/sharedDirectory/currentlyProcessing/dupe-example-5f8a8094-9134-4333-a625-519bacf594d1/objects/ocr-image.png matched ./ocr-image.png
4737e4dacfc9510915ea58cf12e51712 /var/archivematica/sharedDirectory/currentlyProcessing/dupe-example-5f8a8094-9134-4333-a625-519bacf594d1/objects/View_from_lookout_over_Queenstown_towards_the_Remarkables_in_spring.jpg matched ./View_from_lookout_over_Queenstown_towards_the_Remarkables_in_spring.jpg
2121dca88ad7f701d3f3e2d041004a56 /var/archivematica/sharedDirectory/currentlyProcessing/dupe-example-5f8a8094-9134-4333-a625-519bacf594d1/objects/beihai.tif matched ./beihai.tif
2121dca88ad7f701d3f3e2d041004a56 /var/archivematica/sharedDirectory/currentlyProcessing/dupe-example-5f8a8094-9134-4333-a625-519bacf594d1/objects/bird.mp3 matched ./beihai.tif
---
-- non-relative exit code for -x flag
1
-- failTmp output
---
File Does not exist: /var/archivematica/sharedDirectory/currentlyProcessing/dupe-example-5f8a8094-9134-4333-a625-519bacf594d1/metadata/checksum.sha1
File Does not exist: /var/archivematica/sharedDirectory/currentlyProcessing/dupe-example-5f8a8094-9134-4333-a625-519bacf594d1/metadata/checksum.sha256The -w flag is nice as it shows what matched with what. E.g.
ebe44f549cb0406915a9239986f2c72f ./file-one.a matched ./file-one.a
ebe44f549cb0406915a9239986f2c72f ./file-two.b matched ./file-two.bThere doesn't seem to be a sensible workaround for users here. It looks like you are already using -l to create a manifest. The issue itself is confirmed with the three algorithms (md5, sha1, sha256). I can't see other issues with the mechanism.
NB. With the two calls to "${checksumTool} ..." I wonder if this is redundancy we don't really need in a script performing this function?
andrewjbtw
sallain
mjaddis
Expected behaviour If I have a transfer with multiple files that happen to have the same checksum and these are listed correctly in checksum.md5 in the metadata directory, then I expect checksum validation to pass at the Transfer stage of the workflow.
Current behaviour Checkum validation fails with error code 110. However, logs from the script show that the md5deep tests actually passed, for example:
Steps to reproduce create a transfer containing two files that have different filenames but the same checksum. Use md5deep to generate a manifest and put this in checksum.md5. For example:
It looks like md5deep is exiting with a 1 when there is more than one entry in the manifest that matches a given file in the objects directory. If there aren't duplicates then it exits 0. However, ignoring the exit code, actual the test still passes. The Archivematica script concatenates the return codes of
md5deep -r -m,md5deep -r -xand the count of failed files. Hence the 110 code.Better perhaps to use
md5sum -cinstead ofmd5deepas this would involve the correct entry in the manifest being used agains the correct payload file in the objects dir?Run this as a standard Transfer.
Your environment (version of Archivematica, OS version, etc) AM1.7.2 on 16.04 LTS
For Artefactual use: Please make sure these steps are taken before moving this issue from Review to Verified in Waffle: