search

Duke-GCB / DukeDSClient

Command line program to allow uploading, downloading, and managing projects in the duke-data-service.
MIT License
5 stars 6 forks source link

Support for multiple checksums #258

Closed johnbradley closed 5 years ago

johnbradley commented 5 years ago

Adds support to check all known hashes attached to a file when downloading. Before this change DukeDSClient only checked the first MD5 hashes found. DukeDSClient will now validate all MD5 hashes. Files that have both valid and invalid hashes will have status CONFLICTED. If there are CONFLICTED files a warning will be printed but no error raised. If there are any FAILED hashes an error will be raised.

Example terminal output after downloading a project with a conflicted file:

...
Verifying contents of 2 downloaded files using file hashes.
mh/me.jpeg 0106a7dcf3ab09bc47cab34a7ff1fbba md5 OK
mh/stuff.txt 7e11867261e621458a06f3e6bc301c8b md5 CONFLICTED
All downloaded files have at least one valid hash.

WARNING: Some downloaded files also have invalid hashes.

Example project in DukeDS dev that has multiple checksums: https://dev.dataservice.duke.edu/#/project/cfa04bfd-c025-4025-8153-b058d0893a12 The file in the above project has an additional wrong checksum added via the PUT /api/v1/uploads/{id}/hashes DukeDS API.

Background

There was a bug in the DukeDS web portal where large files uploaded had an incorrect hash. A project was created to add the correct hash to these files. This leaves the incorrect hash in the database for these files.

Fixes #256

johnbradley commented 5 years ago

Any feedback on the verbiage would be appreciated.

johnbradley commented 5 years ago

Changed CONFLICTED to WARNING. Updated command line when downloading:

$ ddsclient download -p MultiHashes mh
Fetching list of files/folders.
Downloading 2 files.
Done: 100%                          
Verifying contents of 2 downloaded files using file hashes.
mh/me.jpeg 0106a7dcf3ab09bc47cab34a7ff1fbba md5 OK
mh/stuff.txt 7e11867261e621458a06f3e6bc301c8b md5 WARNING
All downloaded files have been verified successfully.

NOTICE: Data Service reports multiple hashes for 1 files. 
The downloaded files have been verified and confirmed to match one of these hashes. 
You do not need to retry the download. 
For more information, visit https://github.com/Duke-GCB/DukeDSClient/wiki/MD5-Hash-Conflicts.