Add Config.download.disable_checksum boolean option (default=False) to disable the checksum computation during download.
Add Digest class for hash computing:
used in Filesystem methods check() and finalize() to validate checksum
used in DownloadCtx and is updated for every received chunk of data
Add FileCheck enum to specify the status of a file on the filesystem:
Missing: file is not found at any of the expected paths (tmp or data)
Part: file is in tmp/<sha>.part, its download is not complete
BadSize: file is either in tmp/<sha>.done, or in data/<DRS>, its size does not match expectations from the metadata
BadChecksum: file is either in tmp/<sha>.done, or in data/<DRS>, its checksum does not match expectations from the metadata
Done: file is in tmp/<sha>.done and has valid size and checksum
Ok: file is in data/<DRS> and has valid size and checksum
Other changes
Improved output during download: - add file.sha and data_node to progress output - print & log each successful download
Fix an issue with file staying started but not downloaded: - Processor.should_download() gives False but the file was added to db already with starting status
Moving a completed file from tmp to its DRS path is now done inside Filesystem.finalize instead of inside FileObject.__aexit__
Add test cases for Filesystem.check()
Mark test that downloads a file from IPSL data node as xfail
Add
Config.download.disable_checksumboolean option (default=False) to disable the checksum computation during download.Add
Digestclass for hash computing:Filesystemmethodscheck()andfinalize()to validate checksumDownloadCtxand is updated for every received chunk of dataAdd
FileCheckenum to specify the status of a file on the filesystem:tmpordata)tmp/<sha>.part, its download is not completetmp/<sha>.done, or indata/<DRS>, its size does not match expectations from the metadatatmp/<sha>.done, or indata/<DRS>, its checksum does not match expectations from the metadatatmp/<sha>.doneand has valid size and checksumdata/<DRS>and has valid size and checksumOther changes
startedbut not downloaded: -Processor.should_download()givesFalsebut the file was added to db already withstartingstatustmpto its DRS path is now done insideFilesystem.finalizeinstead of insideFileObject.__aexit__Filesystem.check()xfail