Add Config.download.disable_checksum boolean option (default=False) to disable the checksum computation during download.
Add Digest class for hash computing:
used in Filesystem methods check() and finalize() to validate checksum
used in DownloadCtx and is updated for every received chunk of data
Add FileCheck enum to specify the status of a file on the filesystem:
Missing: file is not found at any of the expected paths (tmp or data)
Part: file is in tmp/<sha>.part, its download is not complete
BadSize: file is either in tmp/<sha>.done, or in data/<DRS>, its size does not match expectations from the metadata
BadChecksum: file is either in tmp/<sha>.done, or in data/<DRS>, its checksum does not match expectations from the metadata
Done: file is in tmp/<sha>.done and has valid size and checksum
Ok: file is in data/<DRS> and has valid size and checksum
Other changes
Improved output during download: - add file.sha and data_node to progress output - print & log each successful download
Fix an issue with file staying started but not downloaded: - Processor.should_download() gives False but the file was added to db already with starting status
Moving a completed file from tmp to its DRS path is now done inside Filesystem.finalize instead of inside FileObject.__aexit__
Add test cases for Filesystem.check()
Mark test that downloads a file from IPSL data node as xfail
Add
Config.download.disable_checksum
boolean option (default=False) to disable the checksum computation during download.Add
Digest
class for hash computing:Filesystem
methodscheck()
andfinalize()
to validate checksumDownloadCtx
and is updated for every received chunk of dataAdd
FileCheck
enum to specify the status of a file on the filesystem:tmp
ordata
)tmp/<sha>.part
, its download is not completetmp/<sha>.done
, or indata/<DRS>
, its size does not match expectations from the metadatatmp/<sha>.done
, or indata/<DRS>
, its checksum does not match expectations from the metadatatmp/<sha>.done
and has valid size and checksumdata/<DRS>
and has valid size and checksumOther changes
started
but not downloaded: -Processor.should_download()
givesFalse
but the file was added to db already withstarting
statustmp
to its DRS path is now done insideFilesystem.finalize
instead of insideFileObject.__aexit__
Filesystem.check()
xfail