NOAA-PSL / observation-archive

Tools to assemble observational archive record
Apache License 2.0
0 stars 0 forks source link

data mover app: (3 optional) add md5 sum text file output/computation to the copy process #4

Closed frolovsa closed 2 years ago

frolovsa commented 2 years ago

compute md5 sum for the file when it is copied. output md5 to a text file at the destination bucket.

This can be done in a separate path on already copied files. can be also combined with the some basic plotting of the data.

frolovsa commented 2 years ago

not needed. md5 sums are available from boto3 interface as bucket.Object(key).e_tag[1:-1] or

md5sum = boto3.client('s3').head_object(
            Bucket=bucket_name,
            Key=resource_name
        )['ETag'][1:-1]