search

aces / Loris-MRI

The set of scripts that preprocess and insert MRI data into the database.
10 stars 50 forks source link

Add method to check and skip duplicate content uploads to S3 #1015

Closed breen0074 closed 7 months ago

breen0074 commented 1 year ago

Description

The changes here are intended to check to see if the content of file that would be uploaded to S3 has already been uploaded. It does this by checking to see if the hash of a file content is already available at the targeted S3 object key location before attempting to upload new content. If it already exists, it will skip it.

This helps to resolve an issue where sometimes the same content would be uploaded to an S3 bucket, even if that file already existed. Normally this would be fine, but in versioning enabled buckets this creates duplicate copies of the files when no changes are needed.

This does not appear to cause any breaking changes.

github-actions[bot] commented 8 months ago

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 20 days.

cmadjar commented 7 months ago

Closing in favour of https://github.com/aces/Loris-MRI/pull/1032 (which is identical to this PR but rebased on 24.1-release branch instead of main)