matrix-org / synapse-s3-storage-provider

Synapse storage provider to fetch and store media in Amazon S3
Apache License 2.0
118 stars 33 forks source link

Add support for thumbnail offload to `scripts/s3_media_upload` #60

Closed squahtx closed 3 years ago

squahtx commented 3 years ago

Reviewable commit by commit Built on top of #59: ignore all but the last 3 4 commits

Update `s3_media_upload` to handle thumbnails

Media are now considered deleted only if both the original file and all
thumbnails have been deleted.

`cache.db`s built before this change may incorrectly count media as deleted
while their thumbnails still exist in the local cache. This can be resolved
by either:
 a) deleting `cache.db` and running an `update` to crawl through the entire
    local cache again. This may take a very long time for large Synapse
    deployments.
 b) uploading the contents of local_thumbnails/ and remote_thumbnail/
    manually, then deleting the uploaded files. Note that a running Synapse
    instance may write new thumbnails during the process.

    If the S3 storage provider has been installed since the very start, the
    thumbnails should already be in S3 and the upload step can be skipped.

This commit changes the behavior of the `write` command. Previously, `write`
would only output undeleted file paths. Now the output contains *possibly*
undeleted paths of files and thumbnail directories.

note to self: commit description for Include thumbnail paths in s3_media_upload's write output is out of date

squahtx commented 3 years ago

Moved to #61