`scripts/s3_media_upload` ignores media files that have been deleted, but then pulled back into the local media store

matrix-org / synapse-s3-storage-provider

Synapse storage provider to fetch and store media in Amazon S3

Apache License 2.0

132 stars 36 forks source link

`scripts/s3_media_upload` ignores media files that have been deleted, but then pulled back into the local media store #63

Open squahtx opened 3 years ago

squahtx commented 3 years ago

When (re)generating thumbnails, Synapse will ensure that the original image is in its local media store by downloading it back from storage providers. The s3_media_upload script does not notice when this happens and won't re-delete such files.

H-Shay commented 2 years ago

For my own edification, what would be the impact of this? Storage space being wasted?

squahtx commented 2 years ago

That's pretty much it. The script is supposed to offload media to S3 to save on storage locally.

H-Shay commented 2 years ago

Thanks for answering!

kpfleming commented 5 months ago

It seems like a workaround for this would be to delete the SQlite3 database that s3_media_upload uses as a database cache; it will no longer know what it has previously deleted, and will have to regenerate that list. It will take longer to run and generate more activity for the Synapse database.