search

matrix-org / synapse-s3-storage-provider

Synapse storage provider to fetch and store media in Amazon S3
Apache License 2.0
118 stars 33 forks source link

Handle cases where `last_access_ts` is null in s3_media_upload #38

Closed PeterCxy closed 3 years ago

PeterCxy commented 4 years ago

Some local media files may not have been accessed at all since uploading. This may sound counterintuitive, but some bridges, for example the mautrix_telegram bridge, can generate such media files when users are not online to check the messages.

Currently, the s3_media_upload script does not handle these files at all. On my instance this has resulted in more than 20 gigabytes of media files that simply haven't been accessed at all (because they are all from the Telegram bridge) and are not removed from the local storage or uploaded to S3.

erikjohnston commented 4 years ago

I think this is more a synapse issue, so transferring this there.

erikjohnston commented 4 years ago

Oh, my bad, I forgot that the script was a thing and you didn't just mean the synapse integration :man_facepalming:

erikjohnston commented 4 years ago

The script should just fall back to the created ts if last_access_ts is null

HelderFSFerreira commented 4 years ago

I don't know if I understand it or just was unable to replicate. According to this issue should be lines on the table remote_media_cache where last_access_ts is null right?

Have the telegram bridge setup on my server had 0 rows on the query.

PeterCxy commented 4 years ago

@HelderFSFerreira not remote media, but local media. Yes, bridged media files are local.