search

matrix-org / synapse-s3-storage-provider

Synapse storage provider to fetch and store media in Amazon S3
Apache License 2.0
118 stars 33 forks source link

`s3_media_upload` should report number of files skipped due to already being on S3 #75

Open richvdh opened 2 years ago

richvdh commented 2 years ago

This makes for alarming reading:

Aug 25 00:51:50 hippogriff.matrix.org synapse_media_upload[7417]: Uploading/deleting media for media_store
Aug 25 02:04:06 hippogriff.matrix.org synapse_media_upload[7417]: Uploading  92323  files
Aug 25 02:04:06 hippogriff.matrix.org synapse_media_upload[7417]: Uploaded 0 media out of 92323
Aug 25 02:04:06 hippogriff.matrix.org synapse_media_upload[7417]: Uploaded 0 files
Aug 25 02:04:06 hippogriff.matrix.org synapse_media_upload[7417]: Uploaded 0B
Aug 25 02:04:06 hippogriff.matrix.org synapse_media_upload[7417]: Deleted 92323 media

you deleted 92323 media without uploading any of them??

Presumably, these files already existed on S3. Generally the logging here could do with clarifying (what's the difference between a media and a file, and why are their counts logged separately?), but in particular it would be good to keep a count of "already existing" media (or files?) so that we can check that the numbers add up.