User and room avatars missing after s3 migration

[JPinSPACE]

I switched my media store to use the s3 media provider using the tools and instructions here and that seems to go fairly smoothly. My media seems to have made it to s3 with no major issues despite a surprisingly large number of images being reported to be deleted.

The exception is user and room avatars. These are missing entirely and I can't find them either in my old media store or s3. There also seems to be no way to force the server to re-download those avatars.

There is an extra bit of data which may be the real problem so I wanted to mention it here before anyone spent too long trying to tie this to the s3 migration: on the same day I attempted to use the ansible playbooks to upgrade my postgres database and it failed due to the server disk filling up entirely. I had to restore to the backup of the postgres database.

I am simply assuming that this is an issue with the s3 storage provider since it's related to media, so apologies if it turns out this is obviously related to the failed postgres upgrade.

I am a novice matrix server admin so it is entirely possible (probably likely) that I just screwed something up here, but I'd appreciate any advice.

[reivilibre]

All user and room avatars are missing entirely, but nothing else? That seems rather odd — avatars aren't really treated any differently (although they are usually thumbnailed server-side).

I can't find them either in my old media store or s3

sounds silly, but how do you mean? What did you do in order to try to 'find' them?

---

What I'd do: find the URL of one of these pieces of media. To do so, you might get successful with right clicking it and copying image URL (You should see something like https://example.org/_matrix/media/r0/thumbnail/example.org/abcdef?width=45&height=45&method=crop), or otherwise use /devtools to 'Explore Room State' and look at the 'm.room.avatar(You should see something likemxc://example.org/abcdef`). That gives you an indication of what file to look for in your media store or s3. If it's for your own server, you would find it in the local media — otherwise in the remote media.

I would look in the Synapse logs for example.org/abcdef to see what's happening to the requests that request the media. Are they failing? Do they give you any messages about the problem? See https://matrix-org.github.io/synapse/latest/usage/administration/request_log.html for a description of the request log format; the 'request ID' will be the same for log lines about the same request, so might be useful in finding out what's happening.

[JPinSPACE]

Good call. I checked the logs and actually wasn't able to find the request I was looking for but I found what seems to be the main issue. The logs are jam-packed with this:

Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]: 2022-12-09 16:00:21,034 - synapse.rest.media.v1.storage_provider - 113 - ERROR - POST-45187 - Error storing file
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]: Traceback (most recent call last):
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/synapse/rest/media/v1/storage_provider.py", line 109, in store
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     return await maybe_awaitable(
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/twisted/python/threadpool.py", line 244, in inContext
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     result = inContext.theWork()  # type: ignore[attr-defined]
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     return self.currentContext().callWithContext(ctx, func, *args, **kw)
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     return func(*args, **kw)
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/s3_storage_provider.py", line 121, in _store_file
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     self._get_s3_client().upload_file(
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/s3_storage_provider.py", line 111, in _get_s3_client
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     self._s3_client = s3 = b3_session.client("s3", **self.api_kwargs)
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/boto3/session.py", line 299, in client
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     return self._session.create_client(
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 976, in create_client
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     client = client_creator.create_client(
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 155, in create_client
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     client_args = self._get_client_args(
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 485, in _get_client_args
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     return args_creator.get_client_args(
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/botocore/args.py", line 129, in get_client_args
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     endpoint = endpoint_creator.create_endpoint(
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:   File "/usr/local/lib/python3.9/site-packages/botocore/endpoint.py", line 402, in create_endpoint
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]:     raise ValueError("Invalid endpoint: %s" % endpoint_url)
Dec 09 16:00:21 ip-172-31-66-66 matrix-synapse[16816]: ValueError: Invalid endpoint:

I'm confused by it saying it has a bad endpoint, because when I check the documentation it says to just entirely remove the endpoint config line if using S3: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/configuring-playbook-synapse-s3-storage-provider.md

[JPinSPACE]

Update! This has been resolved. The guide I used erroneously told me to not include an endpoint if I'm using S3. But I got curious and searched for the endpoints for my S3 region, threw it in there, and restarted, and lo and behold everything started working again, including my user avatars. I'll go file an issue with the playbook, thanks.

Here's the guide: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/configuring-playbook-synapse-s3-storage-provider.md