Bad Gateway causes failure to retrieve _all_ statuses

[finnoleary]

A bad gateway error during the status retrieval process causes the entire process to hault and all statuses retrieved to be lost.

Traceback (most recent call last):
  File "/usr/bin/mastodon-archive", line 11, in <module>
    load_entry_point('mastodon-archive==1.3.0', 'console_scripts', 'mastodon-archive')()
  File "/usr/lib/python3.7/site-packages/mastodon_archive/__init__.py", line 293, in main
    args.command(args)
  File "/usr/lib/python3.7/site-packages/mastodon_archive/archive.py", line 131, in archive
    first_page = statuses)
  File "/usr/lib/python3.7/site-packages/mastodon/Mastodon.py", line 2977, in fetch_remaining
    current_page = self.fetch_next(current_page)
  File "/usr/lib/python3.7/site-packages/mastodon/Mastodon.py", line 2936, in fetch_next
    return self.__api_request(method, endpoint, params)
  File "/usr/lib/python3.7/site-packages/mastodon/Mastodon.py", line 3307, in __api_request
    error_msg)
mastodon.Mastodon.MastodonBadGatewayError: ('Mastodon API returned error', 502, 'Bad Gateway', None)

[kensanata]

Not sure what you expected? No error? Or saving the partial result?

[finnoleary]

From a user perspective:

The error could either be more detailed -- for example, what site gives a bad gateway, what post was it accessing, etc. -- give me information to debug it / contact the owner.

The alternative of saving the partial result or skipping that specific message / favourite is also important as otherwise you're sacrificing the entire history (which in my case is 1 or 2 years worth) for one bad / offline site.

[kensanata]

Better error messages, yeah, I can get behind that. There’s one thing confusing me in your comment, though: the tool only ever accessed your own instance: thus the cause for the bad gateway error doesn’t really need to be specified: it’s your instance. Or am I misunderstanding something?

[finnoleary]

That's the thing. The instance I use was up and responding to requests at the time of the failure. I've contacted the instance admin to see if there's something I or they need to do to solve it.

Previously I figured the 'statuses' grabbed both interaction statuses as well as your own statuses, so it was a fair assumption to allow skipping of foreign statuses that we can't grab (for whatever reason -- but most likely because the instance is dead (see: the death of witches.town) or otherwise non-responsive (down for maintenance)).

[kensanata]

I must still be missing something. As far as I understand federation, the statuses that originated on foreign servers are still stored on your server. Only media (images) can disappear. Even if the originating instance goes down, their statuses that got delivered to your inbox remain available on your instance. It is very much like email in this regard.

[finnoleary]

Thanks for the explanation. I've started talking to the server admin, we dug into it and it was a database timeout.

[kensanata]

Phew! 😅