General API Problem on use

[ghost]

i followed the instructions up to running the first backup run, and ran into an error after a short while:

./mastodon-backup.py amphetamine@social.wxcafe.net
Get user info
Get statuses (this may take a while)
Traceback (most recent call last):
  File "./mastodon-backup.py", line 79, in <module>
    first_page = statuses)
  File "/usr/local/lib/python3.5/dist-packages/mastodon/Mastodon.py", line 907, in fetch_remaining
    current_page = self.fetch_next(current_page)
  File "/usr/local/lib/python3.5/dist-packages/mastodon/Mastodon.py", line 866, in fetch_next
    return self.__api_request(method, endpoint, params)
  File "/usr/local/lib/python3.5/dist-packages/mastodon/Mastodon.py", line 1124, in __api_request
    raise MastodonAPIError('General API problem.')
mastodon.Mastodon.MastodonAPIError: General API problem.

[kensanata]

How strange. I have no idea. This happens every time you try it? Do you have other accounts and does it work there? I’m trying to figure out whether it is something about your data or about the server that is causing the problem.

[kensanata]

Line 1124 in Mastodon.py is a simple sleep instruction. This remains a mystery. The only place where a General API problem is raised is when a request returns with a code 500 ("Internal Server Error"). That's why I suspect it's a problem on the server side.

[kensanata]

If the problem doesn't occur anymore, I'll just close this issue. What do you think?

[wion]

I just tried running your updated 'archive' package instructions on mammouth.cafe.

The package installed fine.

I opted to skip favourites. Btw, I think you have a mistake in your command there. It should be...

mastodon-archive archive --no-favourites username@domain.tld

It was still a little slow, but eventually it kicked this out, which looks similar to dexamphetamine's error:

Get statuses (this may take a while)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/bin/mastodon-archive", line 11, in <module>
    load_entry_point('mastodon-archive==0.0.3', 'console_scripts', 'mastodon-archive')()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mastodon_archive/__init__.py", line 65, in main
    args.command(args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mastodon_archive/archive.py", line 110, in archive
    first_page = statuses)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mastodon/Mastodon.py", line 907, in fetch_remaining
    current_page = self.fetch_next(current_page)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mastodon/Mastodon.py", line 866, in fetch_next
    return self.__api_request(method, endpoint, params)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mastodon/Mastodon.py", line 1124, in __api_request
    raise MastodonAPIError('General API problem.')
mastodon.Mastodon.MastodonAPIError: General API problem.

I will try it again on the mastodon.social account to see if there's a difference.

Btw, mammouth.cafe is running Mastodon 1.8.x. At least one version behind.

[wion]

Just tested on mastodon.social (again using the 'no favourites' option) and it backed up quickly and without errors.

Maybe there is something related with the version of Mastodon?

Nope, maybe not. social.wxcafe.net appears to be running 2.0.0 so that theory is blown. And come to think of it, DredMorbius backed up his account on mammouth.cafe successfully.

I guess I'm just destined to lose those toots. No way around it.

[kensanata]

Or there is a non-default rate limit on the server. I’ll see whether I can expose that as a parameter and then people can provide different rate (and make it even slower, I guess).

[kensanata]

Sadly, I never posted too much on mammouth.cafe:

Get user info
Get statuses (this may take a while)
Get favourites (this may take a while)
Saving 64 statuses and 65 favourites

[kensanata]

On line 77 is the setup for the Mastodon API in case you have successfully authorized the app before. This is what it looks like:

        mastodon = Mastodon(
            client_id = client_secret,
            access_token = user_secret,
            api_base_url = url)

Based on the documentation this is the equivalent of:

        mastodon = Mastodon(
            client_id = client_secret,
            access_token = user_secret,
            api_base_url = url,
            ratelimit_method='wait',
            ratelimit_pacefactor=1.1,
            request_timeout=300)

Thus, these are the three parameters we can play with. Let's assume the server has not changed the 300s time window for requests. Perhaps we can simply avoid running into timeouts? The documentation has some suggestions. How about pacing our requests and have them be slightly slower than expected. We should never run into a timeout:

        mastodon = Mastodon(
            client_id = client_secret,
            access_token = user_secret,
            api_base_url = url,
            ratelimit_method='pace',
            ratelimit_pacefactor=0.9,
            request_timeout=300)

Give it a try and let us know?

If you have mastodon-archive 0.0.3 installed, you need to download the latest release, unzip it into a directory, make the changes, and install those changes using pip install -e . (run from the directory where the setup.py file is).

Verify:

$ ls /usr/local/lib/python3.6/site-packages/mastodon-archive*
/usr/local/lib/python3.6/site-packages/mastodon-archive.egg-link
$ cat /usr/local/lib/python3.6/site-packages/mastodon-archive.egg-link
/Users/alex/Source/mastodon-backup

As you can see, the system wide installation is but a link that points to my working directory.

To undo your changes, run pip uninstall mastodon-archive followed by pip install mastodon-archive and you should get back the default.

[wion]

If you have mastodon-archive 0.0.3 installed, you need to download the latest release...

Do you mean if I don't have it installed? Because I have the latest version (0.0.3) installed, but I installed it before your notes above. Should I re-install it and then make the noted changes?

Anyway, I'll try it with the install I have, and then again with new if it doesn't work. ;)

[wion]

install those changes using pip install -e . (run from the directory where the setup.py file is)

Is the command going to be pip or pip3? (I've been using the latter up to this point since installing Python 3.6.)

Also, where is setup.py? I can find all kinds of .py files, but not that one.

At this point I've modified archive.py, but I don't quite understand the install changes part your talking about.

[kensanata]

Hm, perhaps I was writing this from a developer perspective where I always have two copies of the thing: /Users/alex/src/mastodon-backup is where I work and make my changes and /Library/Frameworks/Python.framework/Versions/3.6/bin/mastodon-archive is where it was installed. So if I make changes in in the first directory, and run the app, it will run the unchanged files from the second directory. If you only have the second directory, and you make your changes there, then running the app should run the changed files and you're fine. Sorry for the confusion! The setup.py is only available if you checked out the sources or downloaded a zip file from the GitHub releases, which you didn't since you used pip to install it. As for pip vs. pip3: I think the way it was meant to be was the tool is called pip. But then people started switching to Python 3 and others decided to keep Python 2 around, and they both came with a slightly different pip. What to do? Installers these days will simply rename python to python3 and pip to pip3 if they're installing Python 3 on a system where Python 2 already exists.

[wion]

If you only have the second directory, and you make your changes there, then running the app should run the changed files and you're fine.

Okay, I went ahead and tried that anyway. ;) It's still sitting at:

Get statuses (this may take a while)

And that was at least 20 minutes ago.

Anyway, if it doesn't work this time, I'm not going to worry about it anymore for mammouth.cafe. Of all the content there, I would probably only re-publish a fraction of it. I was more interested in some of the ideas I shared because they were topics to expand on as blog articles. That's the only bummer. But I can live it. Sometimes it's better to just let shit go ;)

[kensanata]

Before giving up altogether, you might want to give the alternatives a try. Mastotool downloads the public stuff that you can see on https://mammouth.cafe/@wion i.e. none of the replies but all of your "top level" posts. Mastotool Atom downloads the toots including your replies but Mastodon has a bug such that the feed ends if it ever gets to a "gap" in your toots: if you ever deleted a toot, that's where your Atom feed ends. I haven't tried MastoUserScrape.py.

[ghost]

Switching to pace and reducing the factor like you said worked perfectly, no need to skip favorites. Thanks!

[kensanata]

Switching to pace and reducing the factor like you said worked perfectly, no need to skip favorites. Thanks!

Cool. I’ll make this an option, then.

[wion]

Happy to report it worked for me too! It took a while (I lost track how long) but I came back to the computer later and 2011 statuses were successfully backed up.

[wion]

Nice work on this tool. And thanks for your patience with the likes of me.

[kensanata]

Excellent. 1a876e5 adds the --pace option, so the next release will have this.