Step: Making a backup

[wion]

Just to make this official. ;)

I'm using macOS. I've installed Python 3.6.3.

I run into problems early on when trying to authorize with the account. Here's the commands history:

$ python3
Python 3.6.3 (v3.6.3:2c5fed86e0, Oct  3 2017, 00:32:08) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D

$ pip3 install Mastodon.py
Collecting Mastodon.py
  Downloading Mastodon.py-1.1.2-py2.py3-none-any.whl
... etc ...
Successfully installed Mastodon.py-1.1.2 certifi-2017.11.5 chardet-3.0.4 idna-2.6 python-dateutil-2.6.1 pytz-2017.3 requests-2.18.4 six-1.11.0 urllib3-1.22

$ ./mastodon-backup.py wion@mammouth.cafe
-bash: ./mastodon-backup.py: No such file or directory

First question is, where is mastodon-backup.py supposed to be installed?

[wion]

Here are some things I've been suggested...

Confirm your python interpreter and path. Install using homebrew if necessary.

I'm not sure what an "interpreter" is, but this was added to my _.bashprofile file when installing Python 3:

# Setting PATH for Python 3.6
# The original version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.6/bin:${PATH}"
export PATH

So I'm guessing that is correct. I'll leave the Homebrew retry for a last resort.

Another suggestion was...

Isn't that just failing to find the script itself? Make sure you're in the directory you think you're in, that the script exists, that it's in your path, has +x, etc?

What is "+x"?

./ is your current directory, so your shell has to be in the same directory that the .py script is in

What directory is the .py script in? What script, exactly, are we talking about?

the script itself was probably not installed [where you think it was]. Based on my memory of pip3 it should just add the Right Things to your $PATH, so maybe try using the same command line but without the ./ at the front. Also filenames are case sensitive, so make sure you have that right.

I tried the command without ./ but that didn't change anything.

At the moment I'm confused about where "the scripts" were installed, so I can see what might be there or not.

I've done a file search on my system and it does not find either 'Mastodon.py' or 'mastodon-backup.py' anywhere. Should it?

[wion]

$ find Mastodon.py-1.1.2
find: Mastodon.py-1.1.2: No such file or directory

That's odd, considering it supposedly installed successfully (see head post in this thread).

[wion]

Hmm... I do find it by looking at the install path info...

$ cd /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mastodon/
$ ls
Mastodon.py __init__.py __pycache__ streaming.py

But that doesn't tell me where I should be when trying to run the backup script.

And I don't find mastodon-backup.py anywhere.

At this point I would think there was a problem installing Mastodon.py, but that doesn't seem to be the case based on the "Successfully installed..." notification (head post).

[wion]

Bingo! That's what I didn't understand, I guess. I added the file to my user directory and tried:

$ python3 mastodon-backup.py wion@mammouth.cafe

That seems to be working now...

Get statuses (this may take a while)

:)

[kensanata]

Excellent. I added a section about this to the README file.

[wion]

Question now is, do I also need to manual download these files too for the next steps?

• mastodon-backup-media.py
• mastodon-backup-to-text.py

And also run them as...

python3 file.py username@instance.tld

?

[kensanata]

Yes, exactly.

[wion]

It was taking a looooooong time. Then I had to cook dinner and the computer went to sleep. When I opened again I had a bunch of errors of different nature, e.g.

OSError: [Errno 50] Network is down

self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x10636ef60>: Failed to establish a new connection: [Errno 50] Network is down

Etc. I guess this means the computer must not go to sleep during the file generation process?

[kensanata]

Yeah, it can’t resume a broken download and it can’t reconnect when it looses the connection. And it takes so long because Mastodon servers are rare limiting.

[kensanata]

The Mastodon.py library documentation says:

Mastodons API rate limits per IP. By default, the limit is 300 requests per 5 minute time slot. This can differ from instance to instance and is subject to change.

If every request gets 20 toots, then we can get 6000 toots in 5min, at most.

[wion]

Okay. Thanks. DredMorbius said he got about 3K toots off mammouth.cafe in 3-5 minutes, so that's probably a fairly workable baseline to gauge from.

Here's something I just noticed from my Terminal output...

Get user info
Get statuses (this may take a while)

Get favourites (this may take a while)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
OSError: [Errno 50] Network is down

And it goes on to kick out several more errors seemingly around re-connection issues.

But the thing I want to show here is in the first few lines. The statuses/toots seems to have exported okay, but it trips up on the favourites. In fact, I don't really care about or want the favourites. I just want my own texts.

I think a good evolution of this app, FYI, would be to make statuses and favourites two different export action, which might help with speed and reducing connections errors in each case.

I'll check to see if the toots are okay, and can be converted to text file, and if so, then I'm happy.

[kensanata]

I suspect the data is only saved at the end, so you'll be out of luck. If you want a quick fix, replace this code block:

if data is None or not "favourites" in data:
    print("Get favourites (this may take a while)")
    favourites = mastodon.favourites()
    favourites = mastodon.fetch_remaining(
        first_page = favourites)
else:
    id = data["favourites"][0]["id"]
    print("Get new favourites")
    favourites = fetch_up_to(mastodon.favourites(), id)
    favourites.extend(data["favourites"])

with:

favourites = []

(Untested.)

[wion]

I'll try it.

[wion]

That worked on mastodon.social. (Unfortunately mammouth.cafe slipped away before I could get to it.)

[kensanata]

Excellent. Also, the new app, which should be easier to install, now has a --no-favourites option.