Lynnesbian / mstdn-ebooks

NO LONGER BEING UPDATED, NOT GUARANTEED TO WORK AT ALL
Mozilla Public License 2.0
34 stars 29 forks source link

Grabing posts from Pleroma fails #45

Open Eskuero opened 4 years ago

Eskuero commented 4 years ago
  1. Clean run without any toots.db generated yet
  2. Bot account is following this accounts (raru.me are mastodon and the other two are pleroma instances)
    • bunni@kitsune.cafe
    • bunni@raru.re
    • stellar@raru.re
    • Stellar@fedi.absturztau.be
  3. Running the bot:

Using config.json as configuration file Downloading posts for user @bunni@kitsune.cafe, starting from 0 WebFingering... Using standard mode Downloading and saving posts............................................ Done! Downloading posts for user @bunni@raru.re, starting from 0 WebFingering... Using standard mode Downloading and saving posts................................. Done! Downloading posts for user @stellar@raru.re, starting from 0 WebFingering... Using standard mode Downloading and saving posts........... Done! Downloading posts for user @Stellar@fedi.absturztau.be, starting from 0 WebFingering... Using standard mode Traceback (most recent call last): File "/app/main.py", line 215, in j = r.json() File "/usr/lib/python3.8/site-packages/requests/models.py", line 897, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib/python3.8/json/init.py", line 357, in loads return _default_decoder.decode(s) File "/usr/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

  1. As it can be seen on the fedi.absturztau.me fails to grab posts.
  2. Re-run the sync:

Using config.json as configuration file Downloading posts for user @bunni@kitsune.cafe, starting from 40dd49ac-3b6e-4ea6-9f99-7415a57bed3c WebFingering... Using standard mode Traceback (most recent call last): File "/app/main.py", line 215, in j = r.json() File "/usr/lib/python3.8/site-packages/requests/models.py", line 897, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib/python3.8/json/init.py", line 357, in loads return _default_decoder.decode(s) File "/usr/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

  1. And now the other pleroma instance (kitsune.cafe) starts failing too. I tried waiting for half an hour in case it was a rate-limit but it still kept failing
w3bb commented 3 years ago

For anyone reading this issue, I've made an alternative scraper that isn't broken with Pleroma. It should work perfectly with your existing toots.db. Although make sure to do backups.

https://git.sr.ht/~welt/fuckpleroma

Install Go, and run go get -u spiderden.net/go/fuckpleroma

The binary should end up in your $GOPATH, which by default is ~/go/bin.