Add support for GoToSocial

ynakao commented 1 year ago

GoToSocial is another Mastodon API compatible ActivityPub server implementation, but not a fork of Mastodon or others.

GoTosocial is still in alpha stage, so they hasn't yet implemented Mastodon/ActivityPub API completely, but I think it works like Mastodon server.

I tried to import tweets from twitter-archive.zip to GoToSocial account with minimal config, but KeyError exception occurred.

config.yml


pleroma_base_url: https://gts.example.com
software: mastodon
users:

twitter_username: nakaoyuji

I got an account ID by following the document.

curl 'https://gts.example.com/api/v2/search?q=&resolve=true&limit=5' -H 'Authorization: Bearer xxxxx'

ID on GoTosocial seems to be 26 alphanumeric charactors.

pleroma_username: 02G7NUT8E2RBPBPZS4OXQ89T1T pleroma_token: xxxxx archive: /path/to/twitter-archive.zip

error.log with minimal config above

2023-01-22 08:46:33,872 pleroma_bot ERROR: Exception occurred for user, skipping...
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/pleroma_bot/cli.py", line 556, in main
user = User(user_item, config, base_path, posts_ids)
File "/usr/lib/python3.10/site-packages/pleroma_bot/cli.py", line 289, in __init__
self.mastodon_enforce_limits()
File "/usr/lib/python3.10/site-packages/pleroma_bot/_utils.py", line 547, in mastodon_enforce_limits
if len(self.display_name[t_user]) > 30:
KeyError: 'nakaoyuji'

I also tried to change config.yml. For example, removing software section to make pleroma-bot assume GtS server is like default pleroma server, and changing pleroma_username section to real GoTosocial username, but all had no effects.

error.log when removing software section

2023-01-22 09:37:14,673 pleroma_bot ERROR: Exception occurred for user, skipping...
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/pleroma_bot/cli.py", line 705, in main
user.check_pinned(posted)
File "/usr/lib/python3.10/site-packages/pleroma_bot/_utils.py", line 340, in check_pinned
).format(str(self.pinned_tweet_id)))
AttributeError: 'User' object has no attribute 'pinned_tweet_id'

error.log when changing pleroma_username to actual GoTosocial username was the same error with the first one echoing KeyError.
version info pleroma-bot: v1.2.0 installed from AUR GoToSocial: v0.6.0

robertoszek commented 1 year ago

Hi!

I'll look into it by finding a GoToSocial instance where I can test the bot with, I've been following the development on and off and waiting for the beta before trying it out.

So far, I see that these endpoints are the ones missing that could cause issues with the bot:

/api/v1/statuses/:id/pin POST (Pin a status to profile)
/api/v1/statuses/:id/unpin POST (Unpin a status from profile)

(and /api/v1/statuses POST (Create a new status) is missing the poll parameter)

Regarding the first config you shared:

pleroma_base_url: https://gts.example.com
software: mastodon
users:
- twitter_username: nakaoyuji
  # I got an account ID by following the document.
  # curl 'https://gts.example.com/api/v2/search?q=<username>&resolve=true&limit=5' -H 'Authorization: Bearer xxxxx'
  # ID on GoTosocial seems to be 26 alphanumeric charactors.
  pleroma_username: 02G7NUT8E2RBPBPZS4OXQ89T1T
  pleroma_token: xxxxx
  archive: /path/to/twitter-archive.zip

I'm curious about the error, perhaps it is unrelated to the target instance being GoToSocial.

Could you set up a virtual environment, install 1.2.1rc9 on it and run the config with it?:

$ python3 -m venv testgts
$ source testgts/bin/activate
(testgts) $ pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc9
(testgts) $ pleroma-bot --config /path/to/config

I'm curious if it would fail in some other unexpected way.

ynakao commented 1 year ago

Thank you for your reply.

I installed v1.2.1rc9 and ran pleroma-bot as you described. Now the error message changed to AttributeError: 'User' object has no attribute 'pinned_tweet_id'. So, is this because GoToSocial hasn't implement relevant API yet?

2023-01-23 20:40:15,025 pleroma_bot ERROR: Exception occurred for user, skipping...
Traceback (most recent call last):
  File "/home/ynakao/test/pleroma-bot/pleroma-bot-v1.2.1-rc9/lib/python3.10/site-packages/pleroma_bot/cli.py", line 709, in main
    user.check_pinned(posted)
  File "/home/ynakao/test/pleroma-bot/pleroma-bot-v1.2.1-rc9/lib/python3.10/site-packages/pleroma_bot/_utils.py", line 340, in check_pinned
    ).format(str(self.pinned_tweet_id)))
AttributeError: 'User' object has no attribute 'pinned_tweet_id'

robertoszek commented 1 year ago

So, is this because GoToSocial hasn't implement relevant API yet?

Most likely, perhaps we can simply skip the pinning functionality completely if the target is GoToSocial for now.

Any luck with 1.2.1rc10?:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc10

ynakao commented 1 year ago

Yes! It looks working well! I didn't check whether all imports are properly done though, since it will take over 14 hours to finish.

robertoszek commented 1 year ago

@ynakao did you run into any new issues during your import?

I spun up a local GoToSocial instance to test with and I found that the response code for attachments being too big is different than on Mastodon/Pleroma (HTTP 400 instead of 413). So I made some changes to take that into account on 1.2.1rc16

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc16

Everything else seemed to work as I expected. (except pinning posts and polls, which are not implemented in GoToSocial yet, so we skip them)

Posting images:

Posting videos:

Updating the profile:

I'm going to keep trying to find edgecases but let me know if you ran into anything else weird.

ynakao commented 1 year ago

I encountered the following TypeError on a newly created GoToSocial test account. Is there a way to spot which tweet causes this issue easily?

(pleroma-bot-v1.2.1-rc16) [hetzarch@hetzarch][~/test/pleroma-bot/pleroma-bot-v1.2.1-rc16]% ./bin/pleroma-bot --archive twitter-archive-2022-12-27.zip
...
⚠ 2023-01-28 13:36:58,595 - pleroma_bot - WARNING - Raising max_tweets to the maximum allowed value (_utils.py:610)
ℹ 2023-01-28 13:37:05,061 - pleroma_bot - INFO - tweets gathered:        23497
Processing tweets... : 100%|█████████████████████████████████████████████████████████████| 23497/23497 [04:53<00:00, 80.19it/s]
ℹ 2023-01-28 13:41:58,113 - pleroma_bot - INFO - tweets to post:         23497
Posting tweets... :  82%|██████████████████████████████████████████████████▋           | 19205/23497 [2:54:53<39:28,  1.81it/s]✖ 2023-01-28 16:36:52,759 - pleroma_bot - ERROR - Exception occurred for user, skipping... (cli.py:721)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc16/lib/python3.10/site-packages/pleroma_bot/cli.py", line 688, in main
    post_id = user.post(
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc16/lib/python3.10/site-packages/pleroma_bot/_utils.py", line 794, in post
    post_id = self.post_pleroma(tweet, poll, sensitive, media, cw=cw)
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc16/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 310, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable

robertoszek commented 1 year ago

It looks like it failed to upload an attachment. I've added a handler for the error that should display the Tweet ID, the response from the GoToSocial instance and which file was trying to upload on 1.2.1rc18:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc18

Hopefully that would give us enough info to identify what's the issue there.

ynakao commented 1 year ago

It seems error occurs when uploading mp4 files. I guess this is because video support in GoToSocial will be landed on v0.7.0 and I'm running the latest release v0.6.0. Are you running GoToSocial latest git commit version as you can post videos in your test?

Except for this, all operation worked properly in my case.

Error log

```shell (pleroma-bot-v1.2.1-rc18) [hetzarch@hetzarch][~/test/pleroma-bot/pleroma-bot-v1.2.1-rc18]% ./bin/pleroma-bot --archive twitter-archive-2022-12-27.zip ... ⚠ 2023-01-29 09:59:48,337 - pleroma_bot - WARNING - Raising max_tweets to the maximum allowed value (_utils.py:612) ℹ 2023-01-29 09:59:54,767 - pleroma_bot - INFO - tweets gathered: 23497 Processing tweets... : 100%|█████████████████████████████████████████████████████████████| 23497/23497 [04:37<00:00, 84.69it/s] ℹ 2023-01-29 10:04:32,289 - pleroma_bot - INFO - tweets to post: 23497 Posting tweets... : 82%|██████████████████████████████████████████████████▋ | 19205/23497 [2:54:55<39:33, 1.81it/s]✖ 2023-01-29 12:59:28,928 - pleroma_bot - ERROR - Error uploading media: Status code: 200 Tweet ID: 773124013677654016 Filename: /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/773124013677654016/773124013677654016-PTZmtVxlXHRDGMcg.mp4 Size: 0.09MB Response: null (_pleroma.py:317) ✖ 2023-01-29 12:59:28,929 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333) Traceback (most recent call last): File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma media_id = json.loads(response.text)["id"] TypeError: 'NoneType' object is not subscriptable Posting tweets... : 82%|██████████████████████████████████████████████████▋ | 19209/23497 [2:54:58<41:13, 1.73it/s]✖ 2023-01-29 12:59:31,233 - pleroma_bot - ERROR - Error uploading media: Status code: 200 Tweet ID: 773142132597125120 Filename: /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/773142132597125120/773142132597125120-CrrAqk5VYAAWl88.mp4 Size: 0.1MB Response: null (_pleroma.py:317) ✖ 2023-01-29 12:59:31,234 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333) Traceback (most recent call last): File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma media_id = json.loads(response.text)["id"] TypeError: 'NoneType' object is not subscriptable Posting tweets... : 84%|███████████████████████████████████████████████████▉ | 19699/23497 [2:59:30<36:35, 1.73it/s]✖ 2023-01-29 13:04:03,915 - pleroma_bot - ERROR - Error uploading media: Status code: 200 Tweet ID: 811535032577732609 Filename: /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/811535032577732609/811535032577732609-5xBU-2VYTfLvI_FV.mp4 Size: 0.54MB Response: null (_pleroma.py:317) ✖ 2023-01-29 13:04:03,921 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333) Traceback (most recent call last): File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma media_id = json.loads(response.text)["id"] TypeError: 'NoneType' object is not subscriptable Posting tweets... : 84%|███████████████████████████████████████████████████▉ | 19701/23497 [2:59:31<36:10, 1.75it/s]✖ 2023-01-29 13:04:05,060 - pleroma_bot - ERROR - Error uploading media: Status code: 200 Tweet ID: 811538743119052800 Filename: /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/811538743119052800/811538743119052800-C0MqLTtVQAAwNfx.mp4 Size: 0.22MB Response: null (_pleroma.py:317) ✖ 2023-01-29 13:04:05,061 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333) Traceback (most recent call last): File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma media_id = json.loads(response.text)["id"] TypeError: 'NoneType' object is not subscriptable Posting tweets... : 94%|██████████████████████████████████████████████████████████▍ | 22140/23497 [3:22:16<14:28, 1.56it/s]✖ 2023-01-29 13:26:49,244 - pleroma_bot - ERROR - Error uploading media: Status code: 200 Tweet ID: 1114169013477629952 Filename: /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/1114169013477629952/1114169013477629952-D3ZSzfbU0AAxZg3.mp4 Size: 0.12MB Response: null (_pleroma.py:317) ✖ 2023-01-29 13:26:49,245 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333) Traceback (most recent call last): File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma media_id = json.loads(response.text)["id"] TypeError: 'NoneType' object is not subscriptable Posting tweets... : 99%|█████████████████████████████████████████████████████████████▎| 23253/23497 [3:32:43<02:33, 1.59it/s]✖ 2023-01-29 13:37:16,749 - pleroma_bot - ERROR - Error uploading media: Status code: 200 Tweet ID: 1505537146530430977 Filename: /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/1505537146530430977/1505537146530430977-FOS96kZaMAEFNmy.mp4 Size: 0.11MB Response: null (_pleroma.py:317) ✖ 2023-01-29 13:37:16,749 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333) Traceback (most recent call last): File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma media_id = json.loads(response.text)["id"] TypeError: 'NoneType' object is not subscriptable Posting tweets... : 100%|██████████████████████████████████████████████████████████████| 23497/23497 [3:35:02<00:00, 1.82it/s] ℹ 2023-01-29 13:39:34,789 - pleroma_bot - INFO - Updating profile: (pleroma-bot-v1.2.1-rc18) [hetzarch@hetzarch][~/test/pleroma-bot/pleroma-bot-v1.2.1-rc18]% ```

robertoszek commented 1 year ago

Are you running GoToSocial latest git commit version as you can post videos in your test?

Yep, I just pulled the latest snapshot: https://hub.docker.com/r/superseriousbusiness/gotosocial/tags

https://docs.gotosocial.org/en/latest/installation_guide/docker/#version

robertoszek / pleroma-bot

Add support for GoToSocial #115

I got an account ID by following the document.

curl 'https://gts.example.com/api/v2/search?q=&resolve=true&limit=5' -H 'Authorization: Bearer xxxxx'

ID on GoTosocial seems to be 26 alphanumeric charactors.