Allow to preserve threads

[nemobis]

A thread like https://nitter.it/ecb/status/1592896729091047429#m is currently posted dismembered and in random order https://mamot.fr/@ecb@respublicae.eu/109354553131059238 .

It would be nice to post replies as replies. This probably requires also posting single-threaded, as the order is otherwise difficult to guarantee.

[robertoszek]

I agree, threads are actually a very widely used feature on Twitter and the ability to preserve them when posting them over to the Fedi account would be quite useful. To be honest, it has been on the nice-to-have to-do list for quite a long time. But I think the time has finally come for tackling this next.

The main issue with replies that I haven't found a nice solution to is if they are referencing tweets from other users that don't exist on the Fedi instance. For example, if your user1 (which is mirrored with the bot) replies to a tweet from user2 (which is not mirrored). So user2's tweets don't exist on the Fedi instance and can't be referenced when posting on it.

But in the case of threads, the user is replying to themselves so it would be feasible to implement. We would need to check what user is the author of the tweet that your user is replying to, and if it's themselves replicate the relationship on the Fedi instance.

I'll look into this as it will make for a nice addition and has been requested multiple times.

[robertoszek]

Well, that was a whole lot less painful than I anticipated.

I need to write more extensive tests but feel free to give it a try if you want to: pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.1.1rc12

It should support threads and even replies among the users included in your config if they are on the same Fedi instance. One thing to keep in mind, it only applies for tweets mirrored from this version onwards. It won't work for already posted statuses as it doesn't have a history of the relationship for those.

[nemobis]

Fantastic! It's working! https://respublicae.eu/@EU_Social/109360485015400212

[nemobis]

I'm getting greedy... How difficult would it be to do it for retweets too? :see_no_evil: https://respublicae.eu/@GreensEP/109360828282190126 https://respublicae.eu/@daniel_freund/109359730136814121

[robertoszek]

Alright! This should do it

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.1.1rc15

It will reblog/boost the post if it finds it within the posts published by the users on your config and if they are in the same instance. If it isn't able to find it, it will fallback to the old method ("RT @ blah: blah blah ...").

Same thing as before, this only works for posts published using version 1.1.1rc12 or higher.

Let me know if you run into issues, my local tests only go so far taking into account edgecases. 😅 I tested it with my usual Pleroma, Mastodon and Misskey instances but I may have missed something in the process.

[nemobis]

I don't think the reboost is working... https://respublicae.eu/@EUClimateAction/109370046719699906 https://respublicae.eu/@TimMcPhieEU/109369809693332991

[nemobis]

Oh wait, is that self.posts_ids kept in memory? So this only works for the posts created in the same run of pleroma_bot. I'm running two, and I restart them every few minutes due to https://github.com/robertoszek/pleroma-bot/issues/73 , so that wouldn't work for me. Let's see if I can find a workaround.

[robertoszek]

Ah, that would make sense. The posts_ids were getting dump to disk at the end of the run. If the process was killed before it got the chance to do it then the new posts ids would have been lost.

As a mitigation, you could dump the posts_ids right after every post is successfully published: pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.1.1rc17

Regarding the stuck issue (#73), I've added a flag to specify the number of threads to use when processing tweets. In case it can help us diagnose why is it hanging, perhaps an exception is not being propagated and surfaced to the main thread.

  -t THREADS, --threads THREADS
           number of threads to use when processing tweets

So, you can run it like so:

$ pleroma-bot --threads 1

And it will run in a single thread.

[nemobis]

It's working! I love it. :revolving_hearts:

[nemobis]

One issue is that this fails irrecoverably if, for whatever reason, the JSON file becomes empty. I got a bunch of

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
    posts_ids = json.load(f)      
  File "/usr/lib/python3.9/json/__init__.py", line 293, in load
  File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
  File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
  File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

and had to manually delete the JSON file

[nemobis]

Now with latest rc24 (?), and I think also before, I have some of these:

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://respublicae.eu/api/v1/statuses//reblog

Is it trying to reblog even when the ID is empty?

[robertoszek]

Ah, my bad! I should added a check to make sure there's a post associated to a tweet before trying to boost or reply to it 😅

Regarding the JSON becoming empty issue, I'm thinking that maybe the process is being killed while it has the file opened when updating it. So I've changed the mode it uses to one that doesn't clear the contents of the file when opening it.

I've committed the changes to the develop branch: 7a8eaf429836c6d6adf91fb91b8ac59048f74d43 fa1e1816c7f3114336dc87a41f16391a2544922e

And they are included in 1.1.1rc25, you can install it like so: pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.1.1rc25