[bug] GtS Replies collection not fetched properly by other GtS instances

[stasm]

Describe the bug with a clear and concise description of what the bug is.

GTS doesn't seem to serve replies to posts made by the instance's own users to followers on other instances.

What's your GoToSocial Version?

v0.11.0

GoToSocial Arch

amd64 Docker

What happened?

I follow @gotosocial@superseriousbusiness.org from my single-user GTS instance. The "top-level" posts federate fine, but I don't see many replies to gotosocial's posts (or favorites, or boosts) on my instance. For example, this post has 5 replies, 76 favorites, and 63 boots when I write this. On my instance, I see 0, 0, and 0, respectively.

What you expected to happen?

I'd like gotosocial's GTS instance to forward replies and other activity on their posts to my GTS instance.

How to reproduce it?

If you follow @gotosocial@superseriousbusiness.org, try to view the replies to this post on your instance. As of today, there should be 5.

Anything else we need to know?

No response

[tsmethurst]

Thanks for opening this!

So just for some clarity, it's not necessary, I think, that GoToSocial forwards replies to followers, since this could provide quite a DOS vector (spam replies at a GtS instance and watch it send out thousands and thousands of messages, depending on how many followers you have).

What we do normally in GtS to populate replies on remote posts is just iterate through the Replies collection on that post, look for URIs, and dereference them. In other words, it's normally a pull operation from your instance's side, not a push from the other instance's side.

I think the issue here is mostly that our Replies collection is probably not formatted correctly in a way that is mutually intelligible between GtS instances, and we just hadn't really noticed before because not that many people were using GtS. For fetching replies from Mastodon posts (among others) the pull model seems to work just fine. So I think we just have to investigate how we're serializing Replies; there's probably a little bug somewhere that's making this not work.

[tsmethurst]

^^ Updated issue title, hope that's alright!

[stasm]

Thanks, @tsmethurst. I admit I haven't read the spec, so I don't know what it mandates. My understanding of "forwarding" the replies was based on this post: https://seb.jambor.dev/posts/understanding-activitypub-part-3-the-state-of-mastodon/.

From what you say, it sounds like making sure that two GtS instances can pull replies from each other would achieve the same effect. One question: with the pulling approach, when browsing the timeline, would posts already have correct numbers of replies, boosts and favorites next to them, or would these be 0 at first and the get set to appropriate values once the post is interacted with?

[tsmethurst]

Well, those numbers are generated from the number of likes, boosts, and replies that your instance knows about, so those numbers will probably stay low. The 'source of truth' for a fedi post is almost always from the instance that actually created the post, so your best bet for a mostly-accurate count of those numbers will be to open the post in the browser and see what the origin server thinks.

For what it's worth, this behavior is prettymuch the same on Mastodon. By its decentralized nature ActivityPub confounds attempts to present a 'complete' picture of a post's reception, your server can only ever present a partial slice view of it.

[NyaaaWhatsUpDoc]

With #2220 i'm now finding that for the above post, I can load most of the replies. Would love if anyone else just updating main can confirm this too :)

Also while doing this found...

timestamp="23/09/2023 17:26:59.335" func=dereferencing.(*deref).DereferenceStatusDescendants level=WARN username=kim status=https://gts.superseriousbusiness.org/@gotosocial/statuses/01H7WNDPQTNM31CQVR27R28NTG msg="self referencing collection page(s): https://gts.superseriousbusiness.org/users/gotosocial/statuses/01H7WNDPQTNM31CQVR27R28NTG/replies?only_other_accounts=false&page=true"

I was correct :D sadly

[daenney]

Mmm, doesn't seem to fix it for me :disappointed:. Just updated to main, but loading that post in my clients at first I only had 1 reply. Did a refresh from my client and now I have 2, but no more.

Oh unless the other statuses are from instances I've blocked, I suppose.

[NyaaaWhatsUpDoc]

Okay that does help narrow it down at least. So we know it is attempting to traverse the pages but I'm guessing the pages may not be presenting everything as it should, or perhaps we're getting some weird results from the visibility filter in this code path.

[daenney]

Oh unless the other statuses are from instances I've blocked, I suppose.

I do have mastodon.social blocked, so I wouldn't expect that one to show up. The other instance don't seem to be on my blocklist. So in total I would expect to see 4 responses, but I get 2.

[tsmethurst]

Is this fixed now? I can't remember. I seem to think it is.

[daenney]

We never got around to fixing the self-referential bug I think. So it's still a little broken, though GtS itself is now able to handle that situation.

[NyaaaWhatsUpDoc]

We never got around to fixing the self-referential bug I think. So it's still a little broken, though GtS itself is now able to handle that situation.

wait really? i thought it was: https://github.com/superseriousbusiness/gotosocial/commit/916c6d07ba175eb5c04037ca01a3d3882db9246a

[daenney]

Right. If I remember right, with that change we can handle the situation now, but GtS itself still generates replies collections that are self-referential.

[mirabilos]

it does, I just ran into that as well, now that I talk a lot with several people on another GtS

[NyaaaWhatsUpDoc]

this is handled in #2364