Groups over a ZAP Server do not work anymore

rocky-III commented 3 years ago

on Mastodon v3.4.0 AP Groups over a ZAP Server do not work anymore... Groups did work for the last years but apparently there were made some changes in the Mastodon Code that now it does not work anymore. the ZAP dev explains:

After further testing, this appears to be a Mastodon bug/regression. They once supported relay delivery of comment activities (which is required by the AP spec) as long as they are signed with a valid LD-signature by the comment author and a valid HTTP signature of the sender, but they no longer appear to do so. These comments arrive at the Mastodon server and are accepted with a 2xx code, but not only don't arrive in the recipient's stream, but they don't arrive in the public inbox either - and that address is in the primary audience.

So Mastodon appears to simply be losing or discarding these activities without any obvious explanation.

maybe @Gargron can give a hint and solve the problem on the Mastodon side. THANKS

ClearlyClaire commented 3 years ago

The only thing that comes to mind is a change from several years ago, so I doubt this is the issue.

Do you have an example payload, or a group we could follow to investigate the issue?

redmatrix commented 3 years ago

This could be from a few years back. After we got groups working with Mastodon back in 2017-2018 I haven't done anything to verify that it still works; and just blamed the lack of Mastodon participation on the fact that we're not very popular with Mastodon folks.

Here is a sample payload. It was written by https://macgirvin.com/channel/mike as a comment to a post in the Zap group https://z.macgirvin.com/channel/zap (who is responsbile for distributing it to group members). The HTTPsignature belongs to the group actor. You can join the group by following it and post to it by sending a DM to the group. This top level post is embedded in a create/note which is authored by the group actor and sent to all the members. There-after all comments are just normal comments and are relayed by the group to all its members by virtue of having a valid LD-signature and making use of ActivityPub's S2S comment relay (commenter sends the post to the top-level actor who redistirbutes it to all their followers). .

{
    "@context": [
        "https://www.w3.org/ns/activitystreams",
        "https://w3id.org/security/v1",
        {
            "zot": "https://macgirvin.com/apschema#",
            "toot": "http://joinmastodon.org/ns#",
            "ostatus": "http://ostatus.org#",
            "schema": "http://schema.org#",
            "litepub": "http://litepub.social/ns#",
            "sm": "http://smithereen.software/ns#",
            "conversation": "ostatus:conversation",
            "manuallyApprovesFollowers": "as:manuallyApprovesFollowers",
            "oauthRegistrationEndpoint": "litepub:oauthRegistrationEndpoint",
            "sensitive": "as:sensitive",
            "movedTo": "as:movedTo",
            "copiedTo": "as:copiedTo",
            "alsoKnownAs": "as:alsoKnownAs",
            "inheritPrivacy": "as:inheritPrivacy",
            "EmojiReact": "as:EmojiReact",
            "commentPolicy": "zot:commentPolicy",
            "topicalCollection": "zot:topicalCollection",
            "eventRepeat": "zot:eventRepeat",
            "emojiReaction": "zot:emojiReaction",
            "expires": "zot:expires",
            "directMessage": "zot:directMessage",
            "Category": "zot:Category",
            "replyTo": "zot:replyTo",
            "PropertyValue": "schema:PropertyValue",
            "value": "schema:value",
            "discoverable": "toot:discoverable",
            "wall": "sm:wall"
        }
    ],
    "type": "Create",
    "id": "https://macgirvin.com/activity/90ce1ca6-3b5b-41b2-94d6-272caf8b644f",
    "published": "2021-06-10T07:35:36Z",
    "context": "https://z.macgirvin.com/activity/a52d25fd-7eb2-42d6-9aff-009a432d93a4",
    "conversation": "https://z.macgirvin.com/activity/a52d25fd-7eb2-42d6-9aff-009a432d93a4",
    "actor": "https://macgirvin.com/channel/mike",
    "replyTo": "https://z.macgirvin.com/channel/zap",
    "url": "https://macgirvin.com/activity/90ce1ca6-3b5b-41b2-94d6-272caf8b644f",
    "object": {
        "type": "Note",
        "id": "https://macgirvin.com/item/90ce1ca6-3b5b-41b2-94d6-272caf8b644f",
        "published": "2021-06-10T07:35:36Z",
        "attributedTo": "https://macgirvin.com/channel/mike",
        "inReplyTo": "https://z.macgirvin.com/activity/a52d25fd-7eb2-42d6-9aff-009a432d93a4",
        "context": "https://z.macgirvin.com/activity/a52d25fd-7eb2-42d6-9aff-009a432d93a4",
        "conversation": "https://z.macgirvin.com/activity/a52d25fd-7eb2-42d6-9aff-009a432d93a4",
        "content": "test comment",
        "source": {
            "content": "test comment",
            "mediaType": "text/x-multicode"
        },
        "replyTo": "https://z.macgirvin.com/channel/zap",
        "url": "https://macgirvin.com/item/90ce1ca6-3b5b-41b2-94d6-272caf8b644f",
        "tag": [
            {
                "type": "Mention",
                "href": "https://z.macgirvin.com/channel/zap",
                "name": "@zap@z.macgirvin.com"
            }
        ],
        "to": [
            "https://www.w3.org/ns/activitystreams#Public",
            "https://z.macgirvin.com/channel/zap"
        ],
        "cc": [
            "https://macgirvin.com/followers/mike",
            "https://z.macgirvin.com/followers/zap"
        ]
    },
    "tag": [
        {
            "type": "Mention",
            "href": "https://z.macgirvin.com/channel/zap",
            "name": "@zap@z.macgirvin.com"
        }
    ],
    "to": [
        "https://www.w3.org/ns/activitystreams#Public",
        "https://z.macgirvin.com/channel/zap"
    ],
    "cc": [
        "https://macgirvin.com/followers/mike",
        "https://z.macgirvin.com/followers/zap"
    ],
    "signature": {
        "type": "RsaSignature2017",
        "nonce": "81fad5e56f637f2a8a094fe251e660df04c1cf7f204538d34ce075ef937ed4f2",
        "creator": "https://macgirvin.com/channel/mike",
        "created": "2021-06-10T07:35:36Z",
        "signatureValue": "GdSUGXQIXM/IAu7vreRrOJKNRdpgDENJ4S0ZkmMIpkdWLizckz+n7WqjmCBZ2ZXSwtJt3QNsBnYFIjEHspLwpxjtCLiZvNO8BXd8WAMxxuteKmoJHBf5cBWECDY8IrOLT0UeqBW4lv33UmcTkcqocMYgOUS2NRO+O63LbSW81QqmrApNXVF8R4S3hbAIlR67NdnUmkojMMBvs/zLOI4f7eVdp2f0lq5M8ISIahbWyTpdbsTBGzVabbgtZwICQbyKWeM3Nmmnczg0W/aQEq2VlsDWG/nxMBmqpjveTXsxqNn237zHKu8XOU5RAfsDwwN+l/VXf4ko623bu5l1D9KJjVfWF4t1HIsuBSNo5bFIb13Xg/HLYUKSh2bzbsatVvoiuW2Cd2bqgl8G68zVmFs7J724EiXhpDJ3qYf6lsCQIePSESqs0xXZGWdSJY9i9/1hefA/ip7VSzqH6XTQcnCVXcqpiQuKMjJRlqyGbq24SI3teqCTCkX+jzb8YZSBzctfq6fbxuaiwDlSSkkluPpUEhZi+hzV4u+NsMPMWeX33sQtspTAVzcJNzSN2es/R0ehiXPiO1Fg1liKGf2y3rF/sYYf9uIGzDDIGQ6tgWDQsrwBtDtVXB4Rp1DIovmXAHBEo9PhJlhY/9TiVmNxB7AZc3UYT94efHumxT3qDH99Nvs="
    }
}

ClearlyClaire commented 3 years ago

Comments seem to federate just fine, I successfully imported that payload as-is, as if it were properly signed by the ZAP group actor. You can probably see the comment from Mastodon under the account's “posts and replies”.

Now, the comments does not show up in the home timeline or anywhere else than the “posts and replies” tab of the poster because it is a reply to a post that for some reason could not be fetched. I will now investigate why the original post could not be fetched.

EDIT: The main reason seems to be that Mastodon expects inReplyTo to point to the Note (or otherwise supported) object, and not a Create activity. I will investigate whether that's the only reason and whether Mastodon could handle Create-wrapped objects sensibly. But in any case, your representation is seriously confusing, how could the Create and the object it creates have the same ActivityPub id?? EDIT2: I'd have to give a thought about the proper process but indeed, it seems like adding support for fetching Create activities would work for your use case, although I'm still very confused about both your activity and object having the same id, as well as your links embedding the requesting user handle

ClearlyClaire commented 3 years ago

To sum up:

posts directly sent/relayed by the group should work fine, and I don't think any recent change has changed anything to that
however, when they are in-reply-to posts that could not be fetched, they don't appear in the Home TL (this is by design, to avoid showing comments without context)
in this case, fetching the replied-to post does not work as Mastodon expects to find an actual ActivityStreams object like a Note but finds a Create instead: I will consider whether it makes sense to support Create activities as well. This has never been handled by Mastodon

Some side notes on your objects/activities:

having both your Create activities and wrapped objects share the same id is confusing and probably wrong
it seems like when fetched with a signed request, the server rewrites links to include the requester's handle, adding something like “?f=&zid=username%40domain”. I'm not sure what is the exact purpose of this, but clicking on such links redirect to an endpoint https://domain/magic that is not handled by Mastodon. As Mastodon signs every fetch with an actor representing the instance, this will break pretty much all links.

redmatrix commented 3 years ago

@ClearlyClaire

The 'zid' parameter is added by OpenWebAuth (our cross-domain single sign-on solution), which as you note Mastodon does not support. I will investigate why this is being attached to Mastodon fetches - this should not happen unless the Mastodon server provides the relevant OWA discovery info.

Our activities use an 'id' endpoint based on a path of 'activity' and objects within that activity use an 'id' path of 'item' but often with the same UUID. They do not use the same id. Yes, we will occasionally set inReplyTo to an activity instead of the enclosed object. This is entirely legal from my reading of the vocabulary spec but I will review.

inReplyTo: Indicates one or more entities for which this object is considered a response.

The term "entities" seems to be the problem here. What's weird is that comments ever worked with Mastodon because we've used this method of inReplyTo before Mastodon even implemented ActivityPub and yet we've been conversing for several years.

In any event, thank you for the assistance.

redmatrix commented 3 years ago

Aha - I see where the id's are getting confused. It's only on top-level group deliveries and appears to also be the only place we're replying to an activity rather than an object -- and that symptom is due to the same issue. Everywhere else it's all working as expected and in a manner that's compatible with Mastodon's assumptions about what ActivityPub should look like. Should have it sorted shortly.

Haven't yet been able to reproduce seeing an OpenWebAuth link in an ActivityPub object. So that appears to be a particular edge case as well. I'll find it eventually. If you've still got a link that might help to track it down, that might be helpful however I will track it down either way.

Gargron commented 3 years ago

Is OpenWebAuth worth adding to Mastodon?

redmatrix commented 3 years ago

Disappointing: I was hopeful we could quickly put this to rest but fixing the id's and inReplyTo didn't actually fix the problem. On my dev site everything is now fetchable and does not contain any problematic URL parameters and matches the requirements you described earlier and yet third party comments relayed by the group actor still do not arrive in the Mastodon home stream. If the comment is made by a followee it is included in that stream, but not otherwise.

I will update this issue with an updated payload in a day or two when I've thoroughly reviewed the latest fixes and have them on a public facing server.

redmatrix commented 3 years ago

Is OpenWebAuth worth adding to Mastodon?

That's up to you. I kind of like being able to traverse large swaths of the fediverse and have immediate access to restricted objects and be able to post and reply on other sites as if it were a single large connected system, without being asked who I am and where I'm from and sent back to my own site to actually interact with the site I'm currently looking at.

It's a different usage model and changes core security and identity assumptions. Truth be told it's probably much more difficult to change the way these core layers work after so much functionality has been built on top of them than if you had done this back in the beginning.

ClearlyClaire commented 3 years ago

Yes, we will occasionally set inReplyTo to an activity instead of the enclosed object. This is entirely legal from my reading of the vocabulary spec but I will review.

Yes, that seems legal to me too, but that isn't handled by Mastodon (yet?). AP/AS is exceptionally broad in general…

If you've still got a link that might help to track it down, that might be helpful however I will track it down either way.

If you still need help with that, one such link is https://macgirvin.com/channel/mike?f=&zid=sitedethib.com%40social.sitedethib.com

yet third party comments relayed by the group actor still do not arrive in the Mastodon home stream. If the comment is made by a followee it is included in that stream, but not otherwise.

Do the messages properly appear as attached to the parent post for servers who do not follow the replier? If so, that's expected behavior: the Home Timeline is a feed and not a wall, it only contains posts Created or Announced by you or people you follow. As a Mastodon user, I would not expect that to change by following a group. However, I would not be too surprised by the group Announceing the in-group replies.

redmatrix commented 3 years ago

However, I would not be too surprised by the group Announceing the in-group replies.

We did exactly that long ago to federate groups with Diaspora who also didn't support them natively. People hated it and cussed at us for years afterward for that decision.

Thanks for your efforts.

ClearlyClaire commented 3 years ago

Alright, I think we can close this issue now that the Create thing has been fixed, as I think the remaining is just down to both platforms having different designs.

ClearlyClaire commented 3 years ago

Closing as it does not appear to be a regression nor truly a bug, but rather differences in design across those two pieces of software. Feel free to reopen if you disagree.

mastodon / mastodon

Groups over a ZAP Server do not work anymore #16380