search

snarfed / bridgy-fed

🌉 A bridge between decentralized social network protocols
https://fed.brid.gy
Creative Commons Zero v1.0 Universal
754 stars 39 forks source link

interpret Mastodon ActivityPub HTTP 202 response as failure #16

Closed snarfed closed 6 years ago

snarfed commented 7 years ago

mastodon's AP code returns HTTP 202 to inbox POSTs that fail signature verification:

TODO:

snarfed commented 7 years ago

cc @mblaney

snarfed commented 7 years ago

aha, looks like mastodon changed this 202 to a 401 in head (2.0.0): https://github.com/tootsuite/mastodon/commit/dfaa219f8820224d37cd060d253a507111c63460#diff-b71581bf00ae84fbac757c5373533ebb

snarfed commented 6 years ago

tentatively reopening. i'm seeing 202 from mastodon again on Create but the reply isn't showing up, e.g. https://snarfed.org/2018-01-26_chris-aldrich-%f0%9f%8d%8d-anarchivistchaos-social-edsudigipres-club-sn-mastodon to https://mastodon.social/@chrisaldrich/99418039176997784 .

https://github.com/tootsuite/mastodon/blob/master/app/controllers/activitypub/inboxes_controller.rb at master shows that mastodon evidently returns 202 for success, so probably something else is going on that prevents it from appearing.

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "object": {
    "inReplyTo": "https://mastodon.social/users/chrisaldrich/statuses/99418039176997784",
    "cc": [
      "https://www.w3.org/ns/activitystreams#Public",
      "https://mastodon.social/users/chrisaldrich",
      "https://mastodon.social/users/chrisaldrich/followers",
      "https://mastodon.technology/users/snarfed",
      "https://digipres.club/users/edsu",
      "https://chaos.social/users/anarchivist"
    ],
    "attributedTo": [
      {
        "preferredUsername": "snarfed.org",
        "name": "Ryan Barrett",
        "url": "https://snarfed.org/",
        "image": [
          {
            "url": "https://secure.gravatar.com/avatar/947b5f3f323da0ef785b6f02d9c265d6?s=96&d=blank&r=g",
            "type": "Image"
          }
        ],
        "type": "Person",
        "id": "https://fed.brid.gy/snarfed.org",
        "icon": [
          {
            "url": "https://secure.gravatar.com/avatar/947b5f3f323da0ef785b6f02d9c265d6?s=96&d=blank&r=g",
            "type": "Image"
          }
        ]
      }
    ],
    "tag": [
      {
        "name": "reply"
      }
    ],
    "id": "https://snarfed.org/2018-01-26_chris-aldrich-%f0%9f%8d%8d-anarchivistchaos-social-edsudigipres-club-sn-mastodon",
    "name": "reply \n\nJanuary 26, 2018 Ryan Barrett\nLeave a comment\n\n\nthanks chris! bridgy fed is definitely open and ready for (more) users. it\u2019s not perfect yet, but it is usable. i\u2019m using it to post this reply right now! \n\nStandard",
    "url": "https://snarfed.org/2018-01-26_chris-aldrich-%f0%9f%8d%8d-anarchivistchaos-social-edsudigipres-club-sn-mastodon",
    "content": " <p>thanks chris! bridgy fed is definitely open and ready for (more) users. it\u2019s not perfect yet, but it is usable. i\u2019m using it to post this reply right now! <a class=\"u-in-reply-to\" href=\"https://mastodon.social/@chrisaldrich/99418039176997784\"></a> <cite class=\"via\"><a href=\"https://fed.brid.gy/\"></a></cite></p> ",
    "published": "2018-01-26T12:50:25-08:00",
    "@context": "https://www.w3.org/ns/activitystreams",
    "type": "Note"
  },
  "type": "Create"
}
snarfed commented 6 years ago

i suspect this is due to settings on mastodon.social specifically (currently 2.2.0rc2), since replies and likes still work on lgbt.io (2.1.2), chaos.social (2.1.2), and others.

swentel commented 6 years ago

Some feedback re: 202 responses.

Been playing around with integrating Drupal. I've done 3 tests, of which 2 work fine. Test url is https://mastodon.technology/@snarfed/3194674

What works is like (https://realize.be/like/1468) and repost (https://realize.be/repost/1469) Interestingly enough, I got "202 response! If this is Mastodon 1.x, their signature verification probably failed. :(" back for both responses.

When trying to send a reply, I get back the same response, but nothing shows up on the post on mastodon itself.

(It might be that I'm missing something, currently, I have the .htaccess and atom feed, but nothing done yet with WebSub. But AFAICT, that shouldn't matter for sending a webmention.)

valpackett commented 6 years ago

mastodon 2.4.2, replies still don't show up (log), and likes do still seem to work

swentel commented 6 years ago

So out of curiosity, I tried a reply again today :) (to https://jawns.club/@timplunkett/100606109220664799)

The latest reply I tested now gives something back like 'Public key not found for key acct:realize.be@realize.be'. I haven't checked yet what that means and/or if it's relevant, but just wanted to leave it here.

snarfed commented 6 years ago

@swentel thanks for the update! that looks like #29, but the conclusion there was that they needed a representative h-card (https://github.com/snarfed/bridgy-fed/issues/29#issuecomment-416076503), which isn't your problem. https://fed.brid.gy/.well-known/webfinger?resource=realize.be@realize.be works fine.

...ah, but your site isn't actually set up for bridgy fed. https://realize.be/.well-known/host-meta and https://realize.be/.well-known/webfinger currently 404. they should redirect to fed.brid.gy instead. https://fed.brid.gy/#setup

swentel commented 6 years ago

Right - my bad, I commented out the lines in my .htaccess .. still the same outcome now.

snarfed commented 6 years ago

thanks! it actually gets a 202 from mastodon now. so we're back at the original issue here. whee!

https://fed.brid.gy/log?start_time=1536433273&key=https%3A%2F%2Frealize.be%2Freply%2Fcontent%2F1541+https%3A%2F%2Fjawns.club%2F%40timplunkett%2F100606109220664799

swentel commented 6 years ago

So, I checked mastodon, in app/controllers/activitypub/inboxes_controller.rb it will return 202 when the request is signed valid. But process_payload can still well basically do nothing. Kind of weird behavior if you tell me ..

swentel commented 6 years ago

So I finally found out why replies don't show up. The problem is that mastodon compares the id on the object with the actor url.

See https://github.com/tootsuite/mastodon/blob/master/app/lib/activitypub/activity/create.rb#L8 See https://github.com/tootsuite/mastodon/blob/master/app/lib/activitypub/activity/create.rb#L269

Since fed.brid.gy sends the url of the post on the website as id, and the actor url is something like https://fed.brid.gy/{sitename} this will now fail. Also, the 'url' property in an object gets the same check, but is ignored when it fails, the reply will still show up if the id check is fine.

When sending an id which looks like 'https://fed.brid.gy/realize.be?unique_id_here', the reply will show up. This has an impact on the canonical URL of the post being fed.brid.gy and not the website url. As a consequence, I guess round trips can't work anymore. One workaround could be to create the id in the form of https://fed.brid.gy/realize.be?url={original_url} - that works (see the 'Welcome to the club! AGAIN' canonical url). Of course, if at some point mastodon decides to start ignore get params, then it's back to the drawing table.

(I guess this also has impact on discovery of posts as they basically will have the same problem)

I often wonder whether it would be possible to set the id of https://fed.brid.gy/realize.be' to realize.be, but I'm afraid that ain't possible at all ?

Working replies: https://mastodon.social/users/swentel/statuses/100776214865184909 https://mastodon.technology/@snarfed/3194674

swentel commented 6 years ago

this is the minimal object that I send:

stdClass Object
(
    [@context] => https://www.w3.org/ns/activitystreams
    [type] => Create
    [actor] => https://fed.brid.gy/realize.be
    [to] => Array
        (
            [0] => https://www.w3.org/ns/activitystreams#Public
            [1] => https://mastodon.social/users/swentel
        )

    [object] => stdClass Object
        (
            [id] => https://fed.brid.gy/realize.be?unique-id-needed-7
            [url] => https://realize.be/reply/content/1567
            [type] => Note
            [published] => 2018-09-26T17:27:21Z
            [attributedTo] => https://fed.brid.gy/realize.be
            [inReplyTo] => https://mastodon.social/users/swentel/statuses/100776214865184909
            [content] => <a href="https://mastodon.social/users/swentel">@swentel</a> A minimal object
            [to] => Array
                (
                    [0] => https://www.w3.org/ns/activitystreams#Public
                    [1] => https://mastodon.social/users/swentel
                )

        )

)

This is another one which also includes the 'mention' tag. This makes it so that a notification is also send to the user. The reply above did show up on the post, so it isn't that much of a deal.

So, I guess the when posting, bridgy should prepend @{username}. Or, documentation could also say that we we find @{username} urls in the post, we add the mention tag. (this could be another issue of course, but wanted to document this too)

stdClass Object
(
    [@context] => https://www.w3.org/ns/activitystreams
    [type] => Create
    [actor] => https://fed.brid.gy/realize.be
    [to] => Array
        (
            [0] => https://www.w3.org/ns/activitystreams#Public
            [1] => https://mastodon.social/users/swentel
        )

    [object] => stdClass Object
        (
            [id] => https://fed.brid.gy/realize.be?unique-id-needed-6
            [url] => https://realize.be/reply/content/1567
            [type] => Note
            [published] => 2018-09-26T17:28:38Z
            [attributedTo] => https://fed.brid.gy/realize.be
            [inReplyTo] => https://mastodon.social/users/swentel/statuses/100776214865184909
            [content] => <a href="https://mastodon.social/users/swentel">@swentel</a> with url
            [to] => Array
                (
                    [0] => https://www.w3.org/ns/activitystreams#Public
                    [1] => https://mastodon.social/users/swentel
                )

            [tag] => Array
                (
                    [0] => stdClass Object
                        (
                            [type] => Mention
                            [href] => https://mastodon.social/users/swentel
                            [name] => @swentel
                        )

                )

        )

)
snarfed commented 6 years ago

this is really really great progress. thank you so much @swentel!

swentel commented 6 years ago

So I've been reading the comparison code. It compares the id of the object with @account.uri which is fetched here: https://github.com/tootsuite/mastodon/blob/master/app/services/resolve_account_service.rb#L123 and then goes to https://github.com/tootsuite/mastodon/blob/master/app/services/resolve_account_service.rb#L161

There's a lot going on here, it goes to fetch the atom feed, and then tries to find the author uri there, if I'm reading that right. Now, my atom feed uses granary, but it seems fine afaics. When nothing is found, it either fills in nothing or falls back to author_uri.content - which I have no idea atm what this does. Haven't done much ruby, so clueless.

Anyway, I feel like there might be a way out here, but I might be naive. It's also late now, so might not be fresh anymore :)

swentel commented 6 years ago

So when running that code on the body of my atom feed, it returns http://realize.be - out of ideas again :)

snarfed commented 6 years ago

@swentel i think you're right that the best fix here is probably to wrap our translated post object ids under https://fed.brid.gy/... somewhere, so that they have the same prefix as the already wrapped actor ids.

(i doubt they'd drop query parameters when checking uniqueness, but if they do, we can always put the original post URL in the path instead of query, eg https://fed.brid.gy/realize.be/reply/content/1567.)

one key question: i only see one reply of yours on the original mastodon post you were testing against, https://mastodon.social/users/swentel/statuses/100776214865184909 . mastodon shows its URL as https://fed.brid.gy/realize.be#create . do you know if you put that in both id and url? or just id? i'm wondering which one mastodon uses. if url, then maybe we wouldn't have to actual redirect or serve from our wrapped id URLs at all.

swentel commented 6 years ago

Right, I've deleted some replies. I added a new one. So mastodon either gets it from id or url, see https://github.com/tootsuite/mastodon/blob/master/app/lib/activitypub/activity/create.rb#L52

I've added another reply on https://mastodon.social/@swentel/100776214865184909 with "This is pretty cool!"

This is the payload from the request

stdClass Object
(
    [type] => Create
    [@context] => https://www.w3.org/ns/activitystreams
    [actor] => https://fed.brid.gy/realize.be
    [object] => Array
        (
            [type] => Note
            [to] => Array
                (
                    [0] => https://www.w3.org/ns/activitystreams#Public
                    [1] => https://mastodon.social/users/swentel/inbox
                )

            [id] => https://fed.brid.gy/realize.be?post=https://realize.be/notes/1573
            [inReplyTo] => https://mastodon.social/@swentel/100776214865184909
            [content] => This is pretty cool!
            [published] => 2018-10-11T07:08:01Z
        )

)
swentel commented 6 years ago

So I've looked where the rewrite could happen, but I'm not 100% sure exactly what I'm doing :/ I've attached a diff I have locally, but not sure if the place I'm changing it makes sense .. :)

rewrite.txt

snarfed commented 6 years ago

thanks!

i'm still curious, if we point id to fed.brid.gy but keep url pointing to the user's own site, whether mastodon will render links from url or id. i noticed you omitted url in https://github.com/snarfed/bridgy-fed/issues/16#issuecomment-424803380 . mind trying again with url pointing somewhere other than id to see which one mastodon renders?

and your patch looks good! i'd probably put it in common.postprocess_as2() instead, and i'm inclined to put the full URL in the path like the existing/wm/... URL route, eg https://fed.brid.gy/redir/https://realize.be/ and https://fed.brid.gy/redir/https://realize.be/reply/content/1567. but you're definitely on the right path!

swentel commented 6 years ago

Url pointing to the post will be discarded, see https://github.com/tootsuite/mastodon/blob/master/app/lib/activitypub/activity/create.rb#L243 - it uses the same validation function (which still annoys me, but oh well)

Just to be sure that I'm not fooling myself, send this payload:

stdClass Object
(
    [type] => Create
    [@context] => https://www.w3.org/ns/activitystreams
    [actor] => https://fed.brid.gy/realize.be
    [object] => Array
        (
            [type] => Note
            [to] => Array
                (
                    [0] => https://www.w3.org/ns/activitystreams#Public
                    [1] => https://mastodon.social/users/swentel/inbox
                )

            [id] => https://fed.brid.gy/realize.be?post=https://realize.be/notes/1550
            [url] => https://realize.be/notes/1550
            [inReplyTo] => https://mastodon.social/@swentel/100776214865184909
            [content] => Sending different host in url. Will be discarded normally.
            [published] => 2018-10-11T21:55:10Z
        )

)

The url has been ignored here. Payload underneath has bridgy host in the url, and then it takes that one

stdClass Object
(
    [type] => Create
    [@context] => https://www.w3.org/ns/activitystreams
    [actor] => https://fed.brid.gy/realize.be
    [object] => Array
        (
            [type] => Note
            [to] => Array
                (
                    [0] => https://www.w3.org/ns/activitystreams#Public
                    [1] => https://mastodon.social/users/swentel/inbox
                )

            [id] => https://fed.brid.gy/realize.be?post=https://realize.be/notes/1530
            [url] => https://fed.brid.gy/realize.be?post=https://realize.be/notes/1545
            [inReplyTo] => https://mastodon.social/@swentel/100776214865184909
            [content] => Sending url with host same as actor, should take one now
            [published] => 2018-10-11T21:57:03Z
        )

)

You can see them both on https://mastodon.social/@swentel/100776214865184909

swentel commented 6 years ago

Tested, awesome, awesome :)

I guess we can close this one!

Or maybe remove the log message in bridgy, we know that 202 is ok it terms of signature.

snarfed commented 6 years ago

agreed! i've removed the log message. thanks for the nudge.