search

snarfed / bridgy-fed

🌉 A bridge between decentralized social network protocols
https://fed.brid.gy
Creative Commons Zero v1.0 Universal
556 stars 29 forks source link

translate following, both directions #21

Closed snarfed closed 5 years ago

snarfed commented 6 years ago

we're discussing in IRC right now how following an indieweb site in mastodon isn't yet supported or doesn't work well, and we don't have a good story for the reverse, ie indicating to mastodon that you're following a mastodon account.

i think the feature request is for bridgy fed to translate and deliver indieweb follow posts (ie u-x-follow-of) to ActivityPub Follow activities, and vice versa. (TODO: what's the OStatus equivalent?)

two difficulties:

cc @sknebel @keithjgrant @aaronpk @miklb

swentel commented 6 years ago

I wonder whether can somehow tell the instance that we don't care about following, by which I mean: just allow anyone to follow you. I figure that since it's your site on the fediverse, and anything on your site is public, it doesn't really matter having to confirm a follower.

Not sure which property we'd have to set then - and if possible at all.

It's probably the 'locked' property, and it's set to false, so yeah, bridgy probably needs to respond here. Did you see a request coming in by any chance on bridgy ? :)

snarfed commented 6 years ago

yes! here it is. bridgy fed responded with HTTP 501 Not Implemented.

POST https://fed.brid.gy/realize.be/inbox

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    "https://w3id.org/security/v1",
    {
      "manuallyApprovesFollowers": "as:manuallyApprovesFollowers",
      "sensitive": "as:sensitive",
      "movedTo": {
        "@id": "as:movedTo",
        "@type": "@id"
      },
      "Hashtag": "as:Hashtag",
      "ostatus": "http://ostatus.org#",
      "atomUri": "ostatus:atomUri",
      "inReplyToAtomUri": "ostatus:inReplyToAtomUri",
      "conversation": "ostatus:conversation",
      "toot": "http://joinmastodon.org/ns#",
      "Emoji": "toot:Emoji",
      "focalPoint": {
        "@container": "@list",
        "@id": "toot:focalPoint"
      },
      "featured": {
        "@id": "toot:featured",
        "@type": "@id"
      },
      "schema": "http://schema.org#",
      "PropertyValue": "schema:PropertyValue",
      "value": "schema:value"
    }
  ],
  "id": "https://mastodon.social/6d1af0b9-ef6a-46b0-b662-f79b21d7c983",
  "type": "Follow",
  "actor": "https://mastodon.social/users/swentel",
  "object": "https://fed.brid.gy/realize.be",
  "signature": {
    "type": "RsaSignature2017",
    "creator": "https://mastodon.social/users/swentel#main-key",
    "created": "2018-09-20T17:10:37Z",
    "signatureValue": "[REDACTED]"
  }
}
snarfed commented 6 years ago

it'd probably actually be a pretty simple change to just reply to all AP Follows with AP Accept. feel free to try!

swentel commented 6 years ago

So, I've gotten to the point where I could actually send a request, but the response returned 'Request not signed'.

I've attached the WIP patch. The source_domain is hard coded right now. As I was testing with the output from above, I guess it can never work since the signatureValue is not available there - unless I read the spec wrong for Accept and maybe the signature is not needed anyway there.

sleeping time now, tips appreciated :)

18-wip.txt

snarfed commented 6 years ago

thanks again for working on this! you're right, mastodon requires all inbox POSTs to include a signature: https://github.com/tootsuite/mastodon/issues/4906#issuecomment-328844846

if you want to test with your existing bridgy fed based realize.be account, i'm happy to send you your private key, via IRC DM or however.

alternatively, you could make your own bridgy fed app engine app, deploy there, and switch your site to redirect .well-known/* to it instead, and test with it.

or, if you're ambitious, you could do what i did to test bridgy fed: run a website on localhost, and also run a mastodon instance on localhost, and test between them! i can send you lots of detailed raw notes from when i did this; they may or may not help. 😬

swentel commented 6 years ago

If you could send me the private key that would be great. I want to try and get the existing 'Follow' request confirmed, so it be handier to use the realize.be account. You can send it to swentel @ gmail.

Silly question then maybe, where do I put it exactly ? :)

Edit: I see the new generated key in datastore viewer, so I should probably import it there ..

swentel commented 6 years ago

Thanks for the mail! Didn't resolve anything yet though. When checking the mastodon source, it looks for the 'Signature' in the headers. Which makes me wonder why it works in send_webmentions (like and repost work the same I guess). I copied the code more or less from there. So a bit clueless now how to proceed further to be honest :/

swentel commented 6 years ago

So, because I'm better at PHP, I exported my private key and than started hacking around, and guess what. I got it working :) It took me a while to create the proper object and signature, but now https://mastodon.social/users/swentel/following is following @realize@realize.be (if you would search for swentel, you'd also see that the bridgy user knows it has one follower, also cool).

So, this is how the headers looked like for the accept request:

Array
(
    [Content-Type] => application/activity+json
    [date] => Sat, 22 Sep 2018 20:45:01 GMT
    [signature] => keyId="https://fed.brid.gy/realize.be#main-key",algorithm="rsa-sha256",headers="date",signature="REDACTED"
)

This is how the accept object looked liked that made it work eventually. It took a couple of iterations as I was getting 202 back but nothing happened until the right format - rings a bell right ;-) (I will experiment with the reply thing this weekend to see if I can make it work too by experimenting with the format) The id is a uuid. Other than that, format is really straightforward

(
    [type] => Accept
    [id] => d0ca0cd7-7115-4003-9ef8-18b584fc70e9
    [@context] => https://www.w3.org/ns/activitystreams
    [actor] => https://fed.brid.gy/realize.be
    [object] => stdClass Object
        (
            [type] => Follow
            [actor] => https://mastodon.social/users/swentel
            [object] => https://fed.brid.gy/realize.be
        )

)

Note, this also returns a 202, which is kind of annoying, but I did it twice now. I unfollowed, then did a follow request again, ran the command from my local machine and the accept was ok.

Also, this is how a follow activity looks like, which even easier.

(
    [type] => Follow
    [id] => d0ca0cd7-7115-4003-9ef8-18b584fc70e9
    [@context] => https://www.w3.org/ns/activitystreams
    [actor] => https://fed.brid.gy/realize.be
    [object] => https://mastodon.social/users/swentel
)
swentel commented 6 years ago

So giving that information, I think I can get the object formatted ok in python, not sure about the signature though, but I will give it a shot.

snarfed commented 5 years ago

hey, nice progress! thanks for the detailed investigation!

you may be right about the HTTP Signature header. bridgy fed uses the httpsig library, which says in its docs:

Known Limitations ; ...
Draft 2 added support for the Signature header. As this was principally designed to be an authentication helper, that header is not currently supported. PRs welcome.

presumably right now it uses the Authorization header instead. maybe this is why bridgy fed's other AP requests to Mastodon now often 202 and fail too? I'll add it to my list to try!

snarfed commented 5 years ago

ahh my memory is awful. i discussed this with that library's author a while ago: https://github.com/ahknight/httpsig/issues/9

swentel commented 5 years ago

presumably right now it uses the Authorization header instead. maybe this is why bridgy fed's other AP requests to Mastodon now often 202 and fail too

Hard to say. I've been testing with sending a AP reply now using the Signature header and getting 202 back all the time as well. If I change one character in the signature, the authorization fails, so it means we're getting through, but we don't have a clear idea what happens next then. I'm digging through the mastodon code, because the 202 is not helpful at all :/

swentel commented 5 years ago

With replies now working, the party can start for real now :)

Getting the 'accept' response in might be crucial to start growing an audience, I'll see if I can update my patch for that part!

swentel commented 5 years ago

So think this should actually be workable .. :) The request is to mastodon, but signature fails locally.

Also, the way I get the source_domain is absolutely horrible, but it's a bit to late right now.

diff --git a/activitypub.py b/activitypub.py
index 871208a..9d8a357 100644
--- a/activitypub.py
+++ b/activitypub.py
@@ -1,7 +1,9 @@
 """Handles requests for ActivityPub endpoints: actors, inbox, etc.
 """
+import datetime
 import json
 import logging
+import string

 import appengine_config

@@ -12,13 +14,15 @@ from oauth_dropins.webutil import util
 import webapp2

 import common
-from models import MagicKey
+from models import MagicKey, Response
+from httpsig.requests_auth import HTTPSignatureAuth

 SUPPORTED_TYPES = (
     'Announce',
     'Article',
     'Audio',
     'Create',
+    'Follow',
     'Image',
     'Like',
     'Note',
@@ -83,6 +87,10 @@ class InboxHandler(webapp2.RequestHandler):

         # TODO: verify signature if there is one

+        if type == 'Follow':
+            accept_follow(activity)
+            return
+
         # fetch actor if necessary so we have name, profile photo, etc
         if type in ('Like', 'Announce'):
             for elem in obj, activity:
@@ -95,6 +103,46 @@ class InboxHandler(webapp2.RequestHandler):
         common.send_webmentions(self, as1, protocol='activitypub',
                                 source_as2=json.dumps(activity))

+def accept_follow(activity):
+    logging.info('Sending Accept to inbox')
+
+    activity_accept = {
+        '@context': 'https://www.w3.org/ns/activitystreams',
+        'id': activity['id'],
+        'type': 'Accept',
+        'actor': activity['object'],
+        'object': {
+          'type': 'Follow',
+           'actor': activity['actor'],
+           'object': activity['object'],
+        }
+    }
+
+    # source domain - this is still wrong in so many ways ... :)
+    source_domain = string.replace(activity['object'], 'https://fed.brid.gy/r/', '')
+    source_domain = string.replace(source_domain, 'http://', '')
+    source_domain = string.replace(source_domain, 'https://', '')
+    logging.info('source domain ' + source_domain)
+
+    # inbox url.
+    target = activity['actor']
+    actor = common.get_as2(target).json()
+    inbox_url = actor.get('inbox')
+    logging.info('Inbox url ' + inbox_url)
+
+    acct = 'acct:%s@%s' % (source_domain, source_domain)
+    key = MagicKey.get_or_create(source_domain)
+    signature = HTTPSignatureAuth(secret=key.private_pem(), key_id=acct,
+                             algorithm='rsa-sha256')
+
+    # deliver source object to target actor's inbox.
+    headers = {
+        'Content-Type': common.CONTENT_TYPE_AS2,
+        # required for HTTP Signature
+        # https://tools.ietf.org/html/draft-cavage-http-signatures-07#section-2.1.3
+        'Date': datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT'),
+        'signature': signature,
+    }
+    resp = common.requests_post(inbox_url, json=activity_accept, headers=headers)
+    self.response.write(resp.text)
+
snarfed commented 5 years ago

ooh exciting, this is a great start! i'll merge and update this.

the next step is to translate it to an mf2 "follow" post and send a webmention. based on https://indieweb.org/follow , i think it'd be an h-entry with two key parts:

we'd store this in a Response object (see models.py), call its proxy_url() method to get the webmention source url, use the indieweb user's home page as the target url, and send that webmention.

snarfed commented 5 years ago

ok! done. following should work now. we respond with an Accept, and we also store the follower in the datastore, so we can maybe eventually deliver new posts to them, as in #33.

next i'll work on translating AP Follows to u-follow-of indieweb posts.

snarfed commented 5 years ago

note to myself: in AS1 this is the follow verb with the followee as the object.

swentel commented 5 years ago

Since followers are stored, we should probably also check an 'Unfollow' is send to the inbox too, so that the follower can be removed.

Also, followee, I love that word :)

snarfed commented 5 years ago

yup! looks like it's actually an Undo. here's an example from mastodon:

{
  "@context": "...",
  "id": "https://mastodon.technology/users/snarfed#follows/145220/undo",
  "type": "Undo",
  "actor": "https://mastodon.technology/users/snarfed",
  "object": {
    "id": "https://mastodon.technology/3e362975-8468-4f71-9d13-c4cd86b1547f",
    "type": "Follow",
    "actor": "https://mastodon.technology/users/snarfed",
    "object": "https://fed.brid.gy/snarfed.org"
  },
  "..."
}
snarfed commented 5 years ago

ok! we now translate AP Follows to indie u-follow-of webmentions. https://indieweb.org/follow

example rendered follow on my site: https://snarfed.org/about#comment-2615238

image

source mf2 html:

<article class="h-entry">
  <span class="p-uid">https://mastodon.technology/1ce118f7-a038-422a-b322-f190e184fc3e</span>
  <span class="p-author h-card">
    <data class="p-uid" value="https://mastodon.technology/users/snarfed"></data>
    <a class="p-name u-url" href="https://mastodon.technology/@snarfed">Ryan Barrett</a>
    <img class="u-photo" src="https://static.mastodon.technology/accounts/avatars/000/023/507/original/01401e3558e03feb.png" alt="" />
  </span>
  <a class="u-follow-of" href="https://snarfed.org/"></a>
</article>

which parses to:

{
  "type": ["h-entry"],
  "properties": {
    "uid": ["https://mastodon.technology/1ce118f7-a038-422a-b322-f190e184fc3e"],
    "follow-of": ["https://snarfed.org/"],
    "author": [
      {
        "type": ["h-card"],
        "properties": {
          "uid": ["https://mastodon.technology/users/snarfed"],
          "name": ["Ryan Barrett"],
          "url": ["https://mastodon.technology/@snarfed"],
          "photo": ["https://static.mastodon.technology/accounts/avatars/000/023/507/original/01401e3558e03feb.png"]
        },
        "value": "Ryan Barrett"
      }
    ]
  }
}

next TODOs:

snarfed commented 5 years ago

ok, u-follow-of webmention => AP Follow now works too, as of 2bb418dc72589c2bcdf0488fe45df5c1b8526011. woo! closing this issue.