search

friendica / friendica

Friendica Communications Platform
https://friendi.ca
GNU Affero General Public License v3.0
1.41k stars 332 forks source link

ActivityPub: Add support for attachment property #8676

Closed MrPetovan closed 4 years ago

MrPetovan commented 4 years ago

Link preview "type":"Link"

Original post: https://mammoth.home.koehn.com/api/actor/bkoehn/513a3fc1-4f9b-4168-a341-05e4a1024562 Friendica display: https://friendica.mrpetovan.com/display/9f103a6a-87ddc007c37e6b52-f00b8b4f Source:

{
   "type":"Note",
   "content":"<p>Doctor Seuss, reimagined for 2020.</p>",
   "attributedTo":"https://mammoth.home.koehn.com/api/actor/bkoehn",
   "attachment":[
      {
         "type":"Link",
         "href":"https://www.instagram.com/p/B_44xIhlraQ/"
      }
   ],
   "to":[
      "https://www.w3.org/ns/activitystreams#Public",
      "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
   ],
   "id":"https://mammoth.home.koehn.com/api/actor/bkoehn/513a3fc1-4f9b-4168-a341-05e4a1024562",
   "published":"2020-05-19T20:57:37.457Z",
   "updated":"2020-05-19T20:57:37.457Z",
   "likes":"https://mammoth.home.koehn.com/api/actor/bkoehn/513a3fc1-4f9b-4168-a341-05e4a1024562/likes",
   "replies":"https://mammoth.home.koehn.com/api/actor/bkoehn/513a3fc1-4f9b-4168-a341-05e4a1024562/replies",
   "dislikes":"https://mammoth.home.koehn.com/api/actor/bkoehn/513a3fc1-4f9b-4168-a341-05e4a1024562/dislikes",
   "shares":"https://mammoth.home.koehn.com/api/actor/bkoehn/513a3fc1-4f9b-4168-a341-05e4a1024562/shares",
   "@context":"https://www.w3.org/ns/activitystreams"
}

Link preview "type":"Page"

Original post: https://mammoth.home.koehn.com/api/actor/bkoehn/2958385b-7073-4380-b9de-06ccedc2d3f2 Friendica display: https://friendica.mrpetovan.com/display/9f103a6a-1d9de2d852df4922-7349f4a5 Source:

{
   "type":"Note",
   "content":"<p>That's more like it.</p>",
   "attributedTo":"https://mammoth.home.koehn.com/api/actor/bkoehn",
   "attachment":[
      {
         "id":"https://www.youtube.com/watch?v=Lp-Xqj8wSMg",
         "type":"Page",
         "url":[
            {
               "type":"Link",
               "href":"https://www.youtube.com/watch?v=Lp-Xqj8wSMg",
               "mediaType":"text/html; charset=utf-8"
            },
            {
               "type":"Link",
               "href":"https://i.ytimg.com/vi/Lp-Xqj8wSMg/maxresdefault.jpg",
               "mediaType":"image/jpeg"
            }
         ],
         "name":"The Origin of Opera and the Future of Programming by Jessica Kerr",
         "mediaType":"text/html; charset=utf-8"
      }
   ],
   "to":[
      "https://www.w3.org/ns/activitystreams#Public",
      "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
   ],
   "id":"https://mammoth.home.koehn.com/api/actor/bkoehn/2958385b-7073-4380-b9de-06ccedc2d3f2",
   "published":"2020-05-20T21:00:07.446Z",
   "updated":"2020-05-20T21:00:07.446Z",
   "likes":"https://mammoth.home.koehn.com/api/actor/bkoehn/2958385b-7073-4380-b9de-06ccedc2d3f2/likes",
   "replies":"https://mammoth.home.koehn.com/api/actor/bkoehn/2958385b-7073-4380-b9de-06ccedc2d3f2/replies",
   "dislikes":"https://mammoth.home.koehn.com/api/actor/bkoehn/2958385b-7073-4380-b9de-06ccedc2d3f2/dislikes",
   "shares":"https://mammoth.home.koehn.com/api/actor/bkoehn/2958385b-7073-4380-b9de-06ccedc2d3f2/shares",
   "@context":"https://www.w3.org/ns/activitystreams"
}

Video "mediaType":"video/quicktime"

Original post: https://mammoth.home.koehn.com/api/actor/bkoehn/89e54ca8-dea7-4559-a1a0-341fd99182d3 Friendica display: https://friendica.mrpetovan.com/display/9f103a6a-ae1e7503333d0d57-6d5102c0 Source:

{
   "type":"Note",
   "content":"<p>Old photo carousel.</p>",
   "attributedTo":"https://mammoth.home.koehn.com/api/actor/bkoehn",
   "attachment":[
      {
         "type":"Link",
         "mediaType":"video/quicktime",
         "href":"https://s3.home.koehn.com/mammoth/Screen%20Recording%202020-04-16%20at%208.49.17%20AM.mov"
      }
   ],
   "to":[
      "https://www.w3.org/ns/activitystreams#Public",
      "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
   ],
   "id":"https://mammoth.home.koehn.com/api/actor/bkoehn/89e54ca8-dea7-4559-a1a0-341fd99182d3",
   "published":"2020-05-21T16:42:44.678Z",
   "updated":"2020-05-21T16:42:44.678Z",
   "likes":"https://mammoth.home.koehn.com/api/actor/bkoehn/89e54ca8-dea7-4559-a1a0-341fd99182d3/likes",
   "replies":"https://mammoth.home.koehn.com/api/actor/bkoehn/89e54ca8-dea7-4559-a1a0-341fd99182d3/replies",
   "dislikes":"https://mammoth.home.koehn.com/api/actor/bkoehn/89e54ca8-dea7-4559-a1a0-341fd99182d3/dislikes",
   "shares":"https://mammoth.home.koehn.com/api/actor/bkoehn/89e54ca8-dea7-4559-a1a0-341fd99182d3/shares",
   "@context":"https://www.w3.org/ns/activitystreams"
}

Video "mediaType":"video/mp4"

Original post: https://mammoth.home.koehn.com/api/actor/bkoehn/b5bdd2c5-7034-4bf3-afb7-96f2f7afbede Friendica display: https://friendica.mrpetovan.com/display/9f103a6a-140ee05504ed93ae-072fa098 Source:

{
   "type":"Note",
   "content":"<p>New carousel.</p>",
   "attributedTo":"https://mammoth.home.koehn.com/api/actor/bkoehn",
   "attachment":[
      {
         "type":"Link",
         "mediaType":"video/mp4",
         "href":"https://s3.home.koehn.com/mammoth/new-carousel.mp4"
      }
   ],
   "to":[
      "https://www.w3.org/ns/activitystreams#Public",
      "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
   ],
   "id":"https://mammoth.home.koehn.com/api/actor/bkoehn/b5bdd2c5-7034-4bf3-afb7-96f2f7afbede",
   "published":"2020-05-21T16:58:07.180Z",
   "updated":"2020-05-21T16:59:22.801Z",
   "likes":"https://mammoth.home.koehn.com/api/actor/bkoehn/b5bdd2c5-7034-4bf3-afb7-96f2f7afbede/likes",
   "replies":"https://mammoth.home.koehn.com/api/actor/bkoehn/b5bdd2c5-7034-4bf3-afb7-96f2f7afbede/replies",
   "dislikes":"https://mammoth.home.koehn.com/api/actor/bkoehn/b5bdd2c5-7034-4bf3-afb7-96f2f7afbede/dislikes",
   "shares":"https://mammoth.home.koehn.com/api/actor/bkoehn/b5bdd2c5-7034-4bf3-afb7-96f2f7afbede/shares",
   "@context":"https://www.w3.org/ns/activitystreams"
}
koehn commented 4 years ago

And then there's PeerTube (don't forget to send View activities!): Original Post: https://peertube.koehn.com/videos/watch/60da8014-7074-4505-9904-b81a56443a60

{
  "type": "Video",
  "id": "https://peertube.koehn.com/videos/watch/60da8014-7074-4505-9904-b81a56443a60",
  "name": "Flight into Cabin Creek, Montana",
  "duration": "PT81S",
  "uuid": "60da8014-7074-4505-9904-b81a56443a60",
  "tag": [
    {
      "type": "Hashtag",
      "name": "97MT"
    },
    {
      "type": "Hashtag",
      "name": "GA"
    },
    {
      "type": "Hashtag",
      "name": "aviation"
    },
    {
      "type": "Hashtag",
      "name": "montana"
    }
  ],
  "category": {
    "identifier": "6",
    "name": "Travels"
  },
  "licence": {
    "identifier": "5",
    "name": "Attribution - Non Commercial - Share Alike"
  },
  "views": 66,
  "sensitive": false,
  "waitTranscoding": false,
  "state": 1,
  "commentsEnabled": true,
  "downloadEnabled": true,
  "published": "2018-03-21T17:27:36.460Z",
  "originallyPublishedAt": null,
  "updated": "2020-05-20T11:01:00.164Z",
  "mediaType": "text/markdown",
  "content": "This was taken landing in Cabin Creek, Montana (97MT). The trees on the ridge at the approach end have grown too tall, necessitating a close-in approach.\r\n\r\n",
  "support": null,
  "subtitleLanguage": [],
  "icon": {
    "type": "Image",
    "url": "https://peertube.koehn.com/static/thumbnails/60da8014-7074-4505-9904-b81a56443a60.jpg",
    "mediaType": "image/jpeg",
    "width": 200,
    "height": 110
  },
  "url": [
    {
      "type": "Link",
      "mediaType": "text/html",
      "href": "https://peertube.koehn.com/videos/watch/60da8014-7074-4505-9904-b81a56443a60"
    },
    {
      "type": "Link",
      "mediaType": "video/mp4",
      "href": "https://peertube.koehn.com/static/webseed/60da8014-7074-4505-9904-b81a56443a60-1080.mp4",
      "height": 1080,
      "size": 61791678,
      "fps": -1
    },
    {
      "type": "Link",
      "mediaType": "application/x-bittorrent",
      "href": "https://peertube.koehn.com/static/torrents/60da8014-7074-4505-9904-b81a56443a60-1080.torrent",
      "height": 1080
    },
    {
      "type": "Link",
      "mediaType": "application/x-bittorrent;x-scheme-handler/magnet",
      "href": "magnet:?xs=https%3A%2F%2Fpeertube.koehn.com%2Fstatic%2Ftorrents%2F60da8014-7074-4505-9904-b81a56443a60-1080.torrent&xt=urn:btih:1453fae19decfb98f37322f839e0dff9b2ed85a4&dn=Flight+into+Cabin+Creek%2C+Montana&tr=wss%3A%2F%2Fpeertube.koehn.com%3A443%2Ftracker%2Fsocket&tr=https%3A%2F%2Fpeertube.koehn.com%2Ftracker%2Fannounce&ws=https%3A%2F%2Fpeertube.koehn.com%2Fstatic%2Fwebseed%2F60da8014-7074-4505-9904-b81a56443a60-1080.mp4",
      "height": 1080
    },
    {
      "type": "Link",
      "mediaType": "video/mp4",
      "href": "https://peertube.koehn.com/static/webseed/60da8014-7074-4505-9904-b81a56443a60-240.mp4",
      "height": 240,
      "size": 5397282,
      "fps": -1
    },
    {
      "type": "Link",
      "mediaType": "application/x-bittorrent",
      "href": "https://peertube.koehn.com/static/torrents/60da8014-7074-4505-9904-b81a56443a60-240.torrent",
      "height": 240
    },
    {
      "type": "Link",
      "mediaType": "application/x-bittorrent;x-scheme-handler/magnet",
      "href": "magnet:?xs=https%3A%2F%2Fpeertube.koehn.com%2Fstatic%2Ftorrents%2F60da8014-7074-4505-9904-b81a56443a60-240.torrent&xt=urn:btih:affea6e75f256fbab2a02faac6cbe9209533473d&dn=Flight+into+Cabin+Creek%2C+Montana&tr=wss%3A%2F%2Fpeertube.koehn.com%3A443%2Ftracker%2Fsocket&tr=https%3A%2F%2Fpeertube.koehn.com%2Ftracker%2Fannounce&ws=https%3A%2F%2Fpeertube.koehn.com%2Fstatic%2Fwebseed%2F60da8014-7074-4505-9904-b81a56443a60-240.mp4",
      "height": 240
    },
    {
      "type": "Link",
      "mediaType": "video/mp4",
      "href": "https://peertube.koehn.com/static/webseed/60da8014-7074-4505-9904-b81a56443a60-360.mp4",
      "height": 360,
      "size": 9568894,
      "fps": -1
    },
    {
      "type": "Link",
      "mediaType": "application/x-bittorrent",
      "href": "https://peertube.koehn.com/static/torrents/60da8014-7074-4505-9904-b81a56443a60-360.torrent",
      "height": 360
    },
    {
      "type": "Link",
      "mediaType": "application/x-bittorrent;x-scheme-handler/magnet",
      "href": "magnet:?xs=https%3A%2F%2Fpeertube.koehn.com%2Fstatic%2Ftorrents%2F60da8014-7074-4505-9904-b81a56443a60-360.torrent&xt=urn:btih:051d31f327b97b22ef44730bc3e5aab60217a63e&dn=Flight+into+Cabin+Creek%2C+Montana&tr=wss%3A%2F%2Fpeertube.koehn.com%3A443%2Ftracker%2Fsocket&tr=https%3A%2F%2Fpeertube.koehn.com%2Ftracker%2Fannounce&ws=https%3A%2F%2Fpeertube.koehn.com%2Fstatic%2Fwebseed%2F60da8014-7074-4505-9904-b81a56443a60-360.mp4",
      "height": 360
    },
    {
      "type": "Link",
      "mediaType": "video/mp4",
      "href": "https://peertube.koehn.com/static/webseed/60da8014-7074-4505-9904-b81a56443a60-480.mp4",
      "height": 480,
      "size": 14855710,
      "fps": -1
    },
    {
      "type": "Link",
      "mediaType": "application/x-bittorrent",
      "href": "https://peertube.koehn.com/static/torrents/60da8014-7074-4505-9904-b81a56443a60-480.torrent",
      "height": 480
    },
    {
      "type": "Link",
      "mediaType": "application/x-bittorrent;x-scheme-handler/magnet",
      "href": "magnet:?xs=https%3A%2F%2Fpeertube.koehn.com%2Fstatic%2Ftorrents%2F60da8014-7074-4505-9904-b81a56443a60-480.torrent&xt=urn:btih:f2a37043a077dda753f53664e1f8cf607b6e0285&dn=Flight+into+Cabin+Creek%2C+Montana&tr=wss%3A%2F%2Fpeertube.koehn.com%3A443%2Ftracker%2Fsocket&tr=https%3A%2F%2Fpeertube.koehn.com%2Ftracker%2Fannounce&ws=https%3A%2F%2Fpeertube.koehn.com%2Fstatic%2Fwebseed%2F60da8014-7074-4505-9904-b81a56443a60-480.mp4",
      "height": 480
    },
    {
      "type": "Link",
      "mediaType": "video/mp4",
      "href": "https://peertube.koehn.com/static/webseed/60da8014-7074-4505-9904-b81a56443a60-720.mp4",
      "height": 720,
      "size": 28518492,
      "fps": -1
    },
    {
      "type": "Link",
      "mediaType": "application/x-bittorrent",
      "href": "https://peertube.koehn.com/static/torrents/60da8014-7074-4505-9904-b81a56443a60-720.torrent",
      "height": 720
    },
    {
      "type": "Link",
      "mediaType": "application/x-bittorrent;x-scheme-handler/magnet",
      "href": "magnet:?xs=https%3A%2F%2Fpeertube.koehn.com%2Fstatic%2Ftorrents%2F60da8014-7074-4505-9904-b81a56443a60-720.torrent&xt=urn:btih:abd541c3ecc02ed1dd18626c5e51fa54f437182b&dn=Flight+into+Cabin+Creek%2C+Montana&tr=wss%3A%2F%2Fpeertube.koehn.com%3A443%2Ftracker%2Fsocket&tr=https%3A%2F%2Fpeertube.koehn.com%2Ftracker%2Fannounce&ws=https%3A%2F%2Fpeertube.koehn.com%2Fstatic%2Fwebseed%2F60da8014-7074-4505-9904-b81a56443a60-720.mp4",
      "height": 720
    }
  ],
  "likes": "https://peertube.koehn.com/videos/watch/60da8014-7074-4505-9904-b81a56443a60/likes",
  "dislikes": "https://peertube.koehn.com/videos/watch/60da8014-7074-4505-9904-b81a56443a60/dislikes",
  "shares": "https://peertube.koehn.com/videos/watch/60da8014-7074-4505-9904-b81a56443a60/announces",
  "comments": "https://peertube.koehn.com/videos/watch/60da8014-7074-4505-9904-b81a56443a60/comments",
  "attributedTo": [
    {
      "type": "Person",
      "id": "https://peertube.koehn.com/accounts/bkoehn"
    },
    {
      "type": "Group",
      "id": "https://peertube.koehn.com/video-channels/29e4751f-b15f-4334-b323-2fa5b353b09a"
    }
  ],
  "to": [
    "https://www.w3.org/ns/activitystreams#Public"
  ],
  "cc": [
    "https://peertube.koehn.com/accounts/bkoehn/followers"
  ],
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    "https://w3id.org/security/v1",
    {
      "RsaSignature2017": "https://w3id.org/security#RsaSignature2017",
      "pt": "https://joinpeertube.org/ns#",
      "sc": "http://schema.org#",
      "Hashtag": "as:Hashtag",
      "uuid": "sc:identifier",
      "category": "sc:category",
      "licence": "sc:license",
      "subtitleLanguage": "sc:subtitleLanguage",
      "sensitive": "as:sensitive",
      "language": "sc:inLanguage",
      "expires": "sc:expires",
      "CacheFile": "pt:CacheFile",
      "Infohash": "pt:Infohash",
      "originallyPublishedAt": "sc:datePublished",
      "views": {
        "@type": "sc:Number",
        "@id": "pt:views"
      },
      "state": {
        "@type": "sc:Number",
        "@id": "pt:state"
      },
      "size": {
        "@type": "sc:Number",
        "@id": "pt:size"
      },
      "fps": {
        "@type": "sc:Number",
        "@id": "pt:fps"
      },
      "startTimestamp": {
        "@type": "sc:Number",
        "@id": "pt:startTimestamp"
      },
      "stopTimestamp": {
        "@type": "sc:Number",
        "@id": "pt:stopTimestamp"
      },
      "position": {
        "@type": "sc:Number",
        "@id": "pt:position"
      },
      "commentsEnabled": {
        "@type": "sc:Boolean",
        "@id": "pt:commentsEnabled"
      },
      "downloadEnabled": {
        "@type": "sc:Boolean",
        "@id": "pt:downloadEnabled"
      },
      "waitTranscoding": {
        "@type": "sc:Boolean",
        "@id": "pt:waitTranscoding"
      },
      "support": {
        "@type": "sc:Text",
        "@id": "pt:support"
      }
    },
    {
      "likes": {
        "@id": "as:likes",
        "@type": "@id"
      },
      "dislikes": {
        "@id": "as:dislikes",
        "@type": "@id"
      },
      "playlists": {
        "@id": "pt:playlists",
        "@type": "@id"
      },
      "shares": {
        "@id": "as:shares",
        "@type": "@id"
      },
      "comments": {
        "@id": "as:comments",
        "@type": "@id"
      }
    }
  ]
}
MrPetovan commented 4 years ago

Thanks for the example, however it is slightly a different case since it isn't a Note like we're mostly dealing with at Friendica. It definitely needs its own handler.

koehn commented 4 years ago

The PeerTube url selection criteria is a joy too. I ended up writing code that iterated the various URLs looking for one that was the minimum width while being ≥ the width that I needed, with a relevant mediatype.

MrPetovan commented 4 years ago

I can imagine.

annando commented 4 years ago

We already have code for handling video and audio attachments: https://github.com/friendica/friendica/blob/develop/src/Protocol/ActivityPub/Receiver.php#L953-L1011

MrPetovan commented 4 years ago

The original issue still stands.

annando commented 4 years ago

That's correct :-)

MrPetovan commented 4 years ago

Cursory research showed that I won't be able to test with just the object but with a full activity like these:

"type": "Page"

{
  "_id": "https://mammoth.home.koehn.com/api/actor/bkoehn/92d42d64-ea76-4319-a19c-ffddb9566131",
  "_rev": "1-7b588ee7382575001aa8c8993c4931cc",
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Create",
  "id": "https://mammoth.home.koehn.com/api/actor/bkoehn/92d42d64-ea76-4319-a19c-ffddb9566131",
  "actor": "https://mammoth.home.koehn.com/api/actor/bkoehn",
  "object": {
    "type": "Note",
    "content": "<p>“Instructions given by the military departments to our troops before the Normandy invasion reminded soldiers that ‘The Nazi slogan for destroying us … was “Divide and Conquer.” Our American answer is “In Union there is Strength.”’ We must summon that unity to surmount this crisis—confident that we are better than our politics.”</p>",
    "attributedTo": "https://mammoth.home.koehn.com/api/actor/bkoehn",
    "attachment": [
      {
        "id": "https://www.theatlantic.com/politics/archive/2020/06/james-mattis-denounces-trump-protests-militarization/612640/",
        "type": "Page",
        "url": [
          {
            "type": "Link",
            "href": "https://www.theatlantic.com/politics/archive/2020/06/james-mattis-denounces-trump-protests-militarization/612640/",
            "mediaType": "text/html; charset=utf-8"
          },
          {
            "type": "Link",
            "href": "https://cdn.theatlantic.com/thumbor/1wCGYRQTG3b0Eok6a8clcv-r8uM=/0x0:3478x1961/720x405/media/img/mt/2020/06/ccWEL_Goldberg_MattisSecondary/original.jpg",
            "mediaType": "image/jpeg"
          }
        ],
        "name": "James Mattis Denounces President Trump, Describes Him as a Threat to the Constitution",
        "mediaType": "text/html; charset=utf-8"
      }
    ],
    "to": [
      "https://www.w3.org/ns/activitystreams#Public",
      "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
    ],
    "id": "https://mammoth.home.koehn.com/api/actor/bkoehn/06729dee-5988-4a42-808e-486f5eb4a1a9",
    "published": "2020-06-03T22:05:17.956Z",
    "updated": "2020-06-03T22:05:17.956Z"
  },
  "published": "2020-06-03T22:05:17.956Z",
  "to": [
    "https://www.w3.org/ns/activitystreams#Public",
    "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
  ]
}

@koehn Are you still using the naked "type": "Link"?

MrPetovan commented 4 years ago

"type": "Link" and "mediaType": "image/png"

{
  "_id": "https://mammoth.home.koehn.com/api/actor/bkoehn/6040e456-d6a1-4829-80a5-ba6d5b5b9b15",
  "_rev": "1-f6245c17f36edf4a33d5085fe8d01be6",
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Create",
  "id": "https://mammoth.home.koehn.com/api/actor/bkoehn/6040e456-d6a1-4829-80a5-ba6d5b5b9b15",
  "actor": "https://mammoth.home.koehn.com/api/actor/bkoehn",
  "object": {
    "type": "Note",
    "content": "<p><em>Later I'm going to get a head start on the heat bath.</em></p>",
    "attributedTo": "https://mammoth.home.koehn.com/api/actor/bkoehn",
    "attachment": [
      {
        "type": "Link",
        "mediaType": "image/png",
        "href": "https://imgs.xkcd.com/comics/eventual_consistency.png"
      }
    ],
    "to": [
      "https://www.w3.org/ns/activitystreams#Public",
      "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
    ],
    "id": "https://mammoth.home.koehn.com/api/actor/bkoehn/d288ca8f-0d75-4541-ac67-752475c632e4",
    "published": "2020-06-03T20:51:50.583Z",
    "updated": "2020-06-03T20:51:50.583Z"
  },
  "published": "2020-06-03T20:51:50.583Z",
  "to": [
    "https://www.w3.org/ns/activitystreams#Public",
    "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
  ]
}
MrPetovan commented 4 years ago

I made structural changes to the ActivityPub protocol classes and I'm not sure I won't break anything important other than what I personally tested. However I don't have a robust test setup to receive all sorts of ActivityPub payloads. How should I go about it? @annando would you like to pull my branch and test it on your system?

annando commented 4 years ago

My "test" system is more or less a productive system with several users on it. Because of this I'm always very cautious :-)

MrPetovan commented 4 years ago

I guess I can only submit the changes that pertain to this issue, and not the debugging tool I created just for the occasion that required me to alter the structure of the protocol classes.

MrPetovan commented 4 years ago

All the cases listed in this issue have been handled by #8727, leaving this open for the next batch of unexpected objects I'm sure will come soon.

MrPetovan commented 4 years ago

While working on #8749, I realized that we don't strip the attachment link from the body when converting attachments from activitypub, is it desirable?

See result: https://friendica.mrpetovan.com/display/9f103a6a-18806aac7f6b4c80-70937270

annando commented 4 years ago

What do you mean with "strip the body from the attachment link"? Do you mean to remove the attachment link from the body?

MrPetovan commented 4 years ago

Yes.

annando commented 4 years ago

That's not that easy. When the link is the last entry, then it can be removed. But when it is in the middle of the body then it wouldn't look good when it had been removed.

MrPetovan commented 4 years ago

In this case it is at the end of the body before the attachment is appended.

MrPetovan commented 4 years ago

Pushing to the next release with this new gem of an activity:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Create",
  "id": "https://mammoth.home.koehn.com/api/actor/bkoehn/9d34aa06-2792-41a2-8ad5-a85fd9ed5293",
  "actor": "https://mammoth.home.koehn.com/api/actor/bkoehn",
  "object": {
    "type": "Note",
    "content": "<p>Yesterday we got some #backpacking in at a nearby state park, preparing for a multi-day hike on the Superior Hiking Trail.</p>",
    "attributedTo": "https://mammoth.home.koehn.com/api/actor/bkoehn",
    "attachment": [
      "https://s3.home.koehn.com/mammoth/8634c0b7-1ca3-4e11-bfae-5b8a79ae52db16BA5523-44F9-4F4F-A604-022DF525C437.jpeg",
      "https://s3.home.koehn.com/mammoth/fdc8a6ff-9c03-4f29-8aa2-e9b851022c10FA65F610-593F-43E3-A685-EA1DAD4BA718.jpeg",
      "https://s3.home.koehn.com/mammoth/c0bd92e6-4516-4447-ab80-decbaecf6d72DEC8BC7F-17A3-4486-A7A7-0DDC15189AF6.jpeg",
      "https://s3.home.koehn.com/mammoth/84c080af-e0e7-4fcf-879e-aafa293ea7e30C4FB1CA-940F-4A15-B9A7-92DA1A714DA7.jpeg",
      "https://s3.home.koehn.com/mammoth/5a67843d-ba65-4d32-98e8-ced3fc15f7b9A7C334CF-D7C1-4871-B993-753017B07C45.jpeg",
      "https://s3.home.koehn.com/mammoth/73aceb91-8832-40c3-a622-7ad047eb0c2b92F02BEE-5C0F-46B9-B2BF-2E9B909D49DE.jpeg",
      "https://s3.home.koehn.com/mammoth/83f0c76c-23d4-4c51-8a5d-8267a2ca2472C331C4E4-C9C9-4438-BC5B-79465D05788E.jpeg",
      "https://s3.home.koehn.com/mammoth/614c6f48-8f9b-432e-ada3-9bdea51d32b54B7A83D5-BE35-4AA4-840E-1DD60BEB998E.jpeg"
    ],
    "to": [
      "https://www.w3.org/ns/activitystreams#Public",
      "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
    ],
    "tag": [
      {
        "type": "Hashtag",
        "name": "#backpacking"
      }
    ],
    "id": "https://mammoth.home.koehn.com/api/actor/bkoehn/3a1d2cb1-aa24-45b0-b13d-2c982abd2c76",
    "published": "2020-06-22T23:03:32.113Z",
    "updated": "2020-06-22T23:03:32.113Z",
    "_id": "https://mammoth.home.koehn.com/api/actor/bkoehn/3a1d2cb1-aa24-45b0-b13d-2c982abd2c76"
  },
  "published": "2020-06-22T23:03:32.113Z",
  "to": [
    "https://www.w3.org/ns/activitystreams#Public",
    "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
  ],
  "_id": "https://mammoth.home.koehn.com/api/actor/bkoehn/9d34aa06-2792-41a2-8ad5-a85fd9ed5293"
}
koehn commented 4 years ago

FWIW that was a bug on my part; I'm pushing literally right now a fix that makes uploaded image attachments into Link objects.

https://gitlab.koehn.com/mammoth/mammoth-server/-/merge_requests/5

MrPetovan commented 4 years ago

Woot! It's worth a lot, thanks for the elaboration. Please re-send an updated message to check that we handle multiple attachments.

koehn commented 4 years ago

No prob.

FWIW, storing ActivityPub objects as JSON means that you don't need to design a schema and code that can handle every size/shape of document that the internet can throw at you. The Object above is valid, and your app should be able to handle it.

ActivityPub is ridiculously flexible in how it allows you to represent objects (as I'm sure you know), and it's only one of the formats Friendica needs to support. It's much harder to build a schema to store them all in some baseline format than it is to build a UI that understands the various formats and displays them. That way, when a new representation comes along that your UI doesn't know about, you don't lose data from the server not knowing how to interpret it all, you just tweak the UI and that data becomes consumable by the user. Your server and database then only need to know how documents relate to one another; it doesn't care about the rest.

This design paradigm has made it very easy for me to build a system that can inter-operate with most platforms, although you've seen I need to make some Issues with different providers to get them to tweak a few things. Still, when I stumble across a new way of representing data, it's almost always just the UI that needs tweaking; the back-end rarely needs to change at all.

Sorry for the long comment; just thought I'd share a design that has been working well for me.

One last thing: In a future release, the Link objects I use to represent uploaded images will include a url that is an array with multiple sizes of the image for display. Eventually they'll become Image or Video documents per the AP spec for uploading media.

koehn commented 4 years ago

Here's an object in the latest format:

https://mammoth.home.koehn.com/api/actor/alice/ba177fbb-ba22-45dc-bd76-8cbe8c7d8266

{
  "type": "Note",
  "content": "<p>test image post.</p>",
  "attributedTo": "https://mammoth.home.koehn.com/api/actor/alice",
  "attachment": [
    {
      "type": "Link",
      "href": "https://s3.home.koehn.com/mammoth/bc7e48c1-e0f5-45d8-ba1f-7173f7dac89bScreen%20Shot%202020-06-01%20at%204.09.18%20PM.png",
      "mediaType": "image/png"
    }
  ],
  "to": [
    "https://www.w3.org/ns/activitystreams#Public",
    "https://mammoth.home.koehn.com/api/actor/alice/followers"
  ],
  "id": "https://mammoth.home.koehn.com/api/actor/alice/ba177fbb-ba22-45dc-bd76-8cbe8c7d8266",
  "published": "2020-06-23T12:34:43.274Z",
  "updated": "2020-06-23T12:34:43.274Z",
  "likes": "https://mammoth.home.koehn.com/api/actor/alice/ba177fbb-ba22-45dc-bd76-8cbe8c7d8266/likes",
  "replies": "https://mammoth.home.koehn.com/api/actor/alice/ba177fbb-ba22-45dc-bd76-8cbe8c7d8266/replies",
  "dislikes": "https://mammoth.home.koehn.com/api/actor/alice/ba177fbb-ba22-45dc-bd76-8cbe8c7d8266/dislikes",
  "shares": "https://mammoth.home.koehn.com/api/actor/alice/ba177fbb-ba22-45dc-bd76-8cbe8c7d8266/shares",
  "@context": "https://www.w3.org/ns/activitystreams"
}
MrPetovan commented 4 years ago

Thank you for your words, Brad, I've been thinking about this lately: we could store the source message, no matter the protocol, and when we render the post, we would just convert directly from the source instead of having an intermediate BBCode body.

However, this step is required because not all protocols handle reshares the same way, so we can't always share the original item in our name. Sometimes we have to include the reshared post content as well, and it may be sent to various outgoing protocols, which prevents us to use the original source as is.

Our unique position as compatibility maximalists means that we need a common denominator for post content, and it's been BBCode tags. However, it does make sense for you to use the source object directly as you don't intend on implementing any further protocol than ActivityPub. It is a boon because of the simplicity, but it also means your software lives and dies with ActivityPub as well, which is a liability.

I like the idea that Friendica will always be somewhat relevant. We occupy a Fediverse/Federation niche that no other piece of software competes for, and while there certainly was some pressure to add support for ActivityPub for example, we still had OStatus and Diaspora support to rest on while the work was ongoing.

Thanks for the additional object source!

annando commented 4 years ago

When just storing the raw message, you increase the needed database size (by storing unneeded data) and you decrease the performance because you always have to render from the raw message. Also some stuff like searching for posts with some hash tags or posts from a given contact wouldn't be working as well.

And since we always have to do some rendering, this easily can be done by the preprocessor as well.

MrPetovan commented 4 years ago

Currently we store both the source and the intermediate BBCode body, so I'm not sure where is the storage gain. We also store the cached HTML, which we would also do with a source-only model. Tags and mentions would also be handled the same way we do now, this is just about pure rendering.

annando commented 4 years ago

We delete the source after some time (I guess 90 days).

MrPetovan commented 4 years ago

Another broken display item with the following source:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Create",
  "id": "https://mammoth.home.koehn.com/api/actor/bkoehn/5ecd16b5-9086-4729-b4ec-2cb386714992",
  "actor": "https://mammoth.home.koehn.com/api/actor/bkoehn",
  "object": {
    "type": "Note",
    "content": "<p>Enabling client-side image editing in my #ActivityPub app. I’ve never played with  before; it’s pretty easy to work with.</p>\n<p>https://s3.koehn.com/mammoth/image rotation.mp4</p>",
    "attributedTo": "https://mammoth.home.koehn.com/api/actor/bkoehn",
    "attachment": [
      ""
    ],
    "to": [
      "https://www.w3.org/ns/activitystreams#Public",
      "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
    ],
    "tag": [
      {
        "type": "Hashtag",
        "name": "#ActivityPub"
      }
    ],
    "id": "https://mammoth.home.koehn.com/api/actor/bkoehn/dea10686-92bb-446f-ad5a-39f4fbf74fca",
    "published": "2020-06-24T18:47:32.616Z",
    "updated": "2020-06-24T18:47:32.616Z",
    "_id": "https://mammoth.home.koehn.com/api/actor/bkoehn/dea10686-92bb-446f-ad5a-39f4fbf74fca"
  },
  "published": "2020-06-24T18:47:32.616Z",
  "to": [
    "https://www.w3.org/ns/activitystreams#Public",
    "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
  ],
  "_id": "https://mammoth.home.koehn.com/api/actor/bkoehn/5ecd16b5-9086-4729-b4ec-2cb386714992"
}

@koehn Any idea why the attachement is an empty string?

MrPetovan commented 4 years ago

Another banger: https://friendica.mrpetovan.com/display/9f103a6a-16f542fdc6c1a1ca-cc63e141

Source:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Create",
  "id": "https://mammoth.home.koehn.com/api/actor/bkoehn/4dec0d33-adef-44cf-b127-2359976216d1",
  "actor": "https://mammoth.home.koehn.com/api/actor/bkoehn",
  "object": {
    "type": "Note",
    "content": "<p>Got the new image uploader working on my ActivityPub project. Images can be rotated and are resized client-side (removing all EXIF information) prior to uploading them to the server. Also, a smaller and larger version of the image are uploaded and can be displayed by the UI.</p>",
    "attributedTo": "https://mammoth.home.koehn.com/api/actor/bkoehn",
    "attachment": [
      {
        "type": "Image",
        "mediaType": "image/jpeg",
        "url": [
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/d5d6b70d-1528-4556-a240-e5a6eabb11b2jpeg",
            "mediaType": "image/jpeg",
            "width": 600,
            "height": 450
          },
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/ebc61e64-e41b-4acb-b3fd-5e20ba0809f6jpeg",
            "mediaType": "image/jpeg",
            "width": 1024,
            "height": 768
          }
        ]
      },
      {
        "type": "Image",
        "mediaType": "image/jpeg",
        "url": [
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/b999876a-a4f7-4f8a-8e1a-80ba532611c0jpeg",
            "mediaType": "image/jpeg",
            "width": 600,
            "height": 800
          },
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/203555d7-c5bd-4b14-8e05-5b011e73a028jpeg",
            "mediaType": "image/jpeg",
            "width": 1024,
            "height": 1365
          }
        ]
      },
      {
        "type": "Image",
        "mediaType": "image/jpeg",
        "url": [
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/207ff671-23e7-47f1-b00a-71f3bd1291c8jpeg",
            "mediaType": "image/jpeg",
            "width": 600,
            "height": 800
          },
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/0ff76a3c-beca-436d-8679-dbc67ae53ef1jpeg",
            "mediaType": "image/jpeg",
            "width": 1024,
            "height": 1365
          }
        ]
      },
      {
        "type": "Image",
        "mediaType": "image/jpeg",
        "url": [
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/605295a0-0a31-4f8b-aebb-7bbae6464617jpeg",
            "mediaType": "image/jpeg",
            "width": 600,
            "height": 800
          },
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/a90a7641-5cf1-4162-9bd0-36133cb3e680jpeg",
            "mediaType": "image/jpeg",
            "width": 1024,
            "height": 1365
          }
        ]
      },
      {
        "type": "Image",
        "mediaType": "image/jpeg",
        "url": [
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/11de851f-4282-433a-9ca5-cc0b98e4677cjpeg",
            "mediaType": "image/jpeg",
            "width": 600,
            "height": 800
          },
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/3625e1f8-b25c-4150-af36-053bd1f43481jpeg",
            "mediaType": "image/jpeg",
            "width": 1024,
            "height": 1365
          }
        ]
      },
      {
        "type": "Image",
        "mediaType": "image/jpeg",
        "url": [
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/80d35af1-927d-4b7f-86ac-b590e9a3e698jpeg",
            "mediaType": "image/jpeg",
            "width": 600,
            "height": 185
          },
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/8b967855-c347-4e5d-946f-86f308259842jpeg",
            "mediaType": "image/jpeg",
            "width": 1024,
            "height": 316
          }
        ]
      },
      {
        "type": "Image",
        "mediaType": "image/jpeg",
        "url": [
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/e62effa5-69f4-4061-ba86-d997762903b5jpeg",
            "mediaType": "image/jpeg",
            "width": 600,
            "height": 800
          },
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/96721696-e1ee-45f0-9c7a-281471b14c88jpeg",
            "mediaType": "image/jpeg",
            "width": 1024,
            "height": 1365
          }
        ]
      },
      {
        "type": "Image",
        "mediaType": "image/jpeg",
        "url": [
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/5bd7684e-26d7-4f9b-ab46-fa1e28910241jpeg",
            "mediaType": "image/jpeg",
            "width": 600,
            "height": 450
          },
          {
            "type": "Link",
            "href": "https://s3.home.koehn.com/mammoth/470a11be-52c5-4d58-b68d-1b2fdeb44995jpeg",
            "mediaType": "image/jpeg",
            "width": 1024,
            "height": 768
          }
        ]
      }
    ],
    "to": [
      "https://www.w3.org/ns/activitystreams#Public",
      "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
    ],
    "id": "https://mammoth.home.koehn.com/api/actor/bkoehn/fd50fd0a-614d-4902-ba39-b49ad1ffec18",
    "published": "2020-06-26T20:13:55.696Z",
    "updated": "2020-06-26T20:13:55.696Z",
    "_id": "https://mammoth.home.koehn.com/api/actor/bkoehn/fd50fd0a-614d-4902-ba39-b49ad1ffec18"
  },
  "published": "2020-06-26T20:13:55.696Z",
  "to": [
    "https://www.w3.org/ns/activitystreams#Public",
    "https://mammoth.home.koehn.com/api/actor/bkoehn/followers"
  ],
  "_id": "https://mammoth.home.koehn.com/api/actor/bkoehn/4dec0d33-adef-44cf-b127-2359976216d1"
}
koehn commented 4 years ago

I was waiting for this one. 😉

This is the go-forward format I'll be using from here on out. Notes with attachments that are Images, with urls that are arrays of Links. In my (responsive) UI I use a loss-reduction function to determine which is the best Link to put into the HTML img src attribute for display in a post and then for display when enlarged.

MrPetovan commented 4 years ago

What's a loss-reduction function?

koehn commented 4 years ago

It’s an overly complicated term for a function that selects which image to display based on finding the closest match by size. Sorry for the confusion; I’ve been doing some machine learning work lately and it’s a term from that’s been on my mind.

MrPetovan commented 4 years ago

I haven't encountered any odd AP posts in the wild, closing for now.