Closed EdwardHinkle closed 1 year ago
hmm! i'm not sure, but i expect that's client side, not server side. which is a little confusing because mastodon is both for the other users, but not for you. bridgy fed can actually pass through some HTML from your posts, like linked text, and mastodon will accept and render it. i'll try to find an example.
didn't find an easy example at hand, but feel free to add arbitrary <a>
s to notes that you federate to mastodon, and it's possible they'll pass through.
Aha!! You’re right. Upon further inspection, the first post in the thread is from cyber.space and that hashtag links to its own server’s copy of the hashtag. (Cyber.space/tag/indieweb), the second post in the thread is from vicious.play and that hashtag links to its own server (vicious.play/tag/indieweb).
So to match them I would need to add my own hyperlink to eddiehinkle.com/tag/indieweb
So I think the real question is when I reply to a server (like cyber.space) how do I get my federated post to register with their server that I have a hashtag in my post
If you look at the cyber.space Mastodon it has the post from vicious.play even though it’s a remote post. (https://cybre.space/tags/indieweb)
So I think the real todo here (of which I can participate, not just asking you to do the work lol) is figuring out how remote Mastodon’s ping other servers when they have a hashtag
Aha! I see a post from aaronpk has been catalogued on that server with that hashtag, so I suspect he has the answer!
You need to add a new property into the object that describes the tag. For example:
"tag": [{
"type": "Hashtag",
"id": "https://aaronparecki.com/tag/indieweb",
"name": "#indieweb"
}]
@snarfed so the next question is how should Bridgy Fed handle this? Look for any hashtag in the text, or look for p-categories, or look for p-categories that also appear within the content with a #?
I guess the tricky thing is it is looking for an Id that is a url, so maybe you need a hashtag inside of an href, and then you use the href as the id string
hah, funny. from as2.py
:
'hashtag': 'Tag', # not in AS2 spec; needed for correct round trip conversion
(nit: hopefully the type: Hashtag
part isn't strictly necessary. the AS 2.0 vocab spec has a specific example section for hashtags that shows basically this usage, but without type
. the Hashtag
value itself is nowhere in either the vocab spec or the core spec.)
the easy answer here is that indieweb source posts can include arbitrary links, including linking hashtags to search pages or anything else on their own site, and bridgy fed will pass them through, and AP servers like Mastodon i think maybe render them as is.
if you want target Mastodon servers to render them as searches within their own instance, like you describe here, then yes, p-category
is probably best. i think bridgy fed should already translate p-category
to AS2 tag
, so this might already work, but i'm not 100% sure. feel free to try!
friendly ping @EdwardHinkle! feel free to try this with a test post with p-category
and let me know if it works. or if this has dropped off your radar, feel free to close.
It didn’t work with JUST a p-category. I still need to try BOTH a #tag AND p-category:tag
I’ll try out using both and see if that works
@EdwardHinkle interesting, i don't actually see your p-category attempt. no webmentions to fed.brid.gy/webmention
during 4:20pm-5:30pm PDT yesterday. did your site maybe not send a wm?
Oh, no I did that trial back in Feb. I just forgot to mention it here since I didn't get a chance to try the p-category with the hashtag in content
Hmm okay, I just tried this again. https://eddiehinkle.com/2019/05/16/1/note/
The hashtag “#rentlife” exists in the text, the p-category rentlife is on the h-entry. But when I check the tags for the instance I log into that follows my website, it doesn’t seem to show my post under /tag/rentlife
thanks for testing! here's the AS2 activity we sent to Mastodon, below, elided a bit. full object is in the log.
it has a tag
field with Tag
objects that seem ok based on the spec. one difference from the AS2 hashtag example is that the tags in the example both have id
s. guess i could try that next.
{
"@context": "https://www.w3.org/ns/activitystreams",
"type": "Create",
"object": {
"type": "Note",
"id": "https://fed.brid.gy/r/https://eddiehinkle.com/2019/05/16/1/note/",
"url": "https://fed.brid.gy/r/https://eddiehinkle.com/2019/05/16/1/note/",
"content": "Renting a house that is listed on the market for sale is ... not fun. Wife, baby, cat and I are all sitting in the car at the park... lol. #rentlife",
"tag": [{
"type": "Tag",
"name": "rentlife"
}, {
"type": "Tag",
"name": "family"
}, {
"type": "Tag",
"name": "personal"
}],
"..."
},
}
Ohhh yeah interesting. Ids are supposed to be URLs? That means if we put the hashtag in an anchor link, you could use the href as the id. If you decide to go that route let me know and I’ll test one with an anchor link
So I've noticed a peculiarity in the behaviour, here. So, here is an example of a create event where the hashtags are properly represented in Mastodon:
And here is a create event where the hashtags are not properly linked:
The first major distinction, to me, is the latter is a Note while the former as an Article, and we know Masto converts those a bit differently.
In those Notes, I noticed BF is converting the note content into a "summary" and a "name" and that's it. However, the Mastodon API docs indicate that might be causing a problem:
name - Used as status text, if content is not provided on a transformed Object type
And because BF is stripping the HTML on the name, Mastodon isn't getting the p-category microformat markup that's present in the original content.
My guess is that, like Articles, for Notes, BF should be sending a "content" element and not a "summary" element.
@fancypantalons great sleuthing, thank you! That sounds very likely. My logic for converting mf2 HTML to Mastodon was originally based on Twitter, which uses just summary/name for articles instead of full text due to character limits. Mastodon's are much looser though, eg this post from Bridgy Fed with 764 chars (only visible if you're logged into indieweb.social), so this change makes sense.
So doing a bit of poking around, it looks like this is really an issue in the granary mf2->as1 mapping implementation.
Should I open up an issue there?
Heck, I could probably throw together a PR if that's helpful (this issue is a nagging one for me as mastodon uses hashtags for discovery, which means my blog notes are basically invisible right now...)
PRs are always welcome!
Having said that, I'm not sure about the conclusion here just yet. Looking at your two examples above, if you scroll down to the bottom of the logs, the final AS2 activities that BF sent to Mastodon inboxes both have the tag
field populated. The note (log) has one for mbnov
, and the article (log) has one for books
.
Also, the article does have both content
(HTML) and summary
(text), and they both look ok to me.
So, I'm not yet sure that BF/granary are doing something obviously wrong here.
Damn. Yup, you are 100% correct, I misread the JSON... hmm... I shall keep digging!
So I have an alternative theory... but I'm not sure how to test it.
The article contains: name, summary, and content
For the article the summary is being used by Mastodon in the rendered toot. You can see that in the toot here.
I suspect that's because the summary is being overloaded by Mastodon for content warnings, and so it's preferentially using that.
The summary is stripped of HTML and ends up looking like this:
...blah blah blah... #books
The note, in contrast, contains: name, and content. No summary! So Mastodon is using the content attribute to populate the toot.
The content is not stripped of HTML, and it looks like this:
...blah blah blah... <span class=\"category\"> #<a class=\"p-category\" href=\"/archives/category/mbnov/\">mbnov</a> </span>
In the note content, the "#" sits outside the anchor tag. This is so that, when BF parses the p-category element, the tag name is "mbnov" and not "#mbnov", which I believe would be incorrect.
But looking at the Mastodon code, I think it is parsing the toot content, looking for patterns of the form "#
So the article hashtag gets linked, because the tag is represented as "#books", while the note hashtag doesn't get linked because the tag isn't formatted in a way that Mastodon will pick up.
To test this, I'd want to change the note markup to place the "#" inside the anchor tag (so the text is formatted correctly for Mastodon), while hacking BF/Granary to remove the leading "#" before populating the Tag in the AS2.
So... thoughts on this essay? :)
Edit:
BTW, I have a dirty workaround I'm going to try: I'll switch to only having a p-summary in my note h-entry's, with no p-content, and see what happens. :)
Alright, well, that idea didn't work.
Then I changed up the formatting so the p-category element is outside the main content, and the category link itself is embedded in the content.
The corresponding BF syndication is here.
Pulling the content out, we now have:
<div class=\"categories\"><span class=\"category\"> <a href=\"/archives/category/indieweb/\">#indieweb</a> </span></div>
And it's still not rendering properly, as you can see here.
I'm honestly at a bit of a loss at this point...
Edit:
One last hail mary, I altered the HTML so now this is passed in the content:
<div hidden=\"\">#indieweb </div>
And still, not linked.
At this point I'm out of ideas and have probably spammed enough for one night. :)
Ugh. Thank you for the sleuthing, and sorry it's been such a wild goose chase with no answer yet!
It's not a top priority for me personally right now, but if/when I do look myself, I'll definitely post here if I find anything new.
No worries! For now I've switched over to straight brid.gy, puppeteering my account on indieweb.social, and the first test was flawless. And thanks to your hard work, switching between those two methods is, like, an hour of effort on my part, so cutting back will be easy enough once this is resolved.
Meanwhile, I admit I'm tempted to stand up BF and a local Mastodon instance to experiment with this further without spamming the fediverse. It's baffling! And I hate being baffled...
Another data point, this post with p-category
has the tag
in the create request, but hashtag didn't show up on Mastodon:
{
"@context": "https://www.w3.org/ns/activitystreams",
"type": "Create",
"id": "https://fed.brid.gy/r/https://gregorlove.com/2022/11/hello-again-from-my-website/#bridgy-fed-create",
"actor": {
"id": "https://fed.brid.gy/gregorlove.com",
"url": "https://fed.brid.gy/r/https://gregorlove.com/",
"preferredUsername": "gregorlove.com"
},
"object": {
"published": "2022-11-30 11:54-0800",
"content": "<p>@artlung@xoxo.zone Hello again from my website! This time it should show up as a mention instead of a link.</p>",
"url": "https://fed.brid.gy/r/https://gregorlove.com/2022/11/hello-again-from-my-website/",
"@context": "https://www.w3.org/ns/activitystreams",
"type": "Note",
"tag": [
{
"type": "Tag",
"name": "indieweb"
}
],
"id": "https://fed.brid.gy/r/https://gregorlove.com/2022/11/hello-again-from-my-website/",
"to": [
"https://www.w3.org/ns/activitystreams#Public"
]
}
}
Summarizing some sleuthing and discussion from #indieweb-dev:
Fetching the AS2 with curl -H
for a Mastodon post:
{
"@context": [
"https://www.w3.org/ns/activitystreams",
{
"ostatus": "http://ostatus.org#",
"atomUri": "ostatus:atomUri",
"inReplyToAtomUri": "ostatus:inReplyToAtomUri",
"conversation": "ostatus:conversation",
"sensitive": "as:sensitive",
"toot": "http://joinmastodon.org/ns#",
"votersCount": "toot:votersCount",
"Hashtag": "as:Hashtag"
}
],
"id": "https://hachyderm.io/users/liztai/statuses/109907766432901624",
"type": "Note",
"summary": null,
"inReplyTo": "https://hachyderm.io/users/lukzmu/statuses/109905124938988159",
"published": "2023-02-22T09:50:24Z",
"url": "https://hachyderm.io/@liztai/109907766432901624",
"attributedTo": "https://hachyderm.io/users/liztai",
"to": [
"https://www.w3.org/ns/activitystreams#Public"
],
"cc": [
"https://hachyderm.io/users/liztai/followers",
"https://hachyderm.io/users/lukzmu"
],
"sensitive": false,
"atomUri": "https://hachyderm.io/users/liztai/statuses/109907766432901624",
"inReplyToAtomUri": "https://hachyderm.io/users/lukzmu/statuses/109905124938988159",
"conversation": "tag:hachyderm.io,2023-02-21:objectId=28543079:objectType=Conversation",
"content": "<p><span class=\"h-card\"><a href=\"https://hachyderm.io/@lukzmu\" class=\"u-url mention\">@<span>lukzmu</span></a></span> All I have to say is <a href=\"https://hachyderm.io/tags/IndieWeb\" class=\"mention hashtag\" rel=\"tag\">#<span>IndieWeb</span></a>. I will always have a place I can call my own with my website. My social media channels are just a way to distribute and amplify my content.</p>",
"contentMap": {
"en": "<p><span class=\"h-card\"><a href=\"https://hachyderm.io/@lukzmu\" class=\"u-url mention\">@<span>lukzmu</span></a></span> All I have to say is <a href=\"https://hachyderm.io/tags/IndieWeb\" class=\"mention hashtag\" rel=\"tag\">#<span>IndieWeb</span></a>. I will always have a place I can call my own with my website. My social media channels are just a way to distribute and amplify my content.</p>"
},
"attachment": [],
"tag": [
{
"type": "Mention",
"href": "https://hachyderm.io/users/lukzmu",
"name": "@lukzmu"
},
{
"type": "Hashtag",
"href": "https://hachyderm.io/tags/indieweb",
"name": "#indieweb"
}
],
"replies": {
"id": "https://hachyderm.io/users/liztai/statuses/109907766432901624/replies",
"type": "Collection",
"first": {
"type": "CollectionPage",
"next": "https://hachyderm.io/users/liztai/statuses/109907766432901624/replies?only_other_accounts=true&page=true",
"partOf": "https://hachyderm.io/users/liztai/statuses/109907766432901624/replies",
"items": []
}
}
}
Note the tag
property:
{
"type": "Hashtag",
"href": "https://hachyderm.io/tags/indieweb",
"name": "#indieweb"
}
The AS2 extension Hashtag
appears to be from Eugen in Mastdon (discussion):
I have added type: "Hashtag" to Mastodon because when I'm parsing the "tag" attribute and iterating over the items I'd like to know what the item is. A "Mention" type exists so there should be a "Hashtag" type as well.
It was proposed as an AS2 extension here and this issue, but no further discussion since 2017-10 that I can see (did not check mailing list archives).
Hashtag
appears to be an extension of AS2 Link
type since it has an href
and not id
property (Object
type has id
).
I'm leaning towards thinking (hoping) that BF only needs to add "type": "Hashtag"
and ensure the hashtag starts with #
.
OK I don't know what to think. I tried what we thought would work, type: Hashtag
with leading #
in name. I also tried all other permutations of type
, id
vs href
, and leading #
. No dice. None of them resulted in "native" Mastodon hashtags.
Post links:
tag
objects:
[{
"type": "Hashtag",
"name": "type-hashtag",
"href": "https://fed.brid.gy/#tag-type-hashtag",
}, {
"type": "Tag",
"name": "type-tag",
"href": "https://fed.brid.gy/#tag-type-tag",
}, {
"name": "no-type",
"href": "https://fed.brid.gy/#tag-no-type",
}, {
"type": "Hashtag",
"name": "#type-hashtag",
"href": "https://fed.brid.gy/#tag-type-#hashtag",
}, {
"type": "Tag",
"name": "#type-tag",
"href": "https://fed.brid.gy/#tag-#type-tag",
}, {
"name": "#no-type",
"href": "https://fed.brid.gy/#tag-#no-type",
}]
[{
"type": "Hashtag",
"name": "type-hashtag",
"id": "fed.brid.gy-tag-type-hashtag",
}, {
"type": "Tag",
"name": "type-tag",
"id": "fed.brid.gy-tag-type-tag",
}, {
"name": "no-type",
"id": "fed.brid.gy-tag-no-type",
}, {
"type": "Hashtag",
"name": "#type-hashtag",
"id": "fed.brid.gy-tag-type-#hashtag",
}, {
"type": "Tag",
"name": "#type-tag",
"id": "fed.brid.gy-tag-#type-tag",
}, {
"name": "#no-type",
"id": "fed.brid.gy-tag-#no-type",
}]
@snarfed Hm, did any tests include the same hashtag in the content? Maybe Mastodon is matching it, since they link the hashtags in the HTML view?
Like:
Testing with #typehashtag
With tag
object:
{
"type": "Hashtag",
"name": "#typehashtag",
"href": "https://fed.brid.gy/#tag-typehashtag",
}
I took out the hypen in the hashtag because I don't know if that's supported.
Also not sure if I'm supposed to see it there, but when I click those indieweb.social links, I don't see tag
objects in the JSON.
Good points! Added all tag names to the content, removed hyphens, tried again, still nothing. https://indieweb.social/@snarfed.org@snarfed.org/109928901481785754
And yeah you'll only see it on indieweb.social if you're logged in. I'm constructing and sending these AP Create
activities manually, so there's no way to fetch or see them in other instances. When you're logged out, the link above redirects you to a synthetic fragment URL on fed.brid.gy that serves the actor for @snarfed.org@@snarfed.org, which is more or less unrelated.
BookWyrm recently added hashtags. See https://github.com/bookwyrm-social/bookwyrm/pull/2524 and this comment.
Interesting! Notably, from https://github.com/bookwyrm-social/bookwyrm/pull/2524#issuecomment-1435733107 :
Federation support from BookWyrm to Mastodon and takahe: Hashtag links get rewritten to point to the Mastodon/takahe hashtag page
I don't see the difference though. The JSON in your link https://github.com/bookwyrm-social/bookwyrm/pull/2524#issuecomment-1438789816 matches what you described above in https://github.com/snarfed/bridgy-fed/issues/45#issuecomment-1445257933, which I tested in https://indieweb.social/@snarfed.org@snarfed.org/109928901481785754.
The only possible difference I see is that the hashtag in their content is inside an <a>
link. Maybe that's necessary?
@snarfed Yeah, I should have included what @tantek said in chat. It appears the content needs to have the <a href>
with URL matching the href in the JSON.
Weirdly, the URL comparison may not be case-sensitive? That AS2 from https://github.com/snarfed/bridgy-fed/issues/45#issuecomment-1444950494 has content:
<a href=\"https://hachyderm.io/tags/IndieWeb\" class=\"mention hashtag\" rel=\"tag\">#<span>IndieWeb</span></a>
and has JSON tag:
{
"type": "Hashtag",
"href": "https://hachyderm.io/tags/indieweb",
"name": "#indieweb"
}
Oh wow, it worked! Adding the <a>
tag in content did the trick. Here's my test post, created with this AS2:
{
"@context": "https://www.w3.org/ns/activitystreams",
"type": "Create",
"cc": ["as:Public"],
"object": {
"type": "Note",
"id": "https://fed.brid.gy/r/https://snarfed.org/#2023-03-13-hashtag-test-a",
"content": "testing hashtag <a href=\"https://snarfed.org/#foobar\">#foobar</a>",
"tag": [{
"type": "Hashtag",
"href": "https://snarfed.org/#foobar",
"name": "#foobar"
}]
}
}
So does that just require the right markup on the source site or is there some BF work that needs to be done to bring it all together?
@fancypantalons good q! I don't think source sites need new or different markup, p-category
should be enough, but the UX options here are awkward. Mastodon requires hashtags in links in the content, but the content is obviously user-visible, so I'm reluctant to have BF change it programmatically much or at all.
Open to ideas here! @tantek is also discussing this in chat right now.
(btw, note to myself, this issue should obviously track adding docs for whatever we conclude here.)
Maybe it will help to figure out next steps using a previous post of mine:
The post has #KeepMasksInHealthcare
in the content, not linked. It also has a few p-category
because I was trying to see if that would work for BF. We can probably ignore the p-category for now.
Truncated mf2:
"type": [
"h-entry"
],
"properties": {
"category": [
"covid",
"KeepMasksInHealthcare",
"KeepMasksInHealthcareCA"
],
"url": [
"https://gregorlove.com/2023/02/covid-transmission-is-still-high/"
],
"published": [
"2023-02-14 11:51-0800"
],
"content": [
{
"html": "<p>COVID transmission is still high around the US and we need masks to protect our most vulnerable in healthcare settings. Help us protect those who need it most by telling your governor to #KeepMasksInHealthcare:</p>\n\n<p><a href=\"https://peoplescdc.substack.com/p/keep-masks-in-healthcare\">Learn More</a> and <a href=\"https://actionnetwork.org/letters/keep-masks-in-healthcare/\">Send a letter: Keep Masks in Healthcare</a></p>",
"value": "COVID transmission is still high around the US and we need masks to protect our most vulnerable in healthcare settings. Help us protect those who need it most by telling your governor to #KeepMasksInHealthcare:\nLearn More and Send a letter: Keep Masks in Healthcare",
"lang": "en"
}
]
}
What would be the best/easiest steps for me to edit this post and have BF create the tag
JSON object?
<a href>
in the content around the hashtag?E.g. made-up URL: <a href="https://gregorlove.com/tags/KeepMasksInHealthcare">#KeepMasksInHealthcare</a>
I think Mastodon would expect this tag JSON, then:
{
"type": "Hashtag",
"href": "https://gregorlove.com/tags/KeepMasksInHealthcare",
"name": "#KeepMasksInHealthcare"
}
I'm guessing, but I suspect that href
has to match the URL in the content in order for each instance to replace the hashtag link displayed in the post. If that's the case, I don't know how BF can extract that URL without something like rel=tag
Thank you @gRegorLove! I'll have something for you to try soon.
More experiment learnings:
name
and content link text must start with #
.https://fed.brid.gy/[anything]
, the link preview is suppressed. That's not really useful for users though, they're not going to link their hashtags on their site to BF. 😐Disregard, I missed that link preview was coming from the content not the JSON.
~@snarfed What about with id
instead of href
, since that works for @aaronpk? Screenshot: https://media.aaronpk.com/2023/03/14130405-4386.png~
Looks like it's working! Thanks for testing @gRegorLove! https://indieweb.social/@gregorlove.com@gregorlove.com/109864841025927677
OK everyone, feel free to try! Here are the docs: https://fed.brid.gy/docs#hashtags
Will this work if the hashtags are in the p-summary instead of the e-content? For my long-form posts, I use a p-summary that contains the content that's syndicated out to Mastodon, and it's in that block where the hashtags are linked up.
Somewhat, yes. As described in https://fed.brid.gy/docs#hashtags , they'll be invisible: Mastodon search will find them, but they won't be displayed in your post text. That still strictly comes from your original post's content; Bridgy Fed won't change it, nor will Mastodon afaik.
Relevant background discussion in https://github.com/snarfed/bridgy/issues/478
Hmm... so... suppose I have this:
<div class="h-entry">
<p class="p-summary">
This is a summary of my post, complete with a hashtag. <a href="https://mysite.com/tag/throwback" class="p-category">#throwback</a>
</p>
<p class="e-content">
And here's the long-form content for that post.
</p>
</div>
Brid.gy will send over the p-summary as the text for the post, in addition to a backlink.
For example, this Mastodon post is a syndicated copy of this article, and the Mastodon text comes from the p-summary block in that article.
I vaguely recall BF inherits most of that behaviour (minus the backlink, obviously).
Assuming that's right, I'm curious if, in the above example, "#throwback" will get properly sent over as a Hashtag in the AS2 and the text will be linked.
And if this isn't easy to answer I'm more than happy to just cobble something together and give it a shot!
Ah! Looks like Bridgy (non-Fed) created that Mastodon post. You're right, it does prefer summary to content.
Bridgy Fed's behavior is similar but a bit different. (whee!) Under the hood, it doesn't have to choose a single piece of text to post, it federates full fledged AS2 objects. It tries to make those match your post as closely as possible. In this case, it would deliver a Note
with both summary
and content
. Mastodon interprets/abuses summary
for their content warning UI, so I think your text would end up as a content warning, and then show the content if you click through. I don't know if hashtags in summary/content warning work, but feel free to try!
Ooooh yeah! That does ring a bell now that you say it... ahh fun. Thanks!
I posted a reply to a Mastodon post and I included #IndieWeb in the text of my post. When I sent the post through Bridgy Fed, it’s plain text as opposed to the native Mastodon posts that have it linked.
See the posts and my reply in this thread: https://cybre.space/@katastrophe/101661581248390741
Did I need to have a u-category on my hashtag or is this not supported yet on Bridgy Fed?