Bluesky: publishing

[JoelOtter]

We only have backfeed for bsky currently - will use this issue to track publishing work.

[JoelOtter]

Trying to work out how to actually turn this on. Setting CAN_PUBLISH = True and adding Bluesky to the list of sources in publish.py doesn't seem to be enough. I think it's something to do with adding 'publish' to the source's features field but I'm not sure when to do that when Bluesky doesn't actually use auth scopes at all. Do you think it's OK to just add this feature to the source at initial login, and if so, where's the best place to do that? Is it in bridgy or in oauth-dropins?

[snarfed]

Yes! publish in features is it. And good question re scopes. I guess we could make it a Bridgy-internal toggle that users can opt into, but I'm also fine with always enabling it, at least for now. I think you'd just hard code feature here to 'listen,publish':

https://github.com/snarfed/bridgy/blob/87038d39157476a526a6bd7ce9f566679dd1fb08/bluesky.py#L73

More:

https://github.com/snarfed/bridgy/blob/87038d39157476a526a6bd7ce9f566679dd1fb08/models.py#L525-L535

Btw backfeed seems pretty stable, congrats again! I'm ready to announce it if you are. Want to do the honors?

[JoelOtter]

Great, thanks, I'll have a go at this :)

Btw backfeed seems pretty stable, congrats again! I'm ready to announce it if you are. Want to do the honors?

Sure! Will write that soon- prob not today as I'm a bit under the weather but in the next few days for sure

[JoelOtter]

Found some energy from somewhere! Hope this is OK. https://www.joelotter.com/posts/2023/10/bridgy-bluesky/

[JoelOtter]

Random note: Bluesky still doesn't have proper embed support, and bsky.link appears to be dead, so I'm skipping post embeds in reply previews.

[JoelOtter]

Bluesky has link cards (previews0, which unlike on other networks are optional and it's up to the user which link in a post becomes the link card. I propose that to start with we just handle the last link in a post as the card, as that feels like the most common usage. Later, we might want it to be adjustable using some specific bridgy property. What do you think?

[JoelOtter]

Though interestingly Mastodon does actually choose the first link. I still think last is the more desired behaviour...

[snarfed]

Last link sounds good! Or even skip embeds altogether to start. I'm all for starting with the simplest thing that works end to end, then expanding.

[JoelOtter]

Think I've got preview pretty much working for the basic feature set. I'm a bit confused how to actually do stuff against Bluesky though. Do I need to write raw ATproto repo-write queries? Does lexrpc include a way to do this? Had a look at how bridgy-fed does it and it seems very low-level!

[snarfed]

Hah, yes! I think you just call com.atproto.repo.createRecord. It's a little hidden because it's in the com.atproto lexicons, not app.bsky. https://atproto.com/blog/create-post

[JoelOtter]

Hmm I'm not sure what I need to add in terms of the endpoint handling for e.g. /bluesky/publish/start, it seems very tied to the OAuth dropins in a way that I find a bit confusing given Bluesky's OAuth is fake. Any advice?

[snarfed]

That is totally true. 😕 I had the same problem when I added delete for Bluesky a bit ago, maybe follow the way that works?

[JoelOtter]

So I'm still mildly baffled by the flow here but I think what I'm missing is functionality for the OAuth Dropins Bluesky Callback's dispatch_request function to be able to read state out of the request, and use the source ID in that to retrieve an existing user auth, if it exists. Does that sound correct? Is that actually safe to do?

[snarfed]

Sounds right, but you can probably get much of it for free, let me come back when I have a bit more time to sketch something. Btw feel free to skip interactive publish/preview entirely for now if you want, I'm happy to ship just webmention-based publish, and then add on interactive afterward!

[JoelOtter]

Oh, I didn't clock that those were separate flows.

[snarfed]

Sorry the auth stuff is so difficult btw! Obviously the non-OAuth part here is awkward, but also the auth code on the Bridgy side in general is probably over-abstracted.

[JoelOtter]

Random discovery is that Bluesky (presumably intentionally given a repo is supposed to be user-owned?) does not validate dates. We can publish posts at the time the source was written. https://bsky.app/profile/joelotter.com/post/3kefvqwq6g424

[JoelOtter]

(I published that post with Bridgy!)

[snarfed]

Awesome! Congrats!

And yeah people like @DavidBuchanan314 have been all over that. The team says indexedAt is the more-trusted timestamp since it's from the relay/appview. Still, amusing!

[JoelOtter]

Am I right in thinking lexrpc doesn't currently support non-JSON request bodies?

[JoelOtter]

Alright, I've got it uploading images!

The next thing is facets for links and mentions. I can see that from_as1 actually has this, but it a) only does facet logic when the HTML content is identical to the text, which I don't see ever being true and b) appears to expect links and mentions to be present as tags on the AS1 object, which I don't think we currently do in vanilla bridgy?

[snarfed]

So cool! Congrats! And yeah good point, lexrpc supports binary output and input (less well tested) server side, but not client side, sorry about that. Did you have to add it? (Thank you if so!)

[snarfed]

Re facets, afaik we've never actually used the AS1 => Bluesky facet code for anything real yet, so yeah I can believe it's untested (at least in real situations) and likely broken in some ways.

For HTML content though, yes, that's intentional, I probably should have added a comment. I did the conversion from AS1 tags since it was easy, but for HTML, I didn't want to try yet since it seemed harder, and I didn't have a use case yet. I guess we do now!

We figured out lots of these details for publishing to Twitter, hopefully those decisions work here too. Eg we parse out mf2 and use it for explicit structural elements like media, and otherwise just run source.html_to_text and use its output as is. For Bluesky facets specifically, I don't actually think we have any precedent in Bridgy publish for @-mentions or links. I guess we could try! I'd be inclined to think about it after we ship a first version with just plain text though.

(Fwiw we do end up with AS1-style tags with indices in granary/Bridgy when we convert from some silos, eg tweets from Twitter with mentions, links, etc. Doesn't really matter here though!)

[JoelOtter]

I ended up just implementing upload with requests directly - I find lexrpc quite hard to reason about internally due to the codegen stuff so I might not be the best person to try and add a whole additional class of input data!

RE facets, that makes sense. I think the linkify stuff we use for preview might be of some help here, though there will assuredly be some weird nuance due to unicode shenanigans.

Are we able to ship the micropub stuff without showing any publishing UI? If so that might work well, otherwise I'd worry about users seeing a broken-ish feature.

[JoelOtter]

nvm I got over my fear https://github.com/snarfed/lexrpc/pull/5

[snarfed]

Are we able to ship the micropub stuff without showing any publishing UI? If so that might work well, otherwise I'd worry about users seeing a broken-ish feature.

Good point! I guess I more meant merge than ship, ie we could merge this with just webmention support first, to keep the PR(s) manageable. We can definitely ship that first too, but yeah we'd want to hide the interactive form part of the publish UI if we do.

[JoelOtter]

Apologies @snarfed, need to focus on some non-Bridgy stuff for a bit that I've been neglecting! I've put my WIP in the bsky-publish branch on my fork of Granary, which is where basically all of the work is: https://github.com/JoelOtter/granary/tree/bsky-publish

I mean to come back to this in two or three weeks but if you'd like to race ahead please don't hold back on my account!

[snarfed]

No worries! Totally ok, no obligation. I probably won't work on it much myself in the meantime, but I'll keep you posted. Best if luck with the rest!

[snarfed]

@JoelOtter just FYI I'm starting to look at this again. Let me know if you have any time freed up and you're interested in working on it with me!

[JoelOtter]

Hello! I'm still a bit swamped - we're bringing our game out of Early Access at the end of February so I have a few things pulling my attention until then. From March on I'll have a lot more time to get back to this, but if you want to go ahead please do! Happy to help out with reviewing code in the interim.

[snarfed]

Thanks for getting this started @JoelOtter! I added tests and tweaked the code and made some progress, first pass at granary Bluesky preview and create are done: https://github.com/snarfed/granary/compare/0cb4ed215207fca997f35f7cef9660f99768c97d..3278fb2a0a3d830acfa7e5ff8270a4e36e8cf15e

[JoelOtter]

Woohoo! Very excited for this :)

[snarfed]

@JoelOtter I've soft launched a first pass at this. Feel free to try on https://brid.gy/bluesky/did:plc:ioz4ztghfznx4s5s4jxqiqun !

Code changes are https://github.com/snarfed/granary/compare/0cb4ed2..0525dab (including your initial commit) and https://github.com/snarfed/bridgy/compare/575c05c..f3c0a0d if you're interested.

[JoelOtter]

It's working! Text and images go through. The missing piece I guess is links? Bluesky is weird, it doesn't automatically linkify anything, example here: https://bsky.app/profile/joelotter.com/post/3kkgccfjmw22r

[JoelOtter]

Replies don't seem to work either. Would you like me to jump in and help? I've got a little bit of time this weekend!

[snarfed]

Awesome, sure! I'm impressed that your blob upload code works, I hadn't tested it yet. And you're right, facets are still TODO. But I did have replies (and likes and reposts) working, so I'm not sure what's up there.

[JoelOtter]

No worries - I'll try and dig into what's happening with replies. I think I did have blob upload working before in my branch so if you're using some of my stuff that's probably why :) Very untested though!

[JoelOtter]

Oh, I just thought to check my cURL logs and sure enough:

{
  "error": "Error: {\"error\":\"InvalidRequest\",\"message\":\"Invalid app.bsky.feed.post record: Record/reply/root/uri must be a valid at-uri\"} 400 Client Error: Bad Request for url: https://bsky.social/xrpc/com.atproto.repo.createRecord"
}

Looks like it currently only works for replies directly to other AT posts.

[snarfed]

Ahh yeah that makes sense. I caved a while back and made publish accept "external" replies like that as normal notes, so I guess we should do that here too.

[JoelOtter]

So if the reply is to a note we check its syndication links and reply to the matching syndicated post on the silo? It works for Masto so I guess I can just follow its implementation :)

[snarfed]

Oh that already happens, in publish.py, before it gets to any Bluesky-specific code. I meant if the in-reply-to wasn't POSSEd to Bluesky, or if it's on a different silo, etc. We originally didn't allow that, but enough people wanted it that I eventually allowed it and just converted to a normal note, no in-reply-to.

[JoelOtter]

Ahh do you think in this case then it's because the in-reply-to is a bsky.app URL, rather than a AT URI?

[snarfed]

Hmm no, bsky.app URLs should work too, I tested with those!

[JoelOtter]

Ah sorry I mis-wrote - the in-reply-to is a note URL, but the syndication link on that note is a bsky.app URL.

[JoelOtter]

This is the note it fails for - it's got a bsky link on it now but that's cos I added it manually https://www.joelotter.com/notes/2024/02/02-celeste2/

[snarfed]

Ah ok! Yeah it's very possible that the in-reply-to synd link handling isn't working for Bluesky. It's here: https://github.com/snarfed/bridgy/blob/f3c0a0da56feed6db362b784a0d9cd9759e83183/publish.py#L470-L528

[JoelOtter]

Fab, thanks! I'll have a crack at that this weekend, will be a good way to ease back into the swing of things. :)

[JoelOtter]

So, I think the naive solution to support the URLs is to make this change in Granary:

diff --git a/granary/bluesky.py b/granary/bluesky.py
index 4e199afb..140b2045 100644
--- a/granary/bluesky.py
+++ b/granary/bluesky.py
@@ -571,7 +571,7 @@ def from_as1(obj, out_type=None, blobs=None, client=None):

     # in reply to
     reply = None
-    in_reply_to = as1.get_object(obj, 'inReplyTo')
+    in_reply_to = client.base_object(obj)
     if in_reply_to:
       parent_ref = from_as1_to_strong_ref(in_reply_to, client=client)
       reply = {

However, this kind of makes from_as1 require client, rather than it be an option. I'm not sure how best to proceed:

• Make base_object a free function rather than on the Bluesky class, as it never uses its self parameter. I'm not sure this can work though as it's overriding a method on Source?
• Make client a required parameter or, even scarier, put from_as1 on the Client itself.
• Just pass the client along everywhere we can (naive solution but feels fragile)
• Make base_object a free function and have the class method call it

I do worry I'm stumbling into doing something silly here so if you have advice I'd appreciate it!

[JoelOtter]

I've made a PR https://github.com/snarfed/granary/pull/673 with the last solution on this list, which does seem to work from my local testing.

[snarfed]

Nice! Pulling from the base object instead of raw inReplyTo looks like exactly the right idea. I'll take a look soon.