Details on rss cloud calls for protocol http-post

[johnspurlock]

Hi there - I'd like to support the cloud element over http-post.

Given an rss feed like: https://rss.firesky.tv/?filter=quake

<rss version="2.0" xmlns:source="http://source.scripting.com/">
  <channel>
    ...
    <cloud domain="rss.firesky.tv" port="443" path="/rss-cloud-notify" registerProcedure="" protocol="http-post" />

I'll just be using the web standard fetch api on the server side.

It would help to see:

• an example HTTP request/response incoming subscription call from clients
• an example HTTP request/response outgoing notification call to notify clients (guessing something like this?)

HTTP method/url/headers/bodies/status should be enough.

Thanks! - John

[scripting]

I had a post here, but it was wrong too.

I don't think I have example code for what you're looking for.

Back in a minute.

[scripting]

I use this cloud server in all my work.

<cloud domain="rpc.rsscloud.io" port="5337" path="/pleaseNotify" registerProcedure="" protocol="http-post"/>

@andrewshell runs the server. it's fast and reliable.

But you can run your own. Since I don't run a cloud server myself I don't have good example code for it.

I'd recommend sticking with rss.cloudPing, you get the benefit without any of the work. ;-)

[scripting]

@andrewshell has made the source for his cloud server open source.

https://github.com/rsscloud/rsscloud-server

[johnspurlock]

hmm, I'm fine with implementing it myself, since its only two calls. I need to be able to know who's subscribed and for what topics to avoid extra processing on every incoming Bluesky post!

I already have the endpoint specified in that rss cloud element ready to go: /rss-cloud-notify implemented on that domain. Just need an example of the two http-post calls.

Or maybe there is a public client that supports rss cloud, and will try to subscribe to this feed? I could then at least see what it sends : )

[scripting]

I think I understand what you’re looking for.

I would try river5, a feed reader I wrote a few years ago, it supports the client side of the protocol.

https://github.com/scripting/river5

[scripting]

Here’s the rssCloud code.

https://github.com/scripting/reader/blob/master/davereader.js#L1949

[johnspurlock]

thanks! will take a look

[scripting]

@johnspurlock -- checking in -- how's it going?

[johnspurlock]

I think I got it, should have it up and running in production later day (I'm in US central time)

For anyone stumbling across this thread, the http-post protocol appears to be:

Given a cloud element:

<cloud domain="server.example.com" port="123" path="/please-notify" registerProcedure="" protocol="http-post" />

The incoming subscribe HTTP request/response looks like:

POST http://server.example.com:123/please-notify
accept: application/json
content-type: application/x-www-form-urlencoded

port=<callback-port>&path=<callback-path>&url1=<feed-url>&protocol=http-post

200 OK
content-type: application/json

{ "success": true }

or

400 Client error
content-type: application/json

{ "success": false, "msg": "<error-message>" }

The server needs to remember the ip address of the subscribe request, let's call it subscriber-ip

Subsequent outgoing feed change notifications are sent to subscribers as follows:

POST http://<subscriber-ip>:<callback-port><callback-path>
content-type: application/x-www-form-urlencoded

url=<feed-url>

200 OK

Note everything is done with unencrypted http, not https

[scripting]

BTW, I should've sent you these pointers yesterday. I had guests over for a BBQ at the time, so I wasn't focused well enough.

http://home.rsscloud.co/

http://scripting.com/opmlHowto/dave/rssCloud/walkthrough.html

http://walkthrough.rsscloud.co/

I think there is other pointers as well.

[scripting]

Also I want to try adding a bit of social glue to the feeds you're building.

Can you surface the names of people other people have subscribed to, as an OPML subscription list?

Happy to show you how to do that..

[andrewshell]

Let me know if you have questions with my implementation.

The standard for http-post is not https but I added support as https-post which works exactly the same only https. There is also an optional domain property on pleaseNotify if you don't want to have it ping the ip address.

Here are the docs: http://rpc.rsscloud.io/docs

[scripting]

I just wrote a blog post about this stuff.

http://scripting.com/2023/05/15.html#a132038

[scripting]

@johnspurlock -- turns out we needed https support for it to work with FeedLand.

[johnspurlock]

Gotcha, I guess I can support https-post for subscribers as well. Hopefully you guys have reader/client software that can test it out (ie subscribe) to one of the firesky.tv feeds when it's live. Seems to be working with a local version of river5.

Keen on implementing any push notification systems for Firesky, given its real-time nature. I could see this being useful for folks subscribing to multiple people and getting new posts as soon as they're posted.

@scripting interested in what you're think regarding surfacing Bluesky graph over opml, maybe start a new issue dedicated to that and we can hash it out.

[scripting]

reader/client software that can test it out (ie subscribe) to one of the firesky.tv feeds when it's live.

https://feedland.org/

I have already subscribed to my own feed. It fully supports rssCloud.

I might start a fresh instance of the feedland software just for bluesky.

It could be really interesting.

It's already set up to be a social network. ;-)

[scripting]

@johnspurlock -- re the OPML angle, I'm thinking about that. ;-)

[johnspurlock]

alright, I deployed a new Firesky version:

• post images, if present, are returned as rss feed item enclosures
• new advanced filter from:username or from:@username or from:@username.com to limit to content from the user, not including text matches in others posts
• rss cloud element should be wired up. the wiring seems to work using your feedland app! it supports rss subcribers using the http-post and https-post protocols

example:

• live view: https://firesky.tv/?filter=from%3A%40davew
• associated feed: https://rss.firesky.tv/?filter=from%3A%40davew

[scripting]

@johnspurlock -- excellent! you're doing great work.

Back in a minute with test results.

[scripting]

@johnspurlock --

• I subscribed to the associated feed in FeedLand.
• Posted a test item to my Bluesky account.
• And as expected, the feed went to the top of my feed list, and the new item appeared beneath. Here's a screen shot.
• So, we have basic rssCloud functionality working. Very well done.

[scripting]

@johnspurlock --

You asked yesterday how to bring lists of Bluesky feeds into it via OPML subscription lists. I gave it some thought and this is what I came up with as a first step.

I'd like to be able to view a list of the accounts I'm subscribed to. Perhaps something like this:

https://opml.firesky.tv/?filter=from%3A%40davew

Here's what an OPML subscription list produced by FeedLand looks like:

https://feedland.org/opml?screenname=davewiner

They don't have to be in any specific order.

There is a spec for subscription lists --

http://opml.org/spec2.opml#1629042482000

If you want I can find code that generates them, but probably wouldn't be necessary because this is a very simple format, imho.

If you did this, I would probably set up a special instance of feedland just for this use, something like blue.feedland.org.

It wouldn't just work with Bluesky, presumably we'll be able to entice other platforms to produce feeds that work this well.

As far as I know yours is the best implementation of feeds on a social media site at this time.

[scripting]

@johnspurlock -- i'm going ahead and creating a special install of feedland to work with these feeds.

really excited about the possibilities.

i sent an email to the account listed on your home page, which i found via your github page.

my address is dave.winer@gmail.com if you don't get it, send me a note and i'll re-send.

[johnspurlock]

I see your email, just getting started over here today (I don't have east coast superpowers these days) - and will probably be at my work desk in a bit.

Happy to create an endpoint to host lists pointing to underlying firesky feeds. Carrying over the filter semantics doesn't make as much sense for the aggregated lists, so I'll probably just create normal path-based urls for them - since other groups of feeds might be useful later. e.g. opml for a users named lists (like Twitter lists) once Bluesky has them, trending users/topics, mutuals, even block/mute lists are public in bsky.

But getting an opml for a users follows, pointing to underlying user account feeds would be a straightforward first step.

The list itself shouldn't be too difficult, but I should first start indexing the underlying firehose posts by account, to make queries limited to accounts more efficient, and enable a longer lookback.

[scripting]

I have a new server with just one app on it -- an instance of FeedLand set up for subscription lists and feeds that emanate from bluesky. Tomorrow I'll start working on that part of it.

[johnspurlock]

Alright check this out:

👉 https://firesky.tv/lists/@davew/follows.opml

[scripting]

@johnspurlock --

I was able to import all the feeds from the OPML file into my new instance of FeedLand.

I'm going to let it run for a while, and then pass you a pointer so you can have a look.

There are some obvious tweaks that will make it work/look better, but there are also some big things I want to try out.

Back in a bit..

[scripting]

@johnspurlock --

i had to provision another server at digital ocean, but now blue.feedland.org is running nicely.

i've subscribed to all the feeds in the opml file you provided. and all the updates seem to be coming through.

you can watch the updates here.

https://blue.feedland.org/?username=dave -- my feed list, it's real-time. and if you click the wedge next to a feed, you'll see the first five items in the feed.

i'm going to take a break now and start thinking about some first tries and UIs for this flow of info.

thank you john. this is a lot of fun. hooking up two disparate systems, that's how tcp/ip was developed, and http, and basically everything. ;-)

dave

[scripting]

btw, go ahead and create an account on blue.feedland.org.

it's a completely separate database from the main feedland.org site.

we're going to use this mostly for feeds coming from social nets like bluesky.

[johnspurlock]

Heads up all of the cloud subscriptions I see coming in still have feedland.org as the domain parameter, not blue.feedland.org, maybe something to configure?

[scripting]

@johnspurlock -- thanks for the report. I found the problem, indeed something is hard-coded. i want to think about how to fix it.

[scripting]

@johnspurlock -- i believe the problem is now fixed.

it just renewed a subscription and appears to have used the correct values for the domain and port (blue.feedland.org and 80).

[scripting]

I just reset the timer on all the subscriptions, so they should resub with the correct domain one every minute until they're all done. And then we should start getting the rssCloud notifications on the new server.

[scripting]

Interesting to see the realtimeness start up, as more feeds are correctly configured for rssCloud we get more notifications.

[johnspurlock]

In theory once everything is subscribed, you should have list of posts that is roughly as live as the bluesky app itself

[johnspurlock]

Ok, after a bunch of backend work, any from: queries should be returned much faster, and can potentially look back farther than a few hours. Figured it would be good to optimize these since those opml lists create hundreds of them

https://rss.firesky.tv/?filter=from%3Agruber.bsky.social

[johnspurlock]

meant to say: be on the lookout for any new weirdness - these new optimized queries go through an entirely different codepath on the backend

[scripting]

@johnspurlock --

just starting work for the day now, and looked through the list of feeds and see some have not renewed.

here's an example of a feed. that has not.

https://rss.firesky.tv/?filter=from%3Amoonalice.bsky.social

I think the source of the problem is that there are no items in the feed.

but moonalice has posted.

here's what feeder app returns.

https://feeder.scripting.com/returnjson?feedurl=https://rss.firesky.tv/?filter=from%3Amoonalice.bsky.social

and sure enough -- no cloud element even though it's present in the feed.

i will fix this in the reallysimple package, but there will be a bunch of feeds that we haven't registered for notification on.

i could generate a list if that would help...

[scripting]

BTW, these are the feeds that (probably) have no items, based on the above.

+-----------------------------------------------------------------+
| feedurl                                                         |
+-----------------------------------------------------------------+
| https://rss.firesky.tv?filter=from%3Aacarvin.bsky.social        |
| https://rss.firesky.tv?filter=from%3Aalexisohanian.bsky.social  |
| https://rss.firesky.tv?filter=from%3Aandrewshell.bsky.social    |
| https://rss.firesky.tv?filter=from%3Aandy.baio.net              |
| https://rss.firesky.tv?filter=from%3Aarchive.org                |
| https://rss.firesky.tv?filter=from%3Aarctictony.bsky.social     |
| https://rss.firesky.tv?filter=from%3Aatproto.com                |
| https://rss.firesky.tv?filter=from%3Abaratunde.com              |
| https://rss.firesky.tv?filter=from%3Abradneuberg.bsky.social    |
| https://rss.firesky.tv?filter=from%3Abrentsimmons.bsky.social   |
| https://rss.firesky.tv?filter=from%3Abrewster.kahle.org         |
| https://rss.firesky.tv?filter=from%3Abrianstelter.bsky.social   |
| https://rss.firesky.tv?filter=from%3Absky.app                   |
| https://rss.firesky.tv?filter=from%3Acharlesfitz.bsky.social    |
| https://rss.firesky.tv?filter=from%3Acharliesavage.bsky.social  |
| https://rss.firesky.tv?filter=from%3Adanielpunkass.punkitup.com |
| https://rss.firesky.tv?filter=from%3Adavidsirota.com            |
| https://rss.firesky.tv?filter=from%3Adavisshaver.bsky.social    |
| https://rss.firesky.tv?filter=from%3Adwillner.bsky.social       |
| https://rss.firesky.tv?filter=from%3Aeli.bsky.social            |
| https://rss.firesky.tv?filter=from%3Aemilynussbaum.bsky.social  |
| https://rss.firesky.tv?filter=from%3Aespiers.bsky.social        |
| https://rss.firesky.tv?filter=from%3Agartenberg.bsky.social     |
| https://rss.firesky.tv?filter=from%3Agreatdismal.bsky.social    |
| https://rss.firesky.tv?filter=from%3Ahrheingold.bsky.social     |
| https://rss.firesky.tv?filter=from%3Ainafried.bsky.social       |
| https://rss.firesky.tv?filter=from%3Ajayrosen.bsky.social       |
| https://rss.firesky.tv?filter=from%3Ajeffsandquist.com          |
| https://rss.firesky.tv?filter=from%3Ajoshuabenton.com           |
| https://rss.firesky.tv?filter=from%3Ajuliaangwin.com            |
| https://rss.firesky.tv?filter=from%3Alaurenweinstein.org        |
| https://rss.firesky.tv?filter=from%3Aleolaporte.me              |
| https://rss.firesky.tv?filter=from%3Alillywachowski.bsky.social |
| https://rss.firesky.tv?filter=from%3Alisalaporte.bsky.social    |
| https://rss.firesky.tv?filter=from%3Ama.tt                      |
| https://rss.firesky.tv?filter=from%3Amathewingram.com           |
| https://rss.firesky.tv?filter=from%3Ameganmccarthy.bsky.social  |
| https://rss.firesky.tv?filter=from%3Amigueldeicaza.bsky.social  |
| https://rss.firesky.tv?filter=from%3Amikedobuski.bsky.social    |
| https://rss.firesky.tv?filter=from%3Amims.bsky.social           |
| https://rss.firesky.tv?filter=from%3Ammccue.bsky.social         |
| https://rss.firesky.tv?filter=from%3Amoonalice.bsky.social      |
| https://rss.firesky.tv?filter=from%3Aniemanlab.org              |
| https://rss.firesky.tv?filter=from%3Anpr.org                    |
| https://rss.firesky.tv?filter=from%3Aomm.bsky.social            |
| https://rss.firesky.tv?filter=from%3Apierre.bsky.social         |
| https://rss.firesky.tv?filter=from%3Apilhofer.bsky.social       |
| https://rss.firesky.tv?filter=from%3Aroj.as                     |
| https://rss.firesky.tv?filter=from%3Astephenking.bsky.social    |
| https://rss.firesky.tv?filter=from%3Astevenlevy.bsky.social     |
| https://rss.firesky.tv?filter=from%3Asupport.bsky.team          |
| https://rss.firesky.tv?filter=from%3Athenation.com              |
| https://rss.firesky.tv?filter=from%3Atheonion.com               |
| https://rss.firesky.tv?filter=from%3Awaltmossberg.bsky.social   |
| https://rss.firesky.tv?filter=from%3Awashingtonpost.com         |
| https://rss.firesky.tv?filter=from%3Awindley.com                |
| https://rss.firesky.tv?filter=from%3Azephoria.bsky.social       |
+-----------------------------------------------------------------+

[johnspurlock]

Good morning! That sounds like a fix on your side? once you start parsing the cloud element even on empty feeds, then your feedland software will start subscribing on its interval, no?

my index right now only goes back to around yesterday midday, so those folks havent posted since then. Just checked washingonpost.com for example and their last post was around 20 hrs ago.

Gonna look into doing a batch process to index older items prior to when I added the index yesterday.

[scripting]

It is a fix on my side, which I have just made. ;-)

So the net-effect is that these feeds will eventually be read anyway, we read all feeds based on time even if they support rssCloud, and we'll pick up the cloud element at that time if everything works as it should, and this will all sort itself out.

I love fixing bugs like this.

[scripting]

Also it would be helpful if you set up an account on blue.feedland.org and had it load your OPML file.

I have a plan for how I want to approach this and it'll be a while before other people will be doing this, because Bluesky membership is by invite only now. Having another user there allows me to test a bunch of stuff I can't otherwise test.

The command to subscribe to a list is in the first menu. Here's a screen shot:

[scripting]

Also, @andrewshell if you're still tuned in, you'd be welcome as well. ;-)

I got in through Andrew's invite. So he's my Bluesky uncle or something like that.

[scripting]

Here's a mailbox view of my recent posts to Bluesky.

[johnspurlock]

Nice, btw just reindexed everything back to 5/13, which is as far back as I can go (that's when I started saving off every firehose post for the firesky.tv lookback when filtering

This should mean more items for feeds that were previously empty: https://firesky.tv/?filter=from%3Awashingtonpost.com

[johnspurlock]

Also, yesterday I put a discovery link from user-focused rss feeds to the corresponding opml subscription list for their follows.

[scripting]

this is part of a bigger question, i want more info like this, and i want to document it, and not just throw it out there in an atom:link element. we're really getting into what the user's profile is. there's more info available about the user, and that should all be part of an object with a clearer name for what it is.

here's a screen shot of a user's profile, review what's that there is not in the feed for the person, and then let's think.

[scripting]

we should think about what should be in the channel-level <title> and <description> elements, for example.

maybe "bluesky" doesn't enter into it. this is john gruber's feed, happens to be using bluesky to write it. that isn't the most important thing about it.

[scripting]

the big picture here is to bring the values of podcasting to text. that's what i want to accomplish.

[scripting]

something i threw together quickly to see what a rendering of a feed might look like in this context.

http://scripting.com/code/testing/blueskyreader/

[scripting]

added a link to the feed.