search

ericvolp12 / jetstream

A simplified JSON event stream for AT Proto
MIT License
86 stars 3 forks source link

JetStream Future #1

Closed shawnphoffman closed 5 months ago

shawnphoffman commented 5 months ago

Love the project! I was wondering if you had any future plans on making this consumable by third-party projects? Or, is it best to fork and host it separately?

Given the current state of the firehose and the increased bandwidth requirements, I feel that something like this is going to super nice for smaller projects to get off the ground without a hefty networking bill.

Thanks again!

ericvolp12 commented 5 months ago

I haven't really poked at this project in a little while, I was thinking of doing a revamp of it that just converts the firehose into a stream of raw record JSON (still allowing you to specify repos and/or collections you care about) but without the whole "trying to keep track of likes" and such bit.

ericvolp12 commented 5 months ago

Okay I just pushed a pretty big update that gets rid of the BadgerDB bits and effectively just converts the CBOR repository stream into a JSON or CBOR stream of records and/or deletion operations. It should be significantly lighter to consume and allow you to filter for collections and repos you care about.

The uncompressed JSON stream right now is ~200kbps down from the ~2.25mbps of the production repo stream with the full MST blocks overhead.

ericvolp12 commented 5 months ago

As for consumption, my recommendation for now would be to host it yourself. I don't actually use it in any of my projects because I wanted them to be plug-and-play with the production firehose to serve as examples for how to interact with the prod firehose.

Maybe I'll update some of them though, this newer lightweight Jetstream should be flexible enough for use with my projects...

shawnphoffman commented 5 months ago

Awesome! Thanks for turning this around so quickly! 🎉