A thank you (and have you considered using JSDB?)

[aral]

Hey Ben, just stumbled on this today and I wanted to both say thank-you for working on a Small Web-style single-person implementation and also to ask if you’ve heard of JSDB (JavaScript Database):

https://codeberg.org/small-tech/jsdb

If not, take a look (it might help with the persistence bits. (Although it is also in-memory and I’d be interested in seeing what sort of sharding you come up for that.)

Best of luck with this. Looking forward to seeing how it evolves :)

[benbrown]

Awesome, thanks for the pointer. I’ll take a look. and thanks for the kind words!

Part of the important part for me is that the data on disk is easy to look at and manage - like I can delete old posts by just deleting folders. Does this package write files in a similar way?

[aral]

Part of the important part for me is that the data on disk is easy to look at and manage - like I can delete old posts by just deleting folders.

I’m a big fan of this too :)

JSDB writes to append-only JavaScript logs so it’s not as great for the design properties you mentioned.

Interestingly, with Heartbeat (about ~24m in at https://www.youtube.com/watch?v=kbfvEBfHO-M; source: https://source.small-tech.org/project/heartbeat-node, https://source.small-tech.org/project/heartbeat-cocoa), I was writing everything to the file system for the same reasons as you and basically using Syncthing to replicate changes to all peers (and using a LevelDB database if I remember correctly to weave all that into timelines). There are lots of advantages to using the file system. (It might be interesting to see whether peer-to-peer sync might be an interesting extension for Shuttlecraft.)

JSDB might be more interesting if you need a key/value store, etc. Apart from settings, etc., I’m going to experiment with using it to store messages (at which point I’ll need to look into some sort of lazy-loading / sharding). Each table is already saved to a different file but they’re all greedily loaded when you open a database (for ease of use).

[benbrown]

Thanks for all the pointers. I appreciate it.

My personal instance now has 11,000 posts in the index and ... so far is doing ok performance wise. But in a few months it will easily hit 100,000 posts.

In particular I think a JSDB that is quick to query would be very useful for the index itself, which just needs to know metadata and not full content of the posts.

[aral]

Oh yeah, it’d be great for that :)

[benbrown]

Scenarios are things like:

• get most recent posts
• get most recent posts by a user
• traverse a thread

[aral]

Take a look at https://codeberg.org/small-tech/jsdb#javascript-query-language-jsql

Basically, it’s all in-memory so it should be more than fast enough. And hopefully the interface is fluent (and if you do try it out and run into any issues, please do let me know) :)

[benbrown]

Ah, looks great!