search

hackergrrl / append-only-log

:train: :train: :train: Test suite & interface for append-only log modules.
3 stars 0 forks source link

did you create any concrete implementations of this? #1

Open dominictarr opened 7 years ago

dominictarr commented 7 years ago

this is very nearly the same api as flumelog (except pull-streams, a few more options on the stream, and since observable)

I was wondering if you made any implementations of this api (which we could probably adapt to flume)

hackergrrl commented 7 years ago

Hey @dominictarr o/

No, I haven't. I got jazzed about a nice interface for append-only logs after squatconf and chatting with @mafintosh, but only got as far as stubbing out the interface -- there aren't even real tests yet! I put this on hold until I found a personal use-case.

It'd be great if flumelog could be the first concrete implementation! Since nobody is using this yet (afaik) I'd be happy to tweak the api to meet flumelog's needs a bit better. You know I dig pull streams. :grinning:

dominictarr commented 7 years ago

yeah, it's basically the same idea, but flumelog has a few extra requirements. I have the startings of a reusable test suite over here https://github.com/flumedb/test-flumelog

eugeneware commented 7 years ago

@dominictarr, @noffle

Darn it. Just discovered this. I've started a similar project. Have a few concrete implementations (knex for postgres, mysql, sqlite) and will be building out a few more.

Similar API though. But I've gone for the log offset to be any javascript object to allow for a few more options (eg. indexing into a file with not just a byte offset, but also the payload length).

See: https://github.com/eugeneware/abstract-log (has some links to some basic implementations) on the README.

dominictarr commented 7 years ago

@eugeneware at first I considered having a key that combined offset+length, but I realized I had tradeoff about size of the key - like, should it be an 64 bit int, divided into a offset in, say 6 bytes, and a length in 2 bytes, but then records could only be 65k, so maybe it should be 5 bytes + 3 and so on, but then I realized I'll want a frame around each write anyway, so the length doesn't need to be in the key at all, and we can then get very small primary key without a tradeoff.

dominictarr commented 7 years ago

btw, flume is doing very well, and we are using it in secure-scuttlebutt. Just merged it into the master branch today!!!