Open dominictarr opened 7 years ago
Hey @dominictarr o/
No, I haven't. I got jazzed about a nice interface for append-only logs after squatconf and chatting with @mafintosh, but only got as far as stubbing out the interface -- there aren't even real tests yet! I put this on hold until I found a personal use-case.
It'd be great if flumelog
could be the first concrete implementation! Since nobody is using this yet (afaik) I'd be happy to tweak the api to meet flumelog's needs a bit better. You know I dig pull streams. :grinning:
yeah, it's basically the same idea, but flumelog has a few extra requirements. I have the startings of a reusable test suite over here https://github.com/flumedb/test-flumelog
@dominictarr, @noffle
Darn it. Just discovered this. I've started a similar project. Have a few concrete implementations (knex
for postgres, mysql, sqlite) and will be building out a few more.
Similar API though. But I've gone for the log offset to be any javascript object to allow for a few more options (eg. indexing into a file with not just a byte offset, but also the payload length).
See: https://github.com/eugeneware/abstract-log (has some links to some basic implementations) on the README.
@eugeneware at first I considered having a key that combined offset+length, but I realized I had tradeoff about size of the key - like, should it be an 64 bit int, divided into a offset in, say 6 bytes, and a length in 2 bytes, but then records could only be 65k, so maybe it should be 5 bytes + 3 and so on, but then I realized I'll want a frame around each write anyway, so the length doesn't need to be in the key at all, and we can then get very small primary key without a tradeoff.
btw, flume is doing very well, and we are using it in secure-scuttlebutt. Just merged it into the master branch today!!!
this is very nearly the same api as flumelog (except pull-streams, a few more options on the stream, and
since
observable)I was wondering if you made any implementations of this api (which we could probably adapt to flume)