add to a set/seq defined with filter

[dominictarr]

hey, so I have just realized that using a function to define a set/seq breaks the add, push, unshift, insert operations.

If you use a key:value to define the set, then when you do a seq.push(row) it sets the correct key property as well, so it's now a proper part of the set - this doesn't work with filter though.

I briefly considered passing a function that mutates the row so that it does match the filter - but then I realized that is ugly... probably the cleanest is to make the filter based set a separate thing.

@Raynos what do you think?

[Raynos]

@dominictarr I never use the "push something into a set" feature. I only use doc.add and treat the doc as an append only thing. I use sets and seq's for reading and use rows for mutation when I'm being a lazy shit.

That being set yes filter function's break this behaviour. It can either be documented as it simply doesnt work or filters can be their own thing.

[dominictarr]

being it's own thing is way cleaner than documenting an edge-case, so...

var filter = doc.createFilter(function (row) { return true })

with none of the set/seq mutation methods.

[Raynos]

@dominictarr I use filterable sequences

[dominictarr]

ah, so you are reordering them, but using a filter not a key:value? please explain

[Raynos]

@dominictarr I have a seq ordered by insertion time. But I want a custom filter.

It's the list of chats that are ordered by timestamp. But to identify chats is a custom filter.

[dominictarr]

right, so what you want is a custom filter, and a custom sort function!

[dominictarr]

it sounds like maybe you are keeping information for more than one chat in a single crdt? are you sure crdt is right for this? what about using append-only for each room?

[Raynos]

@dominictarr right now I have all my state in crdt for convenience. I really need to break it out into multiple scuttlebutts.

But even append-only won't allow me to render items in order when network partitions. I would have to rewrite seq

[dominictarr]

right, crdt was the second replication structure I wrote after snob I decided on the API before I refactored out scuttlebutt. It even predates mux-demux.

Basically, crdt is OLD, and we've learnt a lot since then, rewriting the basic ideas into something similar but better has been on my want-to-do list for a while now...

[juliangruber]

whats the best way to do lists of lists? a doc pointing to several docs?

[dominictarr]

This might sound disturbing, but CRDT is basically like SQL. You do lists with foreign keys. that is how seq/set works. You could also add many-many with a join table pattern... or have some way to give a row a parent_id, which points and the id of another row. then, maybe you could have something like row.children() which returned a Set of rows where parent_id == this.id

Or stuff like that! will merge if you make a PR.

[juliangruber]

yeah, I figured that out too. Just didn't feel like saying out loud :D

So, if we want something that feels more like NoSQL, a distributed leveldb is the way to go, I guess? I want live map reduces and stuff that scales.

[juliangruber]

using crdt feels like a step back and too high level...

[dominictarr]

SQL sucks, but "relational" is a pure abstraction. data still has relations whether it's in a SQL db or not.

I have a few ideas for handling relations in leveldb - also, have a look at this https://github.com/mmckegg/level-match-index @mmckegg wrote that to replace mysql...

[dominictarr]

Also, we need an new indexeddb wrapper now that levelup has leveldown split out, and injectable (in 0.7)

[nrw]

Just ran into this, too. Though, I don't have anything to add, yet.

[tristanz]

It would be nice to be able to create a filtered seq, but just with something simple like

var range = seq.startAt(start).endAt(end).limit(n);

where you can specify any two. This is the way Firebase does limits. A typical use case is a chat room where you only want to see that last 100 messages.

https://www.firebase.com/docs/javascript/firebase/startat.html

[dominictarr]

well, a crdt set is a subset of all the rows, and a seq is a set with an ordering. you could add a limit, but it would still replicate all the data, so in a case like your chatroom you could do it, but it wouldn't save you any bandwidth (if that was the reason)

There are other things you could do though, to forget old records, but you'd have to get into the guts of scuttlebutt.

I'd merge a pull request for ranges like this: doc.range({start: s, end: e}) instead of chaining. that would work like leveldb ranges... you could use this module: https://github.com/dominictarr/ltgt