create sublevels from client

[juliangruber]

Currently clients cannot create sublevels that aren't part of the manifest. I find that to be cumbersome as I have to recreate manifests etc. for every new sublevel I want to use.

Of course clients shouldn't be able to go above their current sublevel in their hirarchy, but in the sublevel they're in right now they should be able to do whatever they want.

@dominictarr

[juliangruber]

@dominictarr I need your feedback on this one

[maiah]

I agree on this. Creating sublevels in the client will open many doors in the client API development.

[juliangruber]

it's not like I yet had a use cases where I couldn't do something because of this behavior, it's more like it's just annoying to keep updating manifest files al the time.

[juliangruber]

yup I'll do this

[dominictarr]

I suspect that this is related https://github.com/dominictarr/level-sublevel/issues/19

[timoxley]

:+1:

[dominictarr]

you would need this: https://github.com/dominictarr/level-sublevel/issues/38 before creating sublevels from multilevel should be considered.

[timoxley]

@dominictarr if you create a sublevel on the client, does the server even need to know about it, and vice versa? i.e. if they could be isolated locally, then any 'leak' would be contained.

[dominictarr]

I think if you are creating sublevels dynamically then we are better off with a different solution. can you describe your usecase? and what sort of operations you perform on the data?

[timoxley]

@dominictarr I'd like to get a pattern working where I can get a key, and the value is a stream.

Ignoring existing apis, it might look something like:

db.get('key').pipe(out)

but an approximation of this using sublevels works:

db.sublevel('key').createReadStream().pipe(out)

The problem is you can't use this pattern with multilevel if you don't know the names of the sublevels when you generate the manifest.

Usecase:

I have directories /keyed/by/paths/ which contain files, which have meta info. User supplies the directory on the clientside. I look up content of directory on some remote machine. This is slow. While I'm looking up the content, I show whatever data I already have. When new data arrives, it automatically updates the clientside view.

My original implementation has a single sublevel for all the data. The directory is a key, its files + metadata as the value.

My new thought is to use a sublevel for the directory, e.g. db.sublevel('/user/supplied/data') and the sublevel would contain the filenames as keys with associated metadata as values.

Goal is to live-stream the sublevel, getting an up-to-date stream of the files + meatdata contained in the directory.

[dominictarr]

ahah, right.

I see this is related to your the other issues you have posted on resuming?

I gotta go eat right now, find me on irc later and we should discuss this

[dominictarr]

okay, I've changed my mind about this. I woke up with an idea about this on the train to nanning.

[juliangruber]

mind to share your discoveries?

[dominictarr]

I think it came down to handling the prefixes differently, so you could just make the key an array, and that will write to sublevels. that is discussed in the linked issue. That pot is still on the stove, there is some code there though.