Adding Slice interface?

[ffernand]

Do you plan on exposing the slice interface in leveldb?

The reason I ask is that I plan on storing large values and would prefer to stream them in and out of a key instead of having the interpreter allocate large chunks of memory.

BTW... thanks for an excellent library!

[wbolster]

No, I didn't have this planned. But maybe now I do. :-)

For storing values I can imagine Plyvel accepting buffer() instances instead of byte strings. For obtaining values I'm not sure what the API should look like. I'd welcome a discussion.

Thanks for letting me know you enjoy Plyvel.

— Wouter

(Sent from my phone. Please ignore the typos.)

Filipe Fernandes notifications@github.com schreef:

Do you plan on exposing the slice interface in leveldb?

The reason I ask is that I plan on storing large values and would prefer to stream them in and out of a key instead of having the interpreter allocate large chunks of memory.

BTW... thanks for an excellent library!

---

Reply to this email directly or view it on GitHub: https://github.com/wbolster/plyvel/issues/25

[ffernand]

Hmm... I guess it could be as simple as providing a generator where the DB.get() method call accepts a chunk size.

import plyvel
db = plyvel.DB('/tmp/testdb')

# some work done here

with open('filename.txt', 'w') as fout:
    for chunk in db.get('some_key', chunk_size=8192)
        fout.write(chunk)

[wbolster]

@ffernand, I don't see how that could possibly work. LevelDB does not provide streaming access to the data stored inside a database, and neither does it allow storing values by providing chunks. The Slice interface is used at the API boundary so that applications can give LevelDB access to application-owned memory, and also the other way around.

[ffernand]

Apologies... I mis-understood the role of the Slice interface. Thinking that someone must have wanted this in the past, I found the following thread...

https://code.google.com/p/leveldb/issues/detail?id=89

Even the WriteBatch construct which describes all data being in memory (I hope I got that right) is an eye opener for me. Clearly, in some ways, I'm not using it right. But the ideas suggested by streaming in and out to a file as a blob with SHA1's will work for me.

Thanks for looking into this! I've never had opportunity to try out cython, but this project is a great launchpad!

[wbolster]

Yes, write batches keep everything in memory until they are written to disk.

And glad to hear you enjoy Plyvel. Cython is a great tool indeed. (This is my first Cython project ever.)