infoforcefeed / OlegDB

Enough works to use this in production
http://olegdb.org
MIT License
132 stars 14 forks source link

Bulk Get #148

Closed qpfiffer closed 9 years ago

qpfiffer commented 9 years ago

Needs to happen. Round-tripping to get a bunch of keys is terrible.

Hamcha commented 9 years ago

I can do it if you don't want to bother with go.

qpfiffer commented 9 years ago

@Hamcha Doesn't matter to me, I'll probably knock it out by this weekend. It'll be a good way to test larger transactions.

qpfiffer commented 9 years ago

Spitablling:

POSTing newline delimited keys to the /<database>/_bulk_unjar endpoint should result in a series of alternating size_t (unsigned 32-bit integer) and unsigned char * streams. The first size_t is the size of the stream following.

The rationale for the return type is that I want to preserve the whole keyset. I figure newline delimited keys in the POST data is fine because why would you put newlines in your keys? I'll think on it some more or change it if I find something dumb later.

Hamcha commented 9 years ago

I agree, if you put \ns in your keys the front end would have tons of other issues anyway. Sadly, size_t is architecture-dependent, we need to use a fixed size.. How about int 64 (since x64 is the main target and you'd have to send what Google has in its data centers several times to overflow it)

-----Original Message----- From: "Quinlan P." notifications@github.com Sent: ‎30/‎01/‎2015 06:01 To: "infoforcefeed/OlegDB" OlegDB@noreply.github.com Cc: "‮ahcmaH" zikyky@gmail.com Subject: Re: [OlegDB] Bulk Get (#148)

Spitablling: POSTing newline delimited keys to the //_bulk_get endpoint should result in a series of alternating size_t (unsigned 32-bit integer) and unsigned char * streams. The first size_t is the size of the stream following. The rationale for the return type is that I want to preserve the whole keyset. I figure newline delimited keys in the POST data is fine because why would you put newlines in your keys? I'll think on it some more or change it if I find something dumb later. — Reply to this email directly or view it on GitHub.=

qpfiffer commented 9 years ago

@Hamcha Definitely, I was thinking a uint64_t would do just fine.

qpfiffer commented 9 years ago

This is done until someone tries to use it for binary data.