infoforcefeed / OlegDB

Enough works to use this in production
http://olegdb.org
MIT License
132 stars 14 forks source link

Proposal: OTERCAL for querying data from OlegDB #160

Open Xe opened 9 years ago

Xe commented 9 years ago

We need a hero. OlegDB is that hero.

I propose we have a new OlegDB Query Language (OTERCAL for obvious reasons which are an exercise best left to the reader) for better querying the key-value pairs we all know and love.

This specification's use of the keywords "must", "must not", "required", "shall", "shall not", "should", "should not", "recommended", "not recommended", "may", and "optional". These are to be interpreted as described in RFC2119.

Basic Syntax

<VERB> [<key> FROM <table> [AS <value>]] :D

All statements must be terminated with the "endless happiness" operator :D.

Verbs

Verb Meaning Needs Arguments? == C API call
JAR Sets <key> to <table> with the contents of <value> Yes and Value ol_jar
UNJAR Gets the contents of <key> from <table> Yes ol_unjar
SNIFF Gets the expiration of <key> from <table> Yes ol_expiration_time
SCOOP Removes <key> from <table> Yes ol_scoop
UPTIME Returns the uptime of the database No ol_uptime
SPOIL Sets <key> in <table> to have an expiry date of <value> Yes and Value ol_spoil
SELECT Matches by prefix in <table> with <key> as the prefix to match against Yes ol_prefix_match
SELECT * Gets all keys in <table>. Yes ol_key_dump
INVENTORY Sees if <key> exists in <table> Yes ol_exists
SQUISH Squishes the AOL file No ol_squish
CRAZY Performs the Malbolge Crazy operation on <key> in <table> against <value> Yes and Value

This could make OlegDB the truly cloud scale performant monster we all know and love.

qpfiffer commented 9 years ago

Holy shit I love you On Mar 5, 2015 11:42 AM, "Christine Dodrill" notifications@github.com wrote:

We need a hero. OlegDB is that hero.

I propose we have a new OlegDB Query Language (OTERCAL for obvious reasons which are an exercise best left to the reader) for better querying the key-value pairs we all know and love.

This specification's use of the keywords "must", "must not", "required", "shall", "shall not", "should", "should not", "recommended", "not recommended", "may", and "optional". These are to be interpreted as described in RFC2119. Basic Syntax

FROM [AS ]:D All statements must be terminated with the "endless happiness" operator :D . Verbs Verb Meaning Needs Arguments? == C API call JAR Sets to
with the contents of Yes and Value ol_jar UNJAR Gets the contents of from
Yes ol_unjar SNIFF Gets the expiration of from
Yes ol_expiration_time SCOOP Removes from
Yes ol_scoop UPTIME Returns the uptime of the database No ol_uptime SPOIL Sets in
to have an expiry date of Yes and Value ol_spoil SELECT Matches by prefix in
with as the prefix to match against Yes ol_prefix_match SELECT \* Gets all keys in
. Yes ol_key_dump INVENTORY Sees if exists in
Yes ol_exists SQUISH Squishes the AOL file No ol_squish CRAZY Performs the Malbolge Crazy operation https://en.wikipedia.org/wiki/Malbolge#Crazy_operation on in
against Yes and Value This could make OlegDB the truly cloud scale performant monster we all know and love. — Reply to this email directly or view it on GitHub https://github.com/infoforcefeed/OlegDB/issues/160.
kyleterry commented 9 years ago

This is awesome

qpfiffer commented 9 years ago

@Xe Proposal for starting/aborting/commiting transactions?

Xe commented 9 years ago
Command Meaning Args?
PONDER Start a transaction No
GOAWAY Abort a transaction No
SHIPIT :shipit: No
Hamcha commented 9 years ago

What should the replies look like? Should we have a "endless despair" (:C or D:) operator for errors?

Also, the requirement for tables in queries might make it a pain to implement, since opening/closing of extra databases are then involved. Any thought on the matter?

What about cursor iteration?

dequis commented 9 years ago

I am not comfortable with the idea of endless despair in olegdb.

I already have enough of that IRL

Xe commented 9 years ago

PURCHASE <tabname> => change the active table?

qpfiffer commented 9 years ago

Okay so I really, really love this idea. This is also an excellent time to put some cool shit into the Go side of OlegDB. I really want this to get done, and I think a lot of people are intimidated by the C side of Oleg so this is probably a good idea and I think you've all had enough jnerula rando C pull requests, so heres my proposal:

  1. Query parser, optimizer and scheduler in the Golang portion of OlegDB
  2. I'll support and dev needed in the C side of things
  3. @Hamcha @Xe @kyleterry hash out and intelligent plan for the language parsing stuff (I'd love to get in on that as well)
  4. Code that shit up
Xe commented 9 years ago

@Hamcha doesn't the Go frontend keep the tables open until olegdb an heroes?

qpfiffer commented 9 years ago

Yeah, it keeps them open until the whole thing shuts down.

Hamcha commented 9 years ago

That was an open bug as well, we always had a need for a timeout or something.

Xe commented 9 years ago

Maybe this could be a set of commands:

Verb Action
PURCHASE Switches active database table
GTFO Closes active database table
kyleterry commented 9 years ago

@qpfiffer we can pair sometime soon and take a look at the Go AST. Go is written in Go so it will give us some good ideas for writing an AST for the query language.

qpfiffer commented 9 years ago

@kyleterry Calm down there ranchero let's just do it with regex, GRESHUNKEL style :six: :six: :six:

Hamcha commented 9 years ago

Regexp considered harmful, proper stateful parsing > *

On 7 March 2015 at 06:53, Quinlan P. notifications@github.com wrote:

@kyleterry https://github.com/kyleterry Calm down there ranchero let's just do it with regex, GRESHUNKEL style [image: :six:] [image: :six:] [image: :six:]

— Reply to this email directly or view it on GitHub https://github.com/infoforcefeed/OlegDB/issues/160#issuecomment-77675282 .

dequis commented 9 years ago

You can always use regex for the tokenization step, might even turn out to be saner and more flexible.

dequis commented 9 years ago

One of the examples says JAR 'Xena' AS [[{"password": "foobang"}]] :D, implying that the keys are quoted with single quotes (that's not defined very well), but what about the values?

As far as i know, with ol_jar(), both keys and values can be arbitrary binary data, they don't even need to be null-terminated strings

qpfiffer commented 9 years ago

@dequis The transition from datastore to database will be silent and slow. The double-quotes won't even see it coming.