search

logv / sybil

columnar storage + NoSQL OLAP engine | https://logv.org
https://logv.org
Other
305 stars 24 forks source link

Pull Column & Row Store into their own packages #7

Closed okayzed closed 7 years ago

okayzed commented 7 years ago

Try to minimize changes to API, but some will be necessary.

It seems like we need Records to be shared between the aggregator and storage engines, so the Record (and other shared data structures) might need to be in their own separate package.

seiflotfy commented 7 years ago

can you assign this to me?

okayzed commented 7 years ago

i was able to separate many structs out and put all functions in the global namespace, but still have trouble going forward (and moving functions into separate modules).

https://github.com/logv/sybil/blob/ef0ef3f8686465c90bc102c89331027a825769c3/src/lib/structs/structs.go is what the public struct layout is.

this re-factor is very busy work (shuffling code around and making sure things compile), it took me 5 - 10 hours to get to the point where structs were all pulled out. i just spent another 2 - 3 hours this AM trying to shuffle functions, but still no compile.

seiflotfy commented 7 years ago

can we define the interface for column? This would allow us to test with different implementations

okayzed commented 7 years ago

there is no interface for column yet. the responsibility of a RB is to return a RecordList that matches the given loadSpec. it will return the data out of the relevant RBC for that block.

in the future, there will be an API to ask for individual columns as a series of arrays (through another loadspec) that the aggregator will use for vectorized processing

okayzed commented 7 years ago

https://github.com/logv/sybil/tree/remove_methods_refactor/src/lib is where this branch currently is. looking forward to it getting very cleaned up.

okayzed commented 7 years ago

sticky spots: