[WIP] Storm v4

[asdine]

After v3, I realized Storm was becoming hard to maintain. It was relying too much on reflection, felt bloated, and was performing poorly. There were a lot of ongoing discussions on the subject, like https://github.com/asdine/storm/issues/211 or https://github.com/asdine/storm/issues/222, that led me to realize what was the actual problem: Storm design was good enough for a simple toolkit but not for an actual database. If fixing some of the performance issues can be done by tweaking a loop there and there, any other addition was a real pain and my motivation wasn't high enough to handle that level of plumbing. Instead, I decided to write an actual database as a separate project, Genji, with the goal of, someday, building Storm on top of it. However, Genji's API is still not yet stable and requires a little bit more work before being able to be relied upon. But as a first step, here is a draft of what it would look like.

Another aspect I wanted to fix was the Storm API. I believe the current API is error-prone and requires too much magic, as we can see in these issues. I think this is the opportunity to improve that design as well.

Storm now uses Genji. What does it mean?

Genji is a document-oriented, embedded SQL database written in Go.

• It uses BoltDB as a storage engine by default: Nothing changes for Storm, it's still a toolkit for BoltDB. What changes though, is how that data is encoded, stored, and queried.

• It supports other backends: This means that Storm can also be used to store data in-memory or in Badger. The Badger engine is still experimental though, but this is possible in the future.

• It supports SQL: This means that if users need advanced queries, they can use SQL directly.

• It's a document database: This means that we can now query nested fields as well

• It supports schemas: This means that we can now apply constraints on some of the fields of the table/bucket

• It is designed for performance: Even though Genji is still in an early phase and lacks optimizations, it was designed to scale and it's much easier to apply changes there.

What are the big changes in the API?

Take a look at the README file.

How can I contribute?

Inputs are welcome! Don't hesitate to comment on the PR or creating an issue.

Next steps

• [ ] Iterate on this PR until having a working version
• [ ] Merge in a dedicated branch
• [ ] Release one or several betas
• [ ] Release version 4

[codecov-commenter]

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@23213e9). Click here to learn what that means. The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #290   +/-   ##
=========================================
  Coverage          ?   33.33%           
=========================================
  Files             ?        2           
  Lines             ?      132           
  Branches          ?        0           
=========================================
  Hits              ?       44           
  Misses            ?       75           
  Partials          ?       13           

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 23213e9...8678f8c. Read the comment docs.