Interview with Noria’s creator: a promising dataflow research database implemented in Rust

[Sunt-ing]

https://www.notamonadtutorial.com/interview-with-norias-creator-a-promising-dataflow-database-implemented-in-rust/

[Sunt-ing]

How does Noria compare to other software like a relational database, in memory key-value stores, MapReduce systems like Spark, stream processing like Storm or timely data flow?

• Users of Noria can change the running dataflow at any time without downtime. If a new SQL query is issued that the system has not seen before, it adapts the running dataflow on the fly to incorporate the operators from the new query. The adaptation also knows to re-use existing operators where possible to produce an overall more efficient dataflow than what you would get if you just ran each query as its own dataflow program.
• Noria supports partial materialization.
  • Noria has a lot of smarts around automatically materializing only the subset of the view that is actually accessed, while presently Materialize requires that you explicitly declare what subsets to materialize.
  • Existing dataflow systems that support materialization are usually either:
  • Windowed (i.e., they only reflect “recent” updates): cannot get all the results for a query.
  • Fully materialized (i.e., all results are always stored and maintained): waste resources on results that your application does not care around. You need the ability to evict from your cache.
• Noria can translate SQL queries into efficient dataflow and adapts the running system to them. The other systems do not generally support this.

The Noria paper states that it can scale to 5x higher load than a hand optimised MySQL database. How does it manage to do that?

• The primary reason for this is Noria’s view materialization. When you issue a query to MySQL, the system has to execute that query to produce the query’s results. In Noria on the other hand, an application query effectively turns into a hashmap lookup in the common case, which is very fast.
• Noria’s reads have very little synchronization overhead, whereas MySQL includes a lot of machinery to support full-fledged transactions (which Noria does not support). Part of the trick here is Noria’s use of a neat little datastructure called an evmap (for “eventual map”; named for its support for eventual consistency). It allows reads and writes to proceed entirely in parallel with very little overhead by having reads and writes go to different hashmaps, and then occasionally atomically switching between them.

What is the relationship between Noria’s paper, the Differential dataflow paper and Naiad: a timely dataflow system?

• Differential dataflow, timely dataflow, and its implementation in Naiad is one of the hallmark systems in the dataflow community. Most papers in this area relate to timely dataflow in one way or another.
• Noria implements an incremental dataflow model, but trys to solve for a different use-case, and therefore arrives at different solutions.
  • Timely established a model for incrementally executing dataflow programs that include iteration and cycles with strong guarantees about consistency through a sophisticated timestamp tracking scheme. Noria does not support iteration or cycles, and is eventually consistent.
  • Timely largely targets arbitrary batch computations over a large, interconnected dataset where reads are less frequent than just observing the “output” of the computation. Noria targets read-heavy applications where repeated and similar queries are common, and where the queries change over time.

Missing features of Noria

• Engineering things Noria doesn’t support
  • Joins whose join condition is not a single column equality; ORDER BY without a limit; CASE statements; LIKE conditions; and of course the SOUNDEX operator. There are also patterns that we support, but that we believe could be optimized further, such as multi-way joins and multiple aggregations in a single query.
• Production-ready features that Noria should have
  • Checkpointing of materialized views. Currently, if the system is turned off and then restarted, all the materialized views are empty. This would be equivalent to a full cache purge. An external system could heat the cache, but it’d be better if Noria internally kept some form of snapshot to aid in this process.
  • Entirely-restarted Fault tolerance. When a machine failure results in related parts of the dataflow being blown away entirely restarted.
  • Noria currently requires ownership of its base storage. If you have an existing data store that you’d like to keep using, you’d have to feed changes to that data to Noria, and Noria would store it a second time. Need engineering efforts with respect to consistency of upqueries.

When would you avoid using Noria?

• Have no well-defined working set. Constantly issuing entirely new queries make materializations useless and add unnecessary overhead.
• Noria will work fine within a fairly high write throughput rather than extremely high write thoughput applications.

Interesting things

• Materialize employee said that:
  • "Noria has a lot of smarts around automatically materializing only the subset of the view that is actually accessed, while presently Materialize requires that you explicitly declare what subsets to materialize. We have some plans for how to bring these smarts to Materialize, but we haven't implemented them yet."
  • "Differential dataflow is consistency preserving—if your source of truth provides strict serializability, differential can also provide strict serializability—while Noria provides only eventual consistency."