Processing large transactions/Streams

[simerplaha]

Problem

There are many situations that require processing large transactions. For example:

• Computing two or more large SQL tables and inserting the results into another table. In SwayDB's case this would be processing multiple Streams of data from multiple Maps, Sets or Queues.
• Streaming, processing or copying data from multiple remote servers concurrently and storing locally.
• Performing arithmetic operations of two large matrices.
• Personally I’ve had to merge multiple Derby and old an MySQL database into a single Postgres instance. In SwayDB this would be merge multiple db instances into a single instance.
• Etc ...

The following requirements would ensure that we can process large Streams of data with a simple API while being memory efficient.

Requirements

• We should be able to define our intensions as a Stream and submit an unmaterialised (unprocessed) Stream to SwayDB for processing. Currently materialised (processed) Streams are required - See.
• Memory efficiency - The Stream could be very large (in GBs) so it should process data as it is available.
• Atomicity on entire Stream - Similar to existing atomic transactions API we need to provide atomic guarantees on large Stream and also provide failure recovery.
• The Stream processing should maintain configured back-pressure (Throttle) so that all reads, writes & compaction maintain a smooth flow of operation.

Expected work

• We need the ability to dynamically adjust Level hierarchy with temporary Levels where each Level could have partial or completed Streams. This implementation could also be used to provide ACID transactions.
• We should also support external Streaming libraries as they have richer APIs - #173
• Compaction process should also check for data in these temporary Levels while maintain back-pressure.

If you have a different opinion on this or any suggestions, please comment!

[simerplaha]

We need to write down some actual use-cases so that requirements are clear:

Use-case 1 - Applying a function on multiple streams

If there are two Maps A & B and we want to perform A + B = C where C is a new Map. If A and B are continuous stream of data, we can configure the + operation (or any other function) such that

• data is committed every few intervals and every committed interval is visible to reads.
• data is committed every few intervals but committed data is not visible until the full stream is processed.
• data is not committed until the entire stream is complete.
• what others?

Interval - can be specified in seconds or in bytes.