2 Phases commit transactions

[tuan08]

• Implement 2 phases commit for hdfs sink and source
• Implement 2 phases commit for kafka sink and source base on kafka low level api https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka#TransactionalMessaginginKafka-ProducerIDsandstategroups

Reliability requirements

• Implement a check sum algorithms to make sure that no messages loss or duplicate. It is good if the algorithm can tell if the error is message loss or duplica
• A transaction involve 3 steps:
  1. buffer(append/consume)
  2. preCommit
  3. postCommit
     • Create a simulation environment to test the failure at before, middle, after each step.
     • The failure simulation has to similar to jvm crash or server crash.
• If the stream allow only one writer(only one process can write at a time to a stream or partition), create a writer that write concurrently to the stream to t

Performancel requirements:

• Analyze the cost of append, commit, count the io operations.
• Measure the performance in record/s , MB/s with the different record size, buffer size.

[tuan08]

A sample or performance requirement taken from the link https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka#TransactionalMessaginginKafka-ProducerIDsandstategroups

Furthermore, we have the following requirements pertaining to performance, usability and implementation complexity: The implementation should be scalable. E.g., a dedicated log per transaction is unacceptable. Performance: The throughput of a transactional producer should be comparable to that of a non-transactional producer. Acceptable latency. E.g., avoid copying the transactional data as much as possible. Any implementation should not make the partition unavailable (say, due to locking) for an unreasonable period of time. Client simplicity: Favor a scheme that lend to a simpler client-side implementation (even if it adds more complexity to the broker). For example, it is acceptable (but not ideal) for a consumer implementation to (internally) buffer and subsequently discard messages from uncommitted transactions. i.e., if the chosen implementation allows the broker to materialize messages from uncommitted transactions in the data logs.