Questions on batch operations

[abhideodhar]

I want to seek some clarity on the batch operations:

1) A connection can have a single batch running on it at a given time - right? Can operations other than PUT/DELETE outside the batch happen on that connection while the batch is active? Or is the server supposed to reject those commands? 2) In a batch, there could be a sequence of operations (puts/deletes). If we have a multi-threaded kinetic client which is using the same connection, then the different threads could send requests in an order. So lets say thread1 sends operations 1->5 and thread2 sends operations 6->10. These operations could arrive in a different order on the server due to the network layer. So server may receive 6->10 before 1->5. So where does the onus of ordering the request lie - I am assuming that the client needs to provide the necessary synchronization and the server simply processes whatever it receives at its end in that order. Is my understanding correct? 3) This question is about the END_BATCH_RESPONSE. If client sent 10 operations in a batch (assume 1->10) and during commit the 5th operation failed, the server will report 5 in the failedSequence. Now - I am thinking that its the client's responsibility to maintain atomicity by rolling back 1->4 or reissuing 5->10? Is my understanding correct?

[abhideodhar]

@chiaming2000 - hey can you please reply to my questions above?

thanks

[chiaming2000]

Hi,

Response in lines. Thanks.

On Sun, Jul 5, 2015 at 8:53 PM, abhideodhar notifications@github.com wrote:

I want to seek some clarity on the batch operations:

1) A connection can have a single batch running on it at a given time - right?

Yes. As long as the device is under an operational state and no limits are exceeded.

Can operations other than PUT/DELETE outside the batch happen on that connection while the batch is active?

Yes.

Or is the server supposed to reject those commands?

No. A device can accept concurrent operations for batch and non-batch operations within a single connection.

2) In a batch, there could be a sequence of operations (puts/deletes). If we have a multi-threaded kinetic client which is using the same connection, then the different threads could send requests in an order. So lets say thread1 sends operations 1->5 and thread2 sends operations 6->10. These operations could arrive in a different order on the server due to the network layer. So server may receive 6->10 before 1->5. So where does the onus of ordering the request lie - I am assuming that the client needs to provide the necessary synchronization and the server simply processes whatever it receives at its end in that order. Is my understanding correct?

Yes. The device will put batch messages received in a queue (FIFO) and process them when END_BATCH is received.

3) This question is about the END_BATCH_RESPONSE. If client sent 10 operations in a batch (assume 1->10) and during commit the 5th operation failed, the server will report 5 in the failedSequence. Now - I am thinking that its the client's responsibility to maintain atomicity by rolling back 1->4 or reissuing 5->10? Is my understanding correct?

All operations in a batch will be committed with( ALL or NOTHING). If any operation failed during END_BATCH, an error code and message are set in the END_BATCH_RESPONSE, and NO operation in this batch will be committed to the DB.

Thanks. chiaming

— Reply to this email directly or view it on GitHub https://github.com/Seagate/kinetic-protocol/issues/26.

[abhideodhar]

@chiaming2000

I have few more followup questions:

1) Are the batch requests expected to be persistent? a) For example - what happens if the client issues a batch start and performs few puts, and then the client crashes without ever sending a END_BATCH or a ABORT_BATCH? What should the server do in this case? b) What should be the behavior if the server crashes after receiving the END/ABORT batch request? Is it supposed to persist the batch requests and complete them after it restarts?

2) Lets say I have two batch operations - both of them operating on the same key.

Connection1, Batch 1 -> put('k1', oldversion=0, newversion=1) Connection2, Batch 2 -> put('k1', oldversion=0, newversion=1)

Now if the COMMIT message for Batch1 is processed before COMMIT message of Batch2, then key k1 will be stored with a version of 1. So what is the expected behavior for Batch2 - is it expected to FAIL because the version for key k1 is not matching with the stored version or is it expected to succeed and simply overwrite key k1?

3) Regarding the order of operations, your reply says "Yes. The device will put batch messages received in a queue (FIFO) and process them when END_BATCH is received.".

This order is the order which is received by the server ?? The client may have sent it in some order, but server cannot guarantee that its the same order in which it received the batch requests. Is my understanding right?

[jphughes]

On Jul 15, 2015, at 1:45 PM, abhideodhar notifications@github.com wrote:

@chiaming2000 https://github.com/chiaming2000 I have few more followup questions:

1) Are the batch requests expected to be persistent? a) For example - what happens if the client issues a batch start and performs few puts, and then the client crashes without ever sending a END_BATCH or a ABORT_BATCH? What should the server do in this case?

If end_batch never comes, the batch never happened. b) What should be the behavior if the server crashes after receiving the END/ABORT batch request? Is it supposed to persist the batch requests and complete them after it restarts?

If the drive crashes (loses power or any other terminal event) after receiving end_batch, but before sending the end_batch response, the batch either happened or it didn’t happen. No partial state. We do not know which case, but we can guarantee all or nothing.

This is very similar to a network failure before or after a drive completed an operation. The caller does not know if the failure occurred sending the message or getting the response.

2) Lets say I have two batch operations - both of them operating on the same key.

Connection1, Batch 1 -> put('k1', oldversion=0, newversion=1) Connection2, Batch 2 -> put('k1', oldversion=0, newversion=1)

Now if the COMMIT message for Batch1 is processed before COMMIT message of Batch2, then key k1 will be stored with a version of 1. So what is the expected behavior for Batch2 - is it expected to FAIL because the version for key k1 is not matching with the stored version or is it expected to succeed and simply overwrite key k1?

The behavior is that, on end_batch, all versions are checked and then the entire batch is committed. During the process of checking the versions and committing the transaction, no other commands are allowed to be processed in the drive.

One can think of the batch as being mindlessly buffered and then completely executed at the end_batch.

Also, over a single connection you can interleave batches. For instance.

put “key”,”value1”, old-version=“”, new-version=1 batch start=1 batch start=2 put batch=2 “key”,”value3”, old-version=1, new-version=3 put batch=1 “key”,”value2”, old-version=1, new-version=2 end batch=1 end batch=2

In this case batch=1 will succeed and batch=2 will fail because the version is incorrect. The value after this will be “key”,”value2”, db-version=2

If these two batches occurred on different connections either 1 or 2 will succeed, but not both.

3) Regarding the order of operations, your reply says "Yes. The device will put batch messages received in a queue (FIFO) and process them when END_BATCH is received.".

This order is the order which is received by the server ?? The client may have sent it in some order, but server cannot guarantee that its the same order in which it received the batch requests. Is my understanding right?

the connection is TCP/IP. TCP/IP guarantees order within a single connection. The order of the puts on a connection are preserved. This is critical in batches and non batches, both.

It is bad form to do more than one put or delete for the same key in the same batch, but it is allowed. This is a bit more complicated. All versions are checked against the stored keys during the end-batch, not against the previous puts in the same batch.

Example: put “key”,”value1”, old-version=“”, new-version=1 batch start=1 put batch=1 “key”,”value2”, old-version=1, new-version=2 put batch=1 “key”,”value3”, old-version=2, new-version=3 delete batch=1 “key”, old-version=3 put batch=1 “key”,”value4”, old-version=3, new-version=4 end batch=1

will fail in the end-batch saying that batch put of value3 failed because of an incorrect version. All of the puts will be compared to the database version. The following is ridiculous, but will work.

put “key”,”value1”, old-version=“”, new-version=1 batch start batch put “key”,”value2”, old-version=1, new-version=2 batch put “key”,”value3”, old-version=1, new-version=3 batch delete “key”, old-version=1 batch put “key”,”value4”, old-version=1, new-version=4 end batch

At the end of the batch the record will be “key”,”value3”, db-version=4

— Reply to this email directly or view it on GitHub https://github.com/Seagate/kinetic-protocol/issues/26#issuecomment-121742049.

[abhideodhar]

@jphughes - I have few more questions on batch operations:

1) The Kinetic protocol has various checks wrt clusterversion, device locked status, ACL, dbVersion for each put/delete. From a protocol perspective, at what stage do we need to perform the above validation checks for batch put/delete - after we receive the batch put/delete or right at the end during the batch commit? The reason I ask is that we could do the validations upfront, but then the clusterversion or the device locked status or even the permissions may change before the commit. So is it expected to do these checks at the batch commit step?

2) What are the semantics of batch operations wrt a) request priority - can each batch put/delete request come at a different priority or will the entire batch be at a given priority? b) TimeQuanta - will this apply to the entire batch or to an individual request within a batch? c) timeout - will this apply to the entire batch or to an individual request within a batch?

3) If the server is busy processing batch puts/deletes, and a commit arrives, there is a possibility of commit hitting a timeout. Can the server send an error code SERVICE_BUSY in this case? How will the client handle this scenario? Will it simply retry the batch commit ?

Thanks!

[jphughes]

1. clusterversion, device locked status, ACL are performed immediately when the batched command arrive. dbVersion is checked at the endBatch time. It has to be checked at endBatch bacause it may be changed after the put is received and before the endBatch.

2) request priority and timeout are against all commands. Priority and can effect when which command is taken from the queue when there are multiple commands pending on multiple TCP connections. Timeout can always happen when the command is in the queue and has not been started processing and the timeout has occurred. This is just like any other command, it has not been started, it will be returned in error. TimeQuanta does not apply to startBatch and endBatch that have actually been started processing. startBatch is trivial, and endBatch will not start processing a batch and then fail because it took too long.

[abhideodhar]

@jphughes -

James, thanks for the quick response. I have a followup question: Similar to dbVersion change for a key between batch put and batch commit, why can't the clusterversion, device locked status, ACL change between the batch put request and the batch commit?

For example - if the device was not locked when the batch put is received, but the device is locked when the batch commit is received, is it OK to perform the commit? Wouldn't that break the "device_locked" semantics?

[jphughes]

The "device locked" and "clusterVersion" both need to be checked at the on receipt of any command including the "endBatch" command. If either stop the endBatch, the commit will fail. This seems like the correct thing to do. Basically, they are correct for all commands in the batch, the batch will succeed.

Maybe I should have said "clusterversion, device locked status, ACL checks are performed when the commands arrive. This includes startBatch, put, delete, and endBatch"

I will look into ACL changes, although letting an in process transaction continue in the face of an ACL change does not seem to be that bad.

As a side note, there is a double-check in that the protocol to make sure all the pieces make it, Batch.count.