handoff_timeout if the receiver does not respond to a sender message within this period, then the sender should assume the receiver is down and exit the fold. The sender can exchange sync messages every handoff_acksync_threshold batches to ensure that batches do not accumulate at the receiver causing the sender to fail to get a timely response to a future request.
handoff_receive_timeout if the receiver does not receive a message from the sender within this timeout, it will assume that the sender is down and exit the process. The sender will interrupt the fold if it has not sent a batch in the last handoff_receive_timeout div 3 to send a sync message as a keepalive to prevent this timeout if it is alive.
The default value of handoff_acksync_threshold is changed to 1 from 25. At 25 batches, and 250 bytes every object size, a backlog of 100K handoff items could develop at the receiver - requiring a sub-1ms response time for each handoff item to avoid triggering the timeout. Alternative default of 1 is less likely to see the timeout, at the small cost of an additional sync round-trip per batch.
A handoff_batch_threshold_count has been added to generate a threshold for a batch when this count of objects is reached. The handoff_batch_threshold is also respected (which triggers a batch on size in bytes). This is to prevent very large batches by count when object sizes are small.
The ?PT_MSG_OLDSYNC is now properly deprecated. The receiver will still respond, should a cluster contain a handoff sender still on a previous version, but the sender will now only send the ?PT_MSG_SYNC message.
Reformatting of code to fit column width.
Standardise handling of sync messages through riakc_core_handoff_sender:send_sync/3 function, and now error logs raised at the actual message where an error is received. The item queue is redacted in the error generated by a throw, so that important details are not obfuscated by a large list of binaries.
A new log has been added to log progress in sending batches every handoff_acklog_threshold batches (default 100). This allows for progress to be tracked through log indexers, rather than manually by operators calling the riak admin transfers command.
All code for (legacy) non-batched handoff removed.
martinsumner
commented
1 year ago
Refactor handoff to avoid TCP RECV error - see https://github.com/basho/riak_core/issues/994.
Clarify meaning of terms:
handoff_timeoutif the receiver does not respond to a sender message within this period, then the sender should assume the receiver is down and exit the fold. The sender can exchange sync messages everyhandoff_acksync_thresholdbatches to ensure that batches do not accumulate at the receiver causing the sender to fail to get a timely response to a future request.handoff_receive_timeoutif the receiver does not receive a message from the sender within this timeout, it will assume that the sender is down and exit the process. The sender will interrupt the fold if it has not sent a batch in the lasthandoff_receive_timeout div 3to send a sync message as a keepalive to prevent this timeout if it is alive.The default value of
handoff_acksync_thresholdis changed to 1 from 25. At 25 batches, and 250 bytes every object size, a backlog of 100K handoff items could develop at the receiver - requiring a sub-1ms response time for each handoff item to avoid triggering the timeout. Alternative default of 1 is less likely to see the timeout, at the small cost of an additional sync round-trip per batch.A
handoff_batch_threshold_counthas been added to generate a threshold for a batch when this count of objects is reached. Thehandoff_batch_thresholdis also respected (which triggers a batch on size in bytes). This is to prevent very large batches by count when object sizes are small.The ?PT_MSG_OLDSYNC is now properly deprecated. The receiver will still respond, should a cluster contain a handoff sender still on a previous version, but the sender will now only send the ?PT_MSG_SYNC message.
Reformatting of code to fit column width.
Standardise handling of sync messages through
riakc_core_handoff_sender:send_sync/3function, and now error logs raised at the actual message where an error is received. The item queue is redacted in the error generated by a throw, so that important details are not obfuscated by a large list of binaries.A new log has been added to log progress in sending batches every
handoff_acklog_thresholdbatches (default 100). This allows for progress to be tracked through log indexers, rather than manually by operators calling theriak admin transferscommand.All code for (legacy) non-batched handoff removed.