Message send api should support iovectors for sending multiple data buffers in 1 shot

[karthick18]

A customer has a requirement to use the iovecs in the message send api. Hence this is a request to have a new API to send messages with iovecs for data.

SaMsgMessageT currently can send only 1 data chunk. The message send IOV variant should be able to take a struct iovec as an argument and send multiple data chunks.

typedef struct SaMsgMessageIovec { [...snipped... common to SaMsgMessageT but without data and size ..] struct iovec *iovec; ClUint32T numIovecs; } SaMsgMessageIovecT;

Now you can change the clMsgMessageInternalSend call graph down to msgsendidl to use SaMsgMessageIovecT instead of SaMsgMessageT which is optimal.

The existing saMsgMessageSend can call clMsgMessageInternalSend with a simple wrapper that converts the SaMsgMessageT to SaMsgMessageIovecT. So you can use iovec as the standard.

saMsgMessageSend() { SaMsgMessageIovecT msgVector; struct iovec iovec = {0}; msgVector.senderName = msg->senderName;

[...SNIPPED... copy remaining msg struct fields to msgiovec struct ] msgVector.iovec = &iovec; msgVector.numIovecs = 1; iovec.iov_base = (void*)msg->data; iovec.iov_len = msg->size; return clMsgMessageInternalSend( ..., &msgVector); }

Hence the msg send now would end up using the standard iovector struct variants as well.

The new message send api's (like saMsgMessageSendIovec or saMsgMessageSendAsyncIovec or groupsendwithkey iovec) that accepts SaMsgMessageIovecT from user instead of SaMsgMessageT can just directly call clMsgMessageInternalSend.

So you just change the base argument to use the SaMsgMessageIovecT internally instead of SaMsgMessageT to msg send.

For IDL, you can have a iovec struct that takes iov_base as a ClUint8T * with iov_len as the size, and then does a ptr array marshall with len="numIovecs" in SaMsgMessageIovecT IDL representation.

You can check the checkpoint write idl that uses ClCkptIoVectorElementT to allow passing iovectors. This is similar but for data.

And finally in the clMsgMessageReceived path, you have to scatter gather if you are not going to provide saMsgMessageReceiveIovec apis to receive the multiple iovecs sent. (like a readv or a recvmsg system call)

So the clMsgQueueTheLocalMessage would accumulate the iovecs in msgIov->numIovecs by calculating the send buffer space, create a flattened chunk and then make it into a single SaMsgMessageT with data and size (combination of iovecs iov_len). That is you can do the opposite if you want to keep the pRecvInfo intact on the receiver side that doesn't accept iovecs like in the sender side. (no receiver iovec apis) But maybe its better to provide iovec on the receiver side later but its upto you.

Also I do not really like the clMsgMessageToMessageCopy api in the current form that tries to allocate and clone the data instead of directly using the allocated data buffer from IDL.

If clMsgMessageToMessageCopy is called from the IDL or rmd context -- clMsgMessageReceived, 4, 0, 0, then you can just directly store pointers by reference and nuke the target.

pTempMessage = clHeapCalloc(1, sizeof(_pTempMessage)); memcp(pTempMessage, pMessage, sizeof(_pTempMessage)); //reset src pointers to avoid IDL from freeing it in the epilogue path pMessage->senderName = NULL; pMessage->data = NULL;

You can do this optimization to avoid copying data,etc. by reusing the IDL allocated buffer if you were called from the RMD/IDL clMsgMessageReceived context. If you were called from the loopback context from clMsgMessageSend that checks if destination and src are part of the same process, then you HAVE to copy like you are doing now. So maybe have a flag to the clMsgMessageToMessageCopy (or to be iovec copy) and see if you can avoid copying.

But with iovec receives of more than 1 iovec, you have to scatter gather by allocating the space for the received iovecs based on sum of iov[i].iov_len for numIovecs and then copy from iovec[i].iov_data for each into the allocated buffer of pRecvInfo before queueing.

But again, if you support iovecs in the receiver side and have apis where the user can receive iovecs, you can do the same optimization suggested above to reuse the IDL allocated space.

I hope things are clear :)

[karthick18]

Also make sure to test properly. (even saMsgMessageSend cases as well and including valgrind checks for leaks, etc.)

[CangTranOC]

Checked in 01f1ff90c0252304c66121b5efb32c1ffd9ce806 for code review. It passes with aspSafTests. It is now under code-reviewing, leak checking & testing with multiple msg sending scenarios.