UCP/DOC: ucp_tag_send_nb() in case of IOV datatype

[shssf]

The function ucp_tag_send_nb() can work with different datatypes (contig, iov, generic) and input parameter const void *buffer expected with different structs. In case of simple contig datatype it expects char * but with iov datatype it expects ucp_dt_iov_t*

it would be good to point this important difference explicitly. Current documentation looks like:

/**
 * @ingroup UCP_COMM
 * @brief Non-blocking tagged-send operations
 *
 * This routine sends a messages that is described by the local address @a
 * buffer, size @a count, and @a datatype object to the destination endpoint
 * @a ep. Each message is associated with a @a tag value that is used for
 * message matching on the @ref ucp_tag_recv_nb "receiver".  The routine is
 * non-blocking and therefore returns immediately, however the actual send
 * operation may be delayed.  The send operation is considered completed when
 * it is safe to reuse the source @e buffer.  If the send operation is
 * completed immediately the routine return UCS_OK and the call-back function
 * @a cb is @b not invoked. If the operation is @b not completed immediately
 * and no error reported then the UCP library will schedule to invoke the
 * call-back @a cb whenever the send operation will be completed. In other
 * words, the completion of a message can be signaled by the return code or
 * the call-back.
 *
 * @note The user should not modify any part of the @a buffer after this
 *       operation is called, until the operation completes.
 *
 * @param [in]  ep          Destination endpoint handle.
 * @param [in]  buffer      Pointer to the message buffer (payload).
 * @param [in]  count       Number of elements to send
 * @param [in]  datatype    Datatype descriptor for the elements in the buffer.
 * @param [in]  tag         Message tag.
 * @param [in]  cb          Callback function that is invoked whenever the
 *                          send operation is completed. It is important to note
 *                          that the call-back is only invoked in a case when
 *                          the operation cannot be completed in place.
 *
 * @return UCS_OK           - The send operation was completed immediately.
 * @return UCS_PTR_IS_ERR(_ptr) - The send operation failed.
 * @return otherwise        - Operation was scheduled for send and can be
 *                          completed in any point in time. The request handle
 *                          is returned to the application in order to track
 *                          progress of the message. The application is
 *                          responsible to released the handle using
 *                          @ref ucp_request_release "ucp_request_release()"
 *                          routine.
 * @todo
 * @li Describe the thread safety requirement for the call-back.
 * @li What happens if the request is released before the call-back is invoked.
 */
ucs_status_ptr_t ucp_tag_send_nb(ucp_ep_h ep, const void *buffer, size_t count,
                                 ucp_datatype_t datatype, ucp_tag_t tag,
                                 ucp_send_callback_t cb);

[shssf]

Also, the completion procedure is not clearly stated or referenced. It would be greate to mention that cb argument might be NULL (currently it can't be - failed) and ucp_request_is_completed().

We have two methods to the request status tracking:

1. Using callback function (parameter cb) 1a. Need to explain why the line request->completed = 0; /* Reset request state before recycling it */ in ucp_hello_world is needed.
2. the function ucp_request_is_completed()

Any of these methods can be used by the user choice.