Add callback to signal successful publish

[lthibault]

On several occasions, I have found myself wanting to know exactly when a particular call to Topic.Publish has written the message to the wire. This is either because of some business logic (e.g. wait until peers have been signaled before doing something) or because I want to pool the []byte argument in high-throughput applications.

From an API perspective, this would ideally take the form of a PubOpt, allowing us to write:

err := t.Publish(ctx, data, WithCallback(func(m *Message) {
    pool.Put(m.Data)  // sync.Pool
})

The main difficulty I'm seeing is that messages are fanned out to a number of peers via buffered channels, so this would likely require some kind of refcounting scheme to ensure the callback is not called prematurely.

Is this a sensible ask? Any recommendations?

[vyzo]

From an api perspective, i wouldnt say no if we had an implementation, as it is indeed quite useful.

On the same time this really isnt straightforward to implement, there are complexities with dropped messages etc.

I guess we could put a barrier in the Message struct and invoke it when (if we can do that reliably) we are done sending or dropping. Probably a 1-buffer channel better than a callback, as it solves thread ownership issues.

I will be happy to review and ultimately merge a pr if you can come up with something reasonable.

[lthibault]

Sounds like a plan.

Do you have any thoughts on how a barrier might be implemented? Are you thinking sync.WaitGroup?