Cleanup BoundaryExchange

[mfdeakin-sandia]

Here's a list of potentially not well thought out ideas on what to do to make it a more intuitive interface and easier to understand:

• Make it so that using boundary exchange on 1D and ND (where N>1) fields is not mutually exclusive
  • If that's not possible, split them into separate classes
• Split long methods into shorter ones. Specifically, pack_and_send, recv_and_unpack, and build_buffers_views_and_requests (especially this one since it looks like a fair amount of code repetition) look viable for cleaning up.
• Get rid of set_num_fields and clean_up, define the number of fields in the constructor and keep it that way for the lifetime of the object

Thoughts?

[mfdeakin-sandia]

Also clean up boundary_exchange_ut, since it's also a lot of repeated code

[bartgol]

Agreed on 2 and 3. For number 1, the reason I made them exclusive is that you never do 1D and 2D/3D at the same time. So if you have, say, some 3D fields registered and one 1D field registered, when you call the minmax exchange, you end up communicating way more than what you need. To avoid this, you should store two sets of send/recv requests, and start/wait only the ones that you actually need to do. You would then have to make sure all the 1D entries are contiguous in the buffer, and all the 2D/3D entries are contiguous in the buffer. It is possible, but it may require some time to come up with a decent design. Splitting the class in two would make it "simpler", at the cost of a probably large code duplication.

I don't have a strong preference in this.