Separate endpoints for recv communicators from same source endpoint

[rauteric]

This patch changes the plugin's endpoint creation behavior as follows:

1. All send communicators will use the same endpoint.
2. When creating an endpoint for receive communicators, first check if any existing endpoint is not already connected to the source endpoint. Only create a new recv endpoint if all existing endpoints are already connected to the source endpoint.

Also has some supporting refactoring commits.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[AmedeoSapio]

bot:aws:retest

[rauteric]

Force-push to 98bf2b1 is a rebase on master.

[rauteric]

Force-push to 8850320 addresses (my own) feedback

[rauteric]

(Note: the CI failure is spurious. Will try again on next revision.)

[rauteric]

Resolved ✅ Somehow I majorly butchered performance in yesterday's update to this PR, when ENDPOINT_PER_COMM=1. Need to sort that out.

Edit: yet another lesson to use NCCL_NET="AWS Libfabric" for my testing. I was failing during init and falling through to sockets provider and not realizing it.

[rauteric]

Moved rail index assignment logic to platform hook and added AWS/EFA implementation.

[rauteric]

Force-push to 73fab34 addressed feedback.

[rauteric]

Force-push to 205a026 is rebase on master.

[rauteric]

• Removed the rail-sorting patches from this PR (will post a separate PR)
• Squashed the ep-per-comm and use-ep-addr-list commits
• Removed include of nccl_ofi.h from ep addr list code.

[rauteric]

Rebased master (fix for CodeChecker workflow)

[rauteric]

bot:aws:retest

[sunkuamzn]

bot:aws:retest

[a-szegel]

bot:aws:retest