oneTBB: port dispatcher, thread limits and add build option

[brechtvl]

Description of Change(s)

After the various simple changes submitted as separate pull requests, this actually does the bulk of the work of adding TBB 2021 support. It is optional, older TBB version should still work exactly the same.

• New build_usd.py option --onetbb, disabled by default.
• Succeeded building and passing tests on Linux and macOS with default options. I did not try Windows or enabling all build options.
• The work dispatcher changed from task to task_group, and also needed a hack to make concurrent wait thread safe. This part of the changes is most likely to affect performance, good or bad, and needs some real world testing to understand the impact. I read on the VFX platform mailing list that there is contact with Intel engineers to improve TBB for USD, this seems worth bringing up.
• Set thread limits now uses global_control, which behaves differently than task_scheduler_init in ways that can't be hidden by the API.

---

The impact of the thread limits changes in complex applications like DCCs is not easy to understand. One would hope that such applications do not use the USD API or environment variables to set thread limits.

If there are multiple parts of the application using and configuring TBB thread limits, here are some potential differences when using oneTBB. This is from my understanding of the docs and code, not testing in practice.

• With older TBB, the first created task_scheduler_init wins and determines the number of threads. USD can create one as part of static initialization, which can happen before a host application creates one, and so override the number of threads for the entire application. With oneTBB, every global_control call changes the number of threads instead of just the first.

• With older TBB, task_scheduler_init affects one thread spawing tasks. It may commonly be used early on in the main thread and affect the entire application, but not necessarily. With oneTBB, global_control always affects the entire application regardless of the thread it is called from.

• After these changes, the thread limit API can still be used to decrease the number of threads used by TBB. But it can no longer increase them beyond the number of cores just by itself. To increase the number of threads, the parallel code now needs to be run inside a task arena with increased max concurrency as well. I could not find a mechanism in oneTBB that allows changing the max concurrency of the current existing task arena.

Fixes Issue(s)

Part of #1471, adding oneTBB support.

• [x] I have verified that all unit tests pass with the proposed changes
• [x] I have submitted a signed Contributor License Agreement

[sunyab]

Filed as internal issue #USD-8384

[loqs]

@brechtvl I noticed this pull request now has merge conflicts.

[brechtvl]

This merge left out the build system changes. But I guess that was intentional, and there are more changes coming before it's considered ready.

[spiffmon]

That's right - we're landing the last few changes now, @brechtvl - thanks so much for your giant contributions, here!