Replace hard-coded instrumentation with configurable instrumentation

[nchaimov]

The existing libTooling-based instrumentor hard-codes the lines of code to be inserted to initialize TAU, start and top timers, etc. This should be changed so that the instrumentation can be specified by a configuration file.

We will need options for things like:

• initialization
• timer definition
• timer start
• timer stop

We should also review other instrumentation APIs to determine what other instrumentation points might be necessary.

[khuck]

Suggested libraries to examine:

• nvtx (cuda) https://docs.nvidia.com/gameworks/content/gameworkslibrary/nvtx/nvidia_tools_extension_library_nvtx.htm
• roctx (hip) https://github.com/ROCm-Developer-Tools/roctracer
• perfstubs (kevin) https://github.com/khuck/perfstubs
• itt (intel) https://www.intel.com/content/www/us/en/develop/documentation/vtune-help/top/api-support/instrumentation-and-tracing-technology-apis.html

Most of these libraries have marker events, timer start/stop, region push/pop, etc. sometimes they are thread-local, sometimes they are not.

[zbeekman]

@PlatinumCD @nchaimov Is there a reason this issue was reopened? Is it because of the dockerhub comments on the PR? Something else?

[PlatinumCD]

@zbeekman I am not sure why I closed and reopened back in July. This might have been a mistake.

[nchaimov]

I don't think this was meant to have been reopened. I'll close it again.