search

kokkos / kokkos-tools

Kokkos C++ Performance Portability Programming Ecosystem: Profiling and Debugging Tools
Other
109 stars 55 forks source link

Create unit tests for all Kokkos Tools libraries and utilities #191

Open vlkale opened 1 year ago

vlkale commented 1 year ago

Complementing the example directory, a set of unit tests and a framework for testing needs to be created for all Kokkos Tools libraries. This includes in-house tooling libraries, e.g., kernel-logger, vendor connectors, e.g., roctx-connector, and utilities, e.g., Kokkos-sampler, and sophisticated tpls, apex Kokkos Tool library.

In the near-term and of priority: the focus will be on new tools integrated being into the Kokkos Tools repo, in particular the sampler utility and fixes in the nvtx-connnector, which is expected to have heavy use. (The rocprof-connector and vtune-connector should also have similar tests).

Longer-term: more tests, perhaps in part community-driven, will be incorporated in to the set of tests.

This may eventually involve a testing framework like Google Test and the testing would then happen within the new Kokkos Tools make build system and nightly CI builds of Kokkos Tools

vlkale commented 1 year ago

I propose a process for developing these tests that is directly motivated by Kokkos applications using Kokkos Tools.

For now, let's define a unit as callback of a particular tool. Each unit test should verify (is the answer correct given the function's implementation), validate (is the answer correct based on requirements of function defined), and assess performance. These three aspects give the Kokkos Tools users an assessment of quality of the implementation.

The process for establishing a set of unit tests for the Kokkos Tools would be something like:

1st phase: Obtain simple example codes from applications and benchmarks, and ensure they work. An example of a test of the sampler utility's parallel_for tools callback applied for the simple kernel timer is shown in the file. outputForSampler with Atomic Benchmark

2nd phase: By stripping out functions and code from applications: Create tests of output of tool, with one test per unit, via assertions in code.

3rd phase: Use testing framework like ctest (and could be subsequently using Google Test framework) and integrate the tests under the framework into the Kokkos Tools cmake build system.

The 1st and 2nd phase would be stored in a separate repo and be not part of the Kokkos Tools repo. The code developed in the 3rd phase would be the official tests of the repo, which would be community-built and agreed upon via a community of Kokkos users.

This process could be followed or follow that of tests in Kokkos-core.