search

tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
399 stars 49 forks source link

Setup UBSAN and ASAN on CI #11453

Open eyonland opened 1 month ago

eyonland commented 1 month ago

We have a lot of host bugs being reported that really sound like they have nothing to do with our devices. The sooner we get that setup on CI the better.

TT-billteng commented 1 month ago

we can help add the workflow for this @tt-rkim @ttmchiou

tt-rkim commented 1 month ago

Is the work for this:

patrickroberts commented 1 month ago

Is the work for this:

  • add option in build_metal.sh and a -D option in cmake
  • if option is detected in top level CMakeLists.txt, add ubsan / asan (-f options) to compile options?
  • create scheduled job to call this build? Maybe add to build artifacts options?

This is part of it, yes, and getting some input / help adding jobs to CI would be awesome, but I anticipate the bulk of the work will actually be getting the existing tests to run to completion and pass with these configurations enabled. Once we get to that point, making ASAN/UBSAN-enabled tests passing should ideally become a precondition for merging new pull requests.

TT-billteng commented 1 month ago

Is the work for this:

  • add option in build_metal.sh and a -D option in cmake
  • if option is detected in top level CMakeLists.txt, add ubsan / asan (-f options) to compile options?
  • create scheduled job to call this build? Maybe add to build artifacts options?

This is part of it, yes, and getting some input / help adding jobs to CI would be awesome, but I anticipate the bulk of the work will actually be getting the existing tests to run to completion and pass with these configurations enabled. Once we get to that point, making ASAN/UBSAN-enabled tests passing should ideally become a precondition for merging new pull requests.

Post-commit has 65k tests currently, hopefully we can craft a representable subset for ASAN/UBSAN