[Feature Request] add include paths, library directories, and libraries to JIT kernel builds

[ct-clmsn]

Is your feature request related to a problem? Please describe.

I would like to add compiled libraries with kernels builds.

Describe the solution you'd like

Add 3 state vectors to Program:

• std::vector include_dirs
• std::vector library_dirs
• std::vector libraries

the first vector stores a list of include directories, the second stores a list of library directories, the third stores a list of libraries that can be statically added to a kernel

Add methods to Program (tt_metal/impl/program/program.hpp)

• int Program::add_include_dirs(std::vector<std::filesystem::path> const& paths)
• int Program::add_library_dirs(std::vector<std::filesystem::path> const& paths)
• int Program::add_static_libraries(std::vector<std::string> const& paths)

add_include_dirs accepts a vector of include paths, validates the paths to make sure they exist, and returns the number of paths that were successfully added

add_library_dirs accepts a vector of library paths, validates the paths to make sure they exist, and returns the number of paths that were successfully added

add_static_libraries accepts a vector of library names (example from '-lmath' just the 'math'), validates the libraries to make sure they exist and are static libraries, and returns the number of libraries that were successfully added

• void Program::get_flag_include_dirs(std::string & flag_str)
• void Program::get_flag_library_dirs(std::string & flag_str)
• void Program::get_flag_static_libraries(std::string & flag_str)

get_flag_include_dirs converts std::vector<std::filesystem::path> include_dirs into the compiler flags "-I -I"

get_flag_library_dirs converts std::vector<std::filesystem::path> library_dirs into the compiler flags "-L -L"

get_flag_static_libraries converts std::vector<std::string> libraries into the compiler flags "-l -l"

JitBuildOptions could store the std::strings output of each of 'Program::get_flag_include_dirs', 'Program::get_flag_library_dirs', and 'Program::get_flag_static_libraries'.

An update to Program::compile can copy the data from Program into JitBuildOptions.

Describe alternatives you've considered

I've considered implementing custom cmake scripts that could enable this functionality but deploying the compiled kernel with the host application would require a substantial amount of custom software.

Additional context

N/A

[pgkeller]

There are some concerns w/ this at present: 1) Kernels have to be built custom for the chip the run on, so libraries won't work. This is a WIP and will be addressed before long 2) We don't (yet) support offline compile. This is also coming, but will be a few months out 3) Kernel compilation will be TT arch specific, cross linking will fail. No plan yet for fat binaries 4) We will may open up the compile options to the Kernel API allowing kernel writes to specify, eg, "-I", etc. TBD