triton-inference-server / server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.
https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
BSD 3-Clause "New" or "Revised" License
8.3k stars 1.48k forks source link

Building and developing with libtritonserver.so #7320

Open asaff1 opened 5 months ago

asaff1 commented 5 months ago

Description Would like to know what is the way to include libtritonserver in a project. I did a build of triton developer tools with -DTRITON_CORE_HEADERS_ONLY=OFF so I get an install/ directory with

include/..
lib/stubs/libtritonserver.so 
lib/libtritonserver.so
src/...

Now, in my project's CMakeLists I have:

set(TRITON_INSTALL_DIR "${CMAKE_SOURCE_DIR}/triton_install")

list(APPEND CMAKE_PREFIX_PATH
    ${TRITON_INSTALL_DIR}/lib/cmake/TritonCommon
    ${TRITON_INSTALL_DIR}/lib/cmake/TritonCore
    ${TRITON_INSTALL_DIR}/lib/cmake/TritonDeveloperToolsServer
)

and:

target_link_libraries(
        myproject
    PRIVATE
    TritonDeveloperToolsServer::triton-developer_tools-server 
)

It does compile and I can run my project, but it uses the stub library so I cannot actually use triton functions. If I remove the stubs/libtritonserver.so the project does not compile. So I assume cmake is looking for the stub lib instead of the "real" one. I tried to replace stubs/libtritonserver.so with the real library - then everything works fine and the real library is used, but I think this is a hack and not how it should be used.

I believe that triton-developer_tools-server is including the stub, because in my cmakelist I don't include the stub. How can I tell cmake to use the real library instead of the stub?

Triton Information 24.03

Are you using the Triton container or did you build it yourself? Build myself

statiraju commented 5 months ago

@nvda-mesharma for viz

asaff1 commented 4 months ago

Any update?

nkinnaird commented 2 months ago

Also curious to the answer.

GuanLuo commented 2 months ago

The parts that depends on Triton core library is linking against the stub library as you can see in the CMakeLists.txt (triton-core-serverstub). I don't think it is a hack to replace the stub library with the real one for execution.

That being said, it's possible that the CMake file can be improved to expose triton-core as target, which may point to either the stub library or the real one depending on whether TRITON_CORE_HEADERS_ONLY is set. CC @fpetrini15