SPIRV linking (and other fedora isms?)

penguin42 commented 9 months ago

I have current head (5f3e3d336ce020d41ea4361c8c19dcb0fe872911) built on Fedora 39; I had to add SPIRV, SPIRV-Tools-opt, and SPIRV-Tools to target_list_libraries in ngscopeclient, tests/{Accleration,FIlters,Primitives} and examples/{curvetrace,scopevna} Note this is using Fedora's shipped spirv packages rather than downloading and building it from source.

I've also got the:

glslc
glfw-devel
yaml-cpp-devel
glew-devel
glslang-devel
libshaderc-devel
libshaderc-static

packages installed.

With that lot, it seems to be building and starting up.

azonenberg commented 9 months ago

I'll look into the missing libraries in the CMake script. Seems some systems need them and others complain if they're there, still investigating why.

In the meantime can you send a PR to scopehal-docs with the additional packages you had to install, as well as skipping the "install glslc from source" bit on Fedora? Our current Fedora install section in the docs is definitely out of date.

daglem commented 9 months ago

I hope the following can be useful as a starting point for Fedora support.

For Fedora 38, it doesn't seem to be necessary to manually install the Vulkan SDK (currently at version 1.3.243.0-1.fc38), i.e. section "3. Install Vulkan SDK" can be skipped, and also the setting of environment variables in "4. Build scopehal and scopehal-apps".

Dependencies in the same order as for Debian/Ubuntu for section "1. Install dependencies":

sudo dnf install cmake pkg-config glm-devel \
gtkmm3.0-devel libsigc++30-devel libyaml-devel yaml-cpp-devel \
    liblxi-devel libtirpc-devel texlive glew-devel \
    catch2-devel vulkan-loader-devel glslang-devel glfw-devel \
    glslc libshaderc-devel libshaderc-static \
    spirv-tools-devel

Patch for scopehal:


diff --git a/scopehal/CMakeLists.txt b/scopehal/CMakeLists.txt
index 489bdda..60c6471 100644
--- a/scopehal/CMakeLists.txt
+++ b/scopehal/CMakeLists.txt
@@ -2,6 +2,7 @@ include_directories(SYSTEM ${GTKMM_INCLUDE_DIRS} ${SIGCXX_INCLUDE_DIRS})
link_directories(${GTKMM_LIBRARY_DIRS} ${SIGCXX_LIBRARY_DIRS})

find_library(LXI_LIB lxi)
+find_library(TIRPC_LIB tirpc)
find_library(LINUXGPIB_LIB gpib)
find_package(glfw3 REQUIRED)

@@ -18,6 +19,16 @@ else() find_package(YAML REQUIRED) endif()

+if(LINUX AND CMAKE_SYSTEM_VERSION MATCHES ".fc[0-9]+.")

set(FEDORA TRUE) +endif()
+if(WIN32 OR FEDORA)
set(SPIRV_LIBRARIES SPIRV SPIRV-Tools-opt SPIRV-Tools) +else()
set(SPIRV_LIBRARIES "") +endif()
Apple has their own idea about stat structs

if(CMAKE_SYSTEM_NAME STREQUAL "Darwin") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Dst_mtim=st_mtimespec") @@ -25,7 +36,12 @@ endif()

if(LXI_LIB) set(HAS_LXI true)
set(LXI_LIBRARIES ${LXI_LIB})
if(FEDORA AND TIRPC_LIB)
On Fedora, tirpc is (still?) not linked into lxi
set(LXI_LIBRARIES ${LXI_LIB} ${TIRPC_LIB})
else()
set(LXI_LIBRARIES ${LXI_LIB})
endif() else() set(HAS_LXI false) set(LXI_LIBRARIES "") @@ -194,9 +210,7 @@ target_link_libraries(scopehal ${LIBFFTS_LIBRARIES} ${OpenMP_CXX_LIBRARIES} ${Vulkan_LIBRARIES}
SPIRV
SPIRV-Tools-opt
SPIRV-Tools
${SPIRV_LIBRARIES} glslang Vulkan::Vulkan -Wl,--start-group @@ -221,6 +235,7 @@ target_link_libraries(scopehal ${LIN_LIBS} ${OpenMP_CXX_LIBRARIES} ${Vulkan_LIBRARIES}
${SPIRV_LIBRARIES} Vulkan::Vulkan Vulkan::glslang Vulkan::shaderc_combined @@ -240,6 +255,7 @@ target_link_libraries(scopehal ${LIBFFTS_LIBRARIES} ${OpenMP_CXX_LIBRARIES} ${Vulkan_LIBRARIES}
${SPIRV_LIBRARIES} Vulkan::Vulkan Vulkan::glslang Vulkan::shaderc_combined

Unfortunately I can't test this on my current laptop, but it does compile, and ngscopeclient starts up.

hansemro commented 9 months ago

@daglem Thanks for sharing the patch. Especially the bit on LXI and linking with libtirpc. On Arch Linux + Fedora 39, I have had to either remove liblxi or not link it to compile without missing references.

The rest of the patch lines up with what is done for scopehal-apps-git AUR package as it opts to use the distro-provided vulkan sdk: https://aur.archlinux.org/cgit/aur.git/tree/target_link_libraries.patch?h=scopehal-apps-git

hansemro commented 9 months ago

@daglem Under install dependencies section, I still need to install "C Development Tools and Libraries" group package as a build dependency.

daglem commented 9 months ago

@daglem Under install dependencies section, I still need to install "C Development Tools and Libraries" group package as a build dependency.

@hansemro I believe it would be better if we could identify exactly which additional packages are required (gcc-c++ and make maybe?), assuming that the end goal is to have an RPM .spec file with the necessary BuildRequires and Requires.

I guess I should just make PRs for the above - at least it would be a step in the right direction. I'll look into it in a few hours.

daglem commented 9 months ago

Pull requests: https://github.com/glscopeclient/scopehal/pull/801 https://github.com/glscopeclient/scopehal-docs/pull/69

hansemro commented 9 months ago

If we end up removing the step to install vulkan sdk manually, here are the dependencies for debian12/ubuntu23.04 (and newer):

sudo apt install g++ cmake pkg-config libglm-dev \
libgtkmm-3.0-dev libsigc++-2.0-dev libyaml-cpp-dev \
liblxi-dev libtirpc-dev texlive texlive-fonts-extra libglew-dev \
catch2 libvulkan-dev glslang-dev libglfw3-dev \
glslc libshaderc1 libshaderc-dev spirv-tools

However, for older releases, the vulkan sdk has to be installed manually. So for the sake of providing distro-generic steps, I think we should still keep this extra step.

azonenberg commented 9 months ago

Now that I've merged those PRs where do we stand on this issue? Can we close it or is there anything missing?

daglem commented 9 months ago

IMO you can close this issue, just remember to update the documentation linked to from the README.

If you'd like to have an RPM .spec file, I can possibly help out with that later (if and when the dependency on FFTS is removed).

hansemro commented 9 months ago

I think we should apply the same changes across all Linux distros (not just Fedora), since:

SunRPC got removed in Glibc 2.26 and later, requiring libtirpc to take its place: https://wiki.gentoo.org/wiki/Project:Toolchain/Glibc_2.26_porting_notes/RPC_implementation
Linking SPIRV Tools will reduce the amount of patches package maintainers will need to apply (without downsides?)

Build error snippet (Arch Linux without patch):

/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::CreateCompactIdsPass()'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::Optimizer::PassToken::~PassToken()'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvBinaryDestroy'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::Optimizer::RegisterPerformancePasses()'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvValidatorOptionsDestroy'
/usr/bin/ld: /usr/lib/liblxi.so: undefined reference to `xdr_char'
/usr/bin/ld: /usr/lib/liblxi.so: undefined reference to `xdr_bool'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvContextDestroy'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvValidatorOptionsSetBeforeHlslLegalization'
/usr/bin/ld: /usr/lib/liblxi.so: undefined reference to `xdr_u_int'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::SpirvTools::SpirvTools(spv_target_env)'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::Optimizer::RegisterLegalizationPasses()'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvTextToBinary'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::Optimizer::Run(unsigned int const*, unsigned long, std::vector<unsigned int, std::allocator<unsigned int> >*, spv_optimizer_options_t*) const'
/usr/bin/ld: /usr/lib/liblxi.so: undefined reference to `xdr_u_short'
/usr/bin/ld: /usr/lib/liblxi.so: undefined reference to `xdr_bytes'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvDiagnosticDestroy'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvValidatorOptionsSetSkipBlockLayout'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::Optimizer::SetMessageConsumer(std::function<void (spv_message_level_t, char const*, spv_position_t const&, char const*)>)'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::Optimizer::~Optimizer()'
/usr/bin/ld: /usr/lib/liblxi.so: undefined reference to `xdr_void'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvOptimizerOptionsSetValidatorOptions'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvContextCreate'
/usr/bin/ld: /usr/lib/liblxi.so: undefined reference to `xdr_enum'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::SpirvTools::SetMessageConsumer(std::function<void (spv_message_level_t, char const*, spv_position_t const&, char const*)>)'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvValidatorOptionsCreate'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::Optimizer::RegisterSizePasses()'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvOptimizerOptionsCreate'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvOptimizerOptionsSetPreserveBindings'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::SpirvTools::Disassemble(std::vector<unsigned int, std::allocator<unsigned int> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, unsigned int) const'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvOptimizerOptionsSetRunValidator'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvOptimizerOptionsDestroy'
/usr/bin/ld: /usr/lib/liblxi.so: undefined reference to `xdr_u_char'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::Optimizer::Optimizer(spv_target_env)'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::SpirvTools::~SpirvTools()'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::CreateStripDebugInfoPass()'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvValidatorOptionsSetRelaxLogicalPointer'
/usr/bin/ld: /usr/lib/liblxi.so: undefined reference to `xdr_int'
/usr/bin/ld: /usr/lib/liblxi.so: undefined reference to `clnt_create'
/usr/bin/ld: ../../lib/scopehal/libscopehal.so: undefined reference to `spvtools::Optimizer::RegisterPass(spvtools::Optimizer::PassToken&&)'
/usr/bin/ld: /usr/lib/liblxi.so: undefined reference to `xdr_string'
collect2: error: ld returned 1 exit status
make[2]: *** [src/ngscopeclient/CMakeFiles/ngscopeclient.dir/build.make:1399: src/ngscopeclient/ngscopeclient] Error 1
make[1]: *** [CMakeFiles/Makefile2:1570: src/ngscopeclient/CMakeFiles/ngscopeclient.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

nm17 commented 9 months ago

Same error on Ubuntu 23.04. Commit cc4335b4b41782a5eb89e41c4c0f49b2485ca90f

ngscopeclient / scopehal-apps

SPIRV linking (and other fedora isms?) #611

Apple has their own idea about stat structs

On Fedora, tirpc is (still?) not linked into lxi