mesonbuild / meson

The Meson Build System
http://mesonbuild.com
Apache License 2.0
5.56k stars 1.61k forks source link

CMake fails for ASM custom rules (example libjpeg-turbo) #8806

Open fzwoch opened 3 years ago

fzwoch commented 3 years ago

Describe the bug When trying to use CMake module with projects with ASM (may be specific to this particular one: https://github.com/libjpeg-turbo/libjpeg-turbo) fails as ASM objects will not be compiled.

To Reproduce CMake way of building:

$ cmake -B build -G Ninja
$ ninja -C build
$ ninja -C build -t targets all | grep asm
simd/CMakeFiles/simd.dir/x86_64/jsimdcpu.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jfdctflt-sse.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jccolor-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jcgray-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jchuff-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jcphuff-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jcsample-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jdcolor-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jdmerge-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jdsample-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jfdctfst-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jfdctint-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jidctflt-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jidctfst-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jidctint-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jidctred-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jquantf-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jquanti-sse2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jccolor-avx2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jcgray-avx2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jcsample-avx2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jdcolor-avx2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jdmerge-avx2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jdsample-avx2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jfdctint-avx2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jidctint-avx2.asm.o: ASM_NASM_COMPILER__simd_Release
simd/CMakeFiles/simd.dir/x86_64/jquanti-avx2.asm.o: ASM_NASM_COMPILER__simd_Release

Meson way to include as subproject:

project('test', 'c')

cmake = import('cmake')

cmake.subproject('libjpeg-turbo')
$ meson build
$ ninja -C build # <-- compilation fails
$ ninja -C build -t targets all | grep asm # <-- returns no output

Most suspicious output from meson call:

libjpeg-turbo| WARNING: Object files do not contain source file extensions, thus falling back to guessing them.

Expected behavior Expected the libjpeg-turbo build to succeed.

system parameters

eli-schwartz commented 3 years ago

FWIW: https://github.com/mesonbuild/libjpeg-turbo

There is a native meson-based wrap for it.

fzwoch commented 3 years ago

I know it exists, but was trying to solve it via CMake module. Eventually it will fix other cases as well.

Also found in the wrap: # TODO: Add simd support

mensinda commented 3 years ago

You can disable the SIMD module with a CMake option. I know that this does not fix the problem, but I am not sure whether I will add any assembler support because this will get very complicated very fast (even for "only" nasm).

The main problem is that meson does not have any native support for compiling *.asm files natively with something like executable('asmExe', 'file1.asm). Thus, we would have to map an already complicated CMakeLists.txt to custom targets and generators. While this is not impossible, it is a massively complicated solution for only a handful of projects (that then also only would support x86).

So I would say adding support for any type of assembler is pretty low until Meson can natively deal with assembler files.

mensinda commented 3 years ago

Specifically, the following meson.build should do the job for you if you do not absolutely require SIMD:

project('test', 'c')

cmake = import('cmake')

jpgopt = cmake.subproject_options()
jpgopt.add_cmake_defines({'WITH_SIMD': 'OFF'})
libjpg = cmake.subproject('libjpeg-turbo', options: jpgopt)
fzwoch commented 3 years ago

I know I could just disable ASM. But I hoped custom build rules could be tackled as well. We all know building software is hard. That is why these build systems exist.

At first I thought that meson's cmake module would just call cmake by itself and parse into it's target names for linking etc. But that seems not the case.

I guess it is back to shell scripts and find_library(). No hard feelings. But there is actually plenty of SSE/AVX/NEON software out there. And just disabling these optimizations often is not an acceptable workaround.

mensinda commented 3 years ago

We all know building software is hard. That is why these build systems exist.

True, and we are doing our best but at some point, there are also some limits for us. See our official statement on this topic.

I am not saying that it will never happen. I am also very optimistic that I could get it to work for libjpeg-turbo, however, that is specifically for libjpeg-turbo on my system (x86_64 Linux). The main problem would be that this opens a huge rabbit hole with all the other architectures, compilers, and operating systems.

ASM support would likely already be nearly complete if all projects would use add_custom_command() for compiling the assembler files since this is already supported and there is an equivalent function in Meson.

To me, adding assembler support looks like it would be a huge maintenance burden long term. Again, this is not a final decision and I am open to adding support if it turns out to be less of a giant hack than I fear it to be. But for now, your best option would be to install these projects locally and to use dependency() to find it (with a cmake_module_path).

At first I thought that meson's cmake module would just call cmake by itself and parse into it's target names for linking etc. But that seems not the case.

This is more or less what we are doing (parse the CMake output and generate a virtual meson.build), but this step is far from trivial. What we are doing is basically an on-the-fly conversion from CMake to Meson.

eli-schwartz commented 3 years ago

I guess this would be unblocked by #9379 which proposes to indeed add

native support for compiling *.asm files natively with something like executable('asmExe', 'file1.asm).

xclaesse commented 3 years ago

Note also that jpeg-turbo is in wrapdb but without asm.