search

mesonbuild / meson

The Meson Build System
http://mesonbuild.com
Apache License 2.0
5.56k stars 1.61k forks source link

generator_obj.process() + library() == race condition #6677

Open denizzzka opened 4 years ago

denizzzka commented 4 years ago

Project with ~1000 source files (https://github.com/denizzzka/d_c_arm_test) It contains some generator and uses it for parsing .dpp files into .d (D language).

Usually ninja run fails - compiler complains what generated file gpio.d used by app.d isn't found (library() function what uses app.d also uses process() result what creates gpio.d). But then after ninja exit this file gpio.d can be found at appropriate place. And if I run ninja again build always sucessfull.

Or you just can run single-process ninja -j 1 and build always will be sucessfull.

Here is piece of generated ninja.build related to file gpio.d what becomes not found sometimes. I only yesterday began to learn ninja syntax but it looks like here is no any dependency what should inform compiler about fact what generator is done its job and app.d compilation (what assumes gpio.d inclusion through -I) can be started:

build d/18ac3e7@@D$ code$ part@sta/gpio.d: CUSTOM_COMMAND /home/denizzz/Dev/d_c_arm_test/d/source/bindings/gpio.dpp | /home/denizzz/Dev/d_c_arm_test/subprojects/dpp/dpp_helper.sh
 DESC = Generating$ 'd/18ac3e7@@D$ code$ part@sta/gpio.d'.
 COMMAND = /home/denizzz/Dev/d_c_arm_test/subprojects/dpp/dpp_helper.sh /home/denizzz/Dev/d_c_arm_test/d/source/bindings/gpio.dpp 'd/18ac3e7@@D$ code$ part@sta/gpio.dpp' --preprocess-only --include-path=/home/denizzz/Dev/d_c_arm_test/subprojects/libopencm3/include/ --define=STM32F1

build d/18ac3e7@@D$ code$ part@sta/meson-generated_gpio.d.o: d_COMPILER d/18ac3e7@@D$ code$ part@sta/gpio.d
 DEPFILE = d/18ac3e7@@D$ code$ part@sta/meson-generated_gpio.d.o.deps
 ARGS = '-I=d/18ac3e7@@D$ code$ part@sta' -I=d -I=../d -I=../d/source -I=../d/source/bindings -enable-color -wi -g -d-debug --mtriple=arm-none-eabi -mcpu=cortex-m3 -march=thumb -relocation-model=pic

build d/18ac3e7@@D$ code$ part@sta/source_app.d.o: d_COMPILER ../d/source/app.d
 DEPFILE = d/18ac3e7@@D$ code$ part@sta/source_app.d.o.deps
 ARGS = '-I=d/18ac3e7@@D$ code$ part@sta' -I=d -I=../d -I=../d/source -I=../d/source/bindings -enable-color -wi -g -d-debug --mtriple=arm-none-eabi -mcpu=cortex-m3 -march=thumb -relocation-model=pic

build d/libD$ code$ part.a: STATIC_LINKER d/18ac3e7@@D$ code$ part@sta/meson-generated_gpio.d.o d/18ac3e7@@D$ code$ part@sta/meson-generated_freertos.d.o d/18ac3e7@@D$ code$ part@sta/source_app.d.o
 LINK_ARGS = csrD

Generator:

dpp_parser = generator(
    helper,
    arguments : [
        '@INPUT@',
        '@BUILD_DIR@/@BASENAME@.dpp',
        '--preprocess-only',
        '@EXTRA_ARGS@',
    ],
    output: '@BASENAME@.d',
    preserve_path_from: meson.current_source_dir(),
)

Parser and its result usage:

libopencm3_binding = dpp.process(
    meson.current_source_dir() + '/source/bindings/gpio.dpp',
    extra_args: ['--include-path=' + subprojects_dir + 'libopencm3/include/', '--define=STM32F1']
)

d_dep = library(
    'D code part',
    [
        'source/app.d',
        libopencm3_binding,
        freertos_binding,
    ],
    include_directories: ['source', 'source/bindings'],
)

system parameters

Rogni commented 4 years ago

https://github.com/Rogni/MesonDBug 

(apps) mak@mak-HP-ProBook-450-G5 ~/Desktop/Dlang/RaceBug $ meson build && ninja -C build -j2
The Meson build system
Version: 0.53.0
Source dir: /home/mak/Рабочий стол/Dlang/RaceBug
Build dir: /home/mak/Рабочий стол/Dlang/RaceBug/build
Build type: native build
Project name: RaceBugTest
Project version: undefined
Host machine cpu family: x86_64
Host machine cpu: x86_64
D compiler for the host machine: ldc2 (llvm 1.8.0 "LDC - the LLVM D compiler (1.8.0):")
D linker for the host machine: ldc2 GNU ld.bfd 2.30
Program python3 found: YES (/home/mak/projects/venvs/apps/bin/python3)
Build targets in project: 1

Found ninja-1.9.0.git.kitware.dyndep-1.jobserver-1 at /home/mak/projects/venvs/apps/bin/ninja
ninja: Entering directory `build'
[1/4] Compiling D object 'program@exe/d_main.d.o'.
FAILED: program@exe/d_main.d.o 
ldc2 -I=program@exe -I=. -I=.. -enable-color -wi -g -d-debug -of='program@exe/d_main.d.o' -c ../d/main.d
../d/main.d(1): Error: module test_struct is in file 'test_struct.d' which cannot be read
import path[0] = /usr/include/d
import path[1] = /usr/lib/ldc/x86_64-linux-gnu/include/d/ldc
import path[2] = /usr/lib/ldc/x86_64-linux-gnu/include/d
import path[3] = program@exe
import path[4] = .
import path[5] = ..
[2/4] Generating 'program@exe/test_struct.d'.
ninja: build stopped: subcommand failed.
(apps) mak@mak-HP-ProBook-450-G5 ~/Desktop/Dlang/RaceBug $ ninja -C build -j2
ninja: Entering directory `build'
[3/3] Linking target program.

reproduced for D lang. in cpp it's ok

Rogni commented 4 years ago

tried use generated sources as dependency - same problem. how to use generated sources?

generated/meson.build

python = import('python').find_installation('python3')

gen = generator(python,
                output  : '@BASENAME@.d',
                arguments : [ 
                    meson.source_root()/'gen.py', 
                    '@INPUT@', 
                    '@OUTPUT@',
                    'd'
                ])

gen_src = gen.process(meson.source_root()/'test_struct.json')

generated_lib_dep = declare_dependency(
        sources: gen_src
    )

meson.build

add_languages('d')

subdir('generated')

executable('program_d', 
           ['main.d'],
           dependencies: [generated_lib_dep]
           )

log

(apps) mak@mak-HP-ProBook-450-G5 ~/Desktop/Dlang/RaceBug $ rm build -rf && meson build && ninja -C build -j8
The Meson build system
Version: 0.53.1
Source dir: /home/mak/Рабочий стол/Dlang/RaceBug
Build dir: /home/mak/Рабочий стол/Dlang/RaceBug/build
Build type: native build
Project name: RaceBugTest
Project version: undefined
Host machine cpu family: x86_64
Host machine cpu: x86_64
D compiler for the host machine: ldc2 (llvm 1.8.0 "LDC - the LLVM D compiler (1.8.0):")
D linker for the host machine: ldc2 GNU ld.bfd 2.30
Program python3 found: YES (/home/mak/projects/venvs/apps/bin/python3)
Build targets in project: 1

Found ninja-1.9.0.git.kitware.dyndep-1.jobserver-1 at /home/mak/projects/venvs/apps/bin/ninja
ninja: Entering directory `build'
[1/4] Compiling D object 'd/18ac3e7@@program_d@exe/main.d.o'.
FAILED: d/18ac3e7@@program_d@exe/main.d.o 
ldc2 -I=d/18ac3e7@@program_d@exe -I=d -I=../d -enable-color -wi -g -d-debug -of='d/18ac3e7@@program_d@exe/main.d.o' -c ../d/main.d
../d/main.d(3): Error: module test_struct is in file 'test_struct.d' which cannot be read
import path[0] = /usr/include/d
import path[1] = /usr/lib/ldc/x86_64-linux-gnu/include/d/ldc
import path[2] = /usr/lib/ldc/x86_64-linux-gnu/include/d
import path[3] = d/18ac3e7@@program_d@exe
import path[4] = d
import path[5] = ../d
[2/4] Generating 'd/18ac3e7@@program_d@exe/test_struct.d'.
ninja: build stopped: subcommand failed.
(apps) mak@mak-HP-ProBook-450-G5 ~/Desktop/Dlang/RaceBug $ ninja -C build -j8
ninja: Entering directory `build'
[3/3] Linking target d/program_d.
denizzzka commented 4 years ago

This problem is still here, catched it again.

denizzzka commented 3 years ago

This problem is still here

upd: problem still here on ~master ee46c0242f97501672c0bd6312498c464370cbeb at 4 Jul 2021

denizzzka commented 3 years ago

Or you just can run single-process ninja -j 1 and build always will be sucessfull.

No, and in case of "-j1" there may be same error. There is no way to get rid of this issue - only a few launches with different -j parameters can help to override it.

Please add "bug" label