Concurrency issues when using the same --output and parallel builds

[rojer]

Describe the bug

The most straightforward way to collect output from multiple build commands is of course to use --append with the same --output file. However, that doesn't appear to play nicely with parallel builds (make -jN). This is evidenced by presence of error messages like the following:

citnames: failed with: Failed to read file: [SNIP]/compile_commands.json, cause: [json.exception.parse_error.101] parse error at line 2664, column 31: syntax error while parsing value - invalid string: missing closing quote; last read: '"-D[SNIP]'

This suggests that the file semantic was reading was incomplete, most likely being written by another compile process.

To Reproduce Build a large project in parallel, with the same --output and --append in parallel.

Expected behavior Produced output is complete and consistent. My suggestion is to use advisory file locking (flock(2)) to serialize access to the output file.

Environment:

• OS name: Linux
• OS version: Ubuntu 22.04
• OS architecture: x86_64
• Bear version: 3.0.18
• Bear install method: from package, 3.0.18-1

Additional context

• Can you give us a reference to the project that you are running against this tool? No, unfortunately
• What build tools this project is using? make, no autotools
• What architecture you are trying to compile for? x86_64, no x-compilation
• Could you attach build logs? I'd rather not

Before you send...

• Have you read the README.md file or man bear? :heavy_check_mark:
• Have you looked what other open issues Bear has? :heavy_check_mark:
• Are you sure that the build works without Bear? :heavy_check_mark:

[rizsotto]

Hey @rojer , thanks for the report. Yes, Bear was not intended to protect the output file from multiple writes.

This project had a report similar to this: https://github.com/rizsotto/Bear/issues/509 Maybe take a read of that issue and that might help to find a workaround.

But I am also curious about your use case. Could you tell me more, how do you run Bear in parallel?