rizsotto
commented
6 years ago Thanks for the report. It will take some time to dig into this. (I'm not active FreeBSD user.) By the way, do you know any public CI server which runs BSD?
Closed umanwizard closed 4 years ago
rizsotto
commented
6 years ago Thanks for the report. It will take some time to dig into this. (I'm not active FreeBSD user.) By the way, do you know any public CI server which runs BSD?
umanwizard
commented
6 years ago I think this is because the system make is statically linked, so libear.so is useless. (i.e., the make binary will jump directly to its own statically linked execve etc. functions, rather than using the ones provided in libear.so ). Might be unfixable.
rizsotto
commented
6 years ago Yes, that is a problem to Bear. :) Can you point me the source code of make?
umanwizard
commented
6 years ago It is at /usr/src/contrib/bmake in the FreeBSD source tree.
But I don't think the source will tell you anything interesting. The relevant thing is the fact that it was linked with the -static flag. I'm not sure if this is true on all FreeBSD systems or just the one I have installed...
Anyway, one way to resolve this would be to simply print a warning if the make binary is statically linked, because in that case it is probably impossible to replace the execve it's using.
rizsotto
commented
6 years ago Thanks for the pointer. I agree, that static linking is the thing this case. But wanted to make sure that if the exec call is covered by Bear. (Just learned recently about execveat which I have not heard before and not covered.)
What shall we do with this ticket?
DavidPu
commented
6 years ago one idea is using strace instread of libear.so to capture exec call. it will fix some of 'Known issues' listed in README.
I have proof-of-concept change for it: https://github.com/DavidPu/Bear/commit/b977aca230b637b133e3c25b6af0b2d1969710f2
I'm not sure if any strace equivalent available for macOS?
rizsotto
commented
6 years ago Thanks for your message @DavidPu . I was looking at the ptrace calls to implement Bear, but then decided not to use them... It's not portable. It might not work within containers... For the statically linked binaries, I will use compiler wrappers. (Work in progress.)
j-rahman
commented
4 years ago The same happens on macOS 10.15.2 too. The JSON is empty if generated with the system make. The same happens with compiledb too. But after seeing this thread, I installed gmake and tried bear gmake and it worked like a charm. I don't think the empty JSON earlier had anything to do with system integrity protection because I haven't touched that switch at all.
wataash
commented
4 years ago Same with OpenBSD's /usr/bin/make, but seems to be another issue since it's dynamically linked:
# ldd /usr/bin/make
/usr/bin/make:
Start End Type Open Ref GrpRef Name
00000a27d4100000 00000a27d412a000 exe 1 0 0 /usr/bin/make
00000a2a5a8a9000 00000a2a5a8be000 rlib 0 1 0 /usr/lib/libutil.so.13.1
00000a2a407ea000 00000a2a408de000 rlib 0 1 0 /usr/lib/libc.so.95.1
00000a2a3cecb000 00000a2a3cecb000 ld.so 0 1 0 /usr/libexec/ld.soHey @wataash , could you send a verbose output of the run? (bear -vvvv make would do it.) This might be another issue. (Maybe statically linked compiler or something else.)
wataash
commented
4 years ago Thank you for replying @rizsotto .
I added | head because I'm compiling OpenBSD kernel, which emits too many outputs.
Empty compile_commands.json is generated even with cc:
root@wshob ~# cc --version
OpenBSD clang version 8.0.1 (tags/RELEASE_801/final) (based on LLVM 8.0.1)
Target: amd64-unknown-openbsd6.6
Thread model: posix
InstalledDir: /usr/bin
root@wshob ~# ldd ^C
root@wshob ~# which cc
/usr/bin/cc
root@wshob ~# cc --version
OpenBSD clang version 8.0.1 (tags/RELEASE_801/final) (based on LLVM 8.0.1)
Target: amd64-unknown-openbsd6.6
Thread model: posix
InstalledDir: /usr/bin
root@wshob ~# ldd /usr/bin/cc
/usr/bin/cc:
Start End Type Open Ref GrpRef Name
0000000000200000 00000000036e3000 exe 2 0 0 /usr/bin/cc
0000000228b78000 0000000228c54000 rlib 0 1 0 /usr/lib/libc++.so.3.0
00000002038cb000 000000020390d000 rlib 0 2 0 /usr/lib/libc++abi.so.1.0
00000002afdf0000 00000002afdfd000 rlib 0 1 0 /usr/lib/libpthread.so.26.1
00000002582b4000 00000002582e3000 rlib 0 1 0 /usr/lib/libm.so.10.1
00000002d0a16000 00000002d0b0a000 rlib 0 1 0 /usr/lib/libc.so.95.1
000000022ef0f000 000000022ef0f000 ld.so 0 1 0 /usr/libexec/ld.so
root@wshob ~# bear -vvvv cc a.c
bear: DEBUG: parse_args_for_intercept_build: Raw arguments ['/usr/local/bin/bear', '-vvvv', 'cc', 'a.c']
bear: DEBUG: parse_args_for_intercept_build: Parsed arguments: Namespace(append=False, build=['cc', 'a.c'], cdb='compile_commands.json', exclude=[], field_output=False, include=[], libear='/usr/local/lib/bear/libear.so', use_cc=['cc'], use_cxx=['c++'], use_fortran=['f95'], use_only=False, verbose=4)
bear: DEBUG: run_build: run build ['cc', 'a.c'], in environment: {'HOME': '/root', 'LC_CTYPE': 'en_US.UTF-8', 'LOGNAME': 'root', 'MAIL': '/var/mail/root', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/sbin:/usr/local/bin', 'PWD': '/root', 'SHELL': '/bin/ksh', 'SHLVL': '1', 'SSH_AUTH_SOCK': '/tmp/ssh-LhfgOaXF3v/agent.33309', 'SSH_CLIENT': '10.0.2.2 35478 22', 'SSH_CONNECTION': '10.0.2.2 35478 10.0.2.15 22', 'SSH_TTY': '/dev/ttyp1', 'TERM': 'xterm-256color', 'USER': 'root', 'INTERCEPT_BUILD_TARGET_DIR': '/tmp/intercept-xxnzn3cq', 'LD_PRELOAD': '/usr/local/lib/bear/libear.so'}
bear: DEBUG: run_build: build finished with exit code: 0
bear: DEBUG: parse_exec_trace: parse exec trace file: /tmp/intercept-xxnzn3cq/execution.iEI67A
bear: DEBUG: _split_command: input was: ['/usr/bin/cc', '-cc1', '-triple', 'amd64-unknown-openbsd6.6', '-emit-obj', '-mrelax-all', '-disable-free', '-disable-llvm-verifier', '-discard-value-names', '-main-file-name', 'a.c', '-mrelocation-model', 'pic', '-pic-level', '1', '-pic-is-pie', '-mthread-model', 'posix', '-mdisable-fp-elim', '-relaxed-aliasing', '-masm-verbose', '-mconstructor-aliases', '-munwind-tables', '-target-cpu', 'x86-64', '-target-feature', '+retpoline-indirect-calls', '-target-feature', '+retpoline-indirect-branches', '-dwarf-column-info', '-debugger-tuning=gdb', '-resource-dir', '/usr/lib/clang/8.0.1', '-fdebug-compilation-dir', '/root', '-ferror-limit', '19', '-fmessage-length', '112', '-fwrapv', '-D_RET_PROTECTOR', '-ret-protector', '-fobjc-runtime=gnustep', '-fdiagnostics-show-option', '-fno-builtin-malloc', '-fno-builtin-calloc', '-fno-builtin-realloc', '-fno-builtin-valloc', '-fno-builtin-free', '-fno-builtin-strdup', '-fno-builtin-strndup', '-o', '/tmp/a-32ef65.o', '-x', 'c', 'a.c', '-faddrsig']
bear: DEBUG: parse_exec_trace: parse exec trace file: /tmp/intercept-xxnzn3cq/execution.24GYVE
bear: DEBUG: _split_command: input was: ['/usr/bin/ld', '-e', '__start', '--eh-frame-hdr', '-Bdynamic', '-dynamic-linker', '/usr/libexec/ld.so', '-o', 'a.out', '/usr/bin/../lib/crt0.o', '/usr/bin/../lib/crtbegin.o', '-L/usr/bin/../lib', '-L/usr/lib', '/tmp/a-32ef65.o', '-lcompiler_rt', '-lc', '-lcompiler_rt', '/usr/bin/../lib/crtend.o']
root@wshob ~# cat compile_commands.json
[]⏎ root@wshob ~# Thanks @wataash to come back with this.
Yes, the compiler looks good to me. (As you can see, it was intercepted the child processes.) The bear -vvvv cc a.c won't make a non empty compilation database, even if it's working well. (bear -vvvv sh -c "cc a.c" is the way to intercept a single compilation. Or pass -MJ output.json flag to Clang.)
wataash
commented
4 years ago Thanks! Okay, I understood that bear cc doesn't make sense.
Here's bear -vvvv sh -c "cc a.c" output, just in case of anyone want to see:
root@wshob ~# bear -vvvv sh -c "cc a.c"
bear: DEBUG: parse_args_for_intercept_build: Raw arguments ['/usr/local/bin/bear', '-vvvv', 'sh', '-c', 'cc a.c']
bear: DEBUG: parse_args_for_intercept_build: Parsed arguments: Namespace(append=False, build=['sh', '-c', 'cc a.c'], cdb='compile_commands.json', exclude=[], field_output=False, include=[], libear='/usr/local/lib/bear/libear.so', use_cc=['cc'], use_cxx=['c++'], use_fortran=['f95'], use_only=False, verbose=4)
bear: DEBUG: run_build: run build ['sh', '-c', 'cc a.c'], in environment: {'HOME': '/root', 'LC_CTYPE': 'en_US.UTF-8', 'LOGNAME': 'root', 'MAIL': '/var/mail/root', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/sbin:/usr/local/bin', 'PWD': '/root', 'SHELL': '/bin/ksh', 'SHLVL': '1', 'SSH_AUTH_SOCK': '/tmp/ssh-vqAfZCzHjQ/agent.23339', 'SSH_CLIENT': '10.0.2.2 44664 22', 'SSH_CONNECTION': '10.0.2.2 44664 10.0.2.15 22', 'SSH_TTY': '/dev/ttyp0', 'TERM': 'xterm-256color', 'USER': 'root', 'INTERCEPT_BUILD_TARGET_DIR': '/tmp/intercept-y6b3affj', 'LD_PRELOAD': '/usr/local/lib/bear/libear.so'}
bear: DEBUG: run_build: build finished with exit code: 0
bear: DEBUG: parse_exec_trace: parse exec trace file: /tmp/intercept-y6b3affj/execution.8k45lC
bear: DEBUG: _split_command: input was: ['/usr/bin/cc', '-cc1', '-triple', 'amd64-unknown-openbsd6.6', '-emit-obj', '-mrelax-all', '-disable-free', '-disable-llvm-verifier', '-discard-value-names', '-main-file-name', 'a.c', '-mrelocation-model', 'pic', '-pic-level', '1', '-pic-is-pie', '-mthread-model', 'posix', '-mdisable-fp-elim', '-relaxed-aliasing', '-masm-verbose', '-mconstructor-aliases', '-munwind-tables', '-target-cpu', 'x86-64', '-target-feature', '+retpoline-indirect-calls', '-target-feature', '+retpoline-indirect-branches', '-dwarf-column-info', '-debugger-tuning=gdb', '-resource-dir', '/usr/lib/clang/8.0.1', '-fdebug-compilation-dir', '/root', '-ferror-limit', '19', '-fmessage-length', '213', '-fwrapv', '-D_RET_PROTECTOR', '-ret-protector', '-fobjc-runtime=gnustep', '-fdiagnostics-show-option', '-fno-builtin-malloc', '-fno-builtin-calloc', '-fno-builtin-realloc', '-fno-builtin-valloc', '-fno-builtin-free', '-fno-builtin-strdup', '-fno-builtin-strndup', '-o', '/tmp/a-73d21b.o', '-x', 'c', 'a.c', '-faddrsig']
bear: DEBUG: parse_exec_trace: parse exec trace file: /tmp/intercept-y6b3affj/execution.yz68hY
bear: DEBUG: _split_command: input was: ['/usr/bin/ld', '-e', '__start', '--eh-frame-hdr', '-Bdynamic', '-dynamic-linker', '/usr/libexec/ld.so', '-o', 'a.out', '/usr/bin/../lib/crt0.o', '/usr/bin/../lib/crtbegin.o', '-L/usr/bin/../lib', '-L/usr/lib', '/tmp/a-73d21b.o', '-lcompiler_rt', '-lc', '-lcompiler_rt', '/usr/bin/../lib/crtend.o']
root@wshob ~# cat compile_commands.json
[]⏎ root@wshob ~#Update I made an issue about OpenBSD's problem at https://github.com/rizsotto/Bear/issues/283
rizsotto
commented
4 years ago @wataash thanks for the update. Since I don't yet have OpenBSD I need your help to fix this. (If it's fixable.)
sh dynamically liked?bear -vvvv sh -c "env && cc a.c". That would dump the environment variables that bear set up.
phil-blain
commented
4 years ago Quick note : Cirrus CI has FreeBSD VMs.
rizsotto
commented
4 years ago There is a release candidate on master branch now, which suppose to fix this issue. Could you give it a try for that?
I'm closing this ticket now, but will re-open if the bug still there.
If you attach log output, please consider to run Bear with
-vvvvto have debug logs available.Expected Behavior
bearworks with bothmakeandgmakeActual Behavior
bearonly works properly withgmake; the db generated with base systemmakeis empty.Environment