search

JuliaLang / julia

The Julia Programming Language
https://julialang.org/
MIT License
45.79k stars 5.49k forks source link

Repeated calls to LibGit2.fetch segfaults on FreeBSD 11.1 #23328

Closed ararslan closed 6 years ago

ararslan commented 7 years ago

I upgraded my FreeBSD box from 11.0 (which is what the FreeBSD CI workers are running) to 11.1, and now I'm consistently getting segfaults in the Pkg tests:

$ JULIA_CPU_CORES=2 JULIA_TEST_MAXRSS_MB=600 ./julia test/runtests.jl pkg
Test (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
INFO: Initializing package repository /tmp/T22S8Ijy/v0.7
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
INFO: No packages to install, update or remove
INFO: Cloning cache of Example from notarealprotocol://github.com/JuliaLang/Example.jl.git
INFO: Cloning cache of Example from https://github.com/JuliaLang/Example.jl.git
INFO: Installing Example v0.4.1
INFO: Package database updated
INFO: Checking out Example master...
INFO: Pulling Example latest master...
INFO: No packages to install, update or remove
INFO: Freeing Example
INFO: No packages to install, update or remove
INFO: Checking out Example master...
INFO: Pulling Example latest master...
INFO: No packages to install, update or remove
INFO: Freeing Example
INFO: No packages to install, update or remove
INFO: Removing Example v0.4.1
INFO: Package database updated
INFO: Package Example is not installed
INFO: Cloning Example from https://github.com/JuliaLang/Example.jl.git
INFO: Computing changes...
INFO: No packages to install, update or remove
INFO: Package database updated
INFO: Freeing Example
INFO: No packages to install, update or remove
INFO: Checking out Example master...
INFO: Pulling Example latest master...
INFO: No packages to install, update or remove
INFO: Freeing Example
INFO: No packages to install, update or remove
INFO: Cloning Example2 from /tmp/T22S8Ijy/v0.7/Example
INFO: Computing changes...
INFO: No packages to install, update or remove
INFO: Cloning Example3 from /tmp/T22S8Ijy/v0.7/Example
INFO: Computing changes...
INFO: No packages to install, update or remove
INFO: Checking out Example2 test-branch-1...
INFO: Pulling Example2 latest test-branch-1...
INFO: No packages to install, update or remove
INFO: Checking out Example3 test-branch-1...
INFO: Pulling Example3 latest test-branch-1...
INFO: No packages to install, update or remove
INFO: Checking out Example master...
INFO: Pulling Example latest master...
INFO: No packages to install, update or remove
INFO: Cloning Example4 from /tmp/T22S8Ijy/v0.7/Example
INFO: Computing changes...
INFO: No packages to install, update or remove
INFO: Checking out Example4 test-branch-2...
INFO: Pulling Example4 latest test-branch-2...
INFO: No packages to install, update or remove
[1]    2356 segmentation fault (core dumped)  JULIA_CPU_CORES=2 JULIA_TEST_MAXRSS_MB=600 ./julia test/runtests.jl pkg

Version info:

julia> versioninfo()
Julia Version 0.7.0-DEV.1383
Commit d126c66a9e* (2017-08-18 16:00 UTC)
Platform Info:
  OS: FreeBSD (x86_64-unknown-freebsd11.1)
  CPU: Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, sandybridge)
Environment:

cc @iblis17

ararslan commented 7 years ago

Running it in GDB, I'm seeing a lot of dwarf errors regarding "wrong version in compilation unit header," as well as

Program received signal SIGSEV, Segmentation fault.
0x0000000801571c19 in ?? () /usr/local/lib/gcc5/libgcc_s.so.1

at the end. Full log here: https://gist.github.com/ararslan/a9694acc54da13f633edde6aa7230a59

It also left a core file in the package directory where the tests were running. Examining that with GDB, I get

(gdb) core julia.core
Core was generated by `./julia test/runtests.jl pkg'.
Program was terminated with signal 11, Segmentation fault.
#0  0x0000000803e75907 in ?? ()
(gdb) bt
Cannot access memory at address 0x7fffdfbfdd80
iblislin commented 7 years ago

your gdb version is too old, please try out devel/gdb?

On Sat, Aug 19, 2017 at 12:00:57AM +0000, Alex Arslan wrote:

Running it in GDB, I'm seeing a lot of dwarf errors regarding "wrong version in compilation unit header," as well as

Program received signal SIGSEV, Segmentation fault.
0x0000000801571c19 in ?? () /usr/local/lib/gcc5/libgcc_s.so.1

at the end. Full log here: https://gist.github.com/ararslan/a9694acc54da13f633edde6aa7230a59

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/JuliaLang/julia/issues/23328#issuecomment-323485385

-- Iblis Lin ζž—ε³»ι €

ararslan commented 7 years ago

Wow, the system GDB is way out of date, no wonder they're going to remove it. Okay, I tried again with the ports GDB. The output is much more informative, but it looks like it's stopping on something else; the test program doesn't get nearly as far before stopping, and it doesn't appear to be hitting a segfault in GDB. The full log is here: https://gist.github.com/ararslan/4347c22e925e98f5fcacda5b438854a9. Could be I'm misusing GDB somehow, so if anything about the session in the log looks fishy let me know and I can try it another way.

ararslan commented 7 years ago

Using LLDB rather than GDB looks like it hits the right thing: https://gist.github.com/ararslan/acc5e587400affd93dff5e41f5dc1cda. (System LLDB, 4.0.0 in FreeBSD 11.1.)

ararslan commented 7 years ago

Looks like it's this line that's hitting it: https://github.com/JuliaLang/julia/blob/master/test/pkg.jl#L298

@test_warn "INFO: Package Example: skipping update (pinned)..." Pkg.update()

I haven't been able to minimally reduce the issue yet though.

iblislin commented 7 years ago

Found a workaround, can you confirm?:

# sysctl security.bsd.stack_guard_page=0

It appears that this option is enabled in FreeBSD 11.1-Release by default.

ararslan commented 7 years ago

Interesting, it does appear that stack_guard_page is 0 in 11.0 and 1 in 11.1, though that isn't mentioned in the 11.1 release notes. Disabling the stack guard page allows the Pkg tests to pass without segfaulting.

iblislin commented 7 years ago

not in release notes, but in wiki: https://wiki.freebsd.org/WhatsNew/FreeBSD11#Security

The stack protector is now set to strong (r288669)

https://svnweb.freebsd.org/base?view=revision&revision=288669

Edit: seems unrelated

-- I found this commit https://svnweb.freebsd.org/base?view=revision&revision=215307

this commit enable it by default https://svnweb.freebsd.org/base?view=revision&revision=320317

ararslan commented 7 years ago

Okay, even better LLDB backtrace: https://gist.github.com/ararslan/62d5dfb03b529a56dce0ab5d239685d8. In particular, it shows libgit2 calls in thread backtrace all starting on line 112.

Edit: Buuuuuuuuut that call is in the wrong thread. :/ Thread 6 in the above gist hits the SIGSEGV.

iblislin commented 7 years ago

Similar issue? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221127#c12

I'm going to give flang a try

ararslan commented 7 years ago

I doubt it's an issue with conflicting libgcc_s between Clang and GCC since we have the build system set up on FreeBSD to link everything to GCC's libgcc_s. That's what I implemented in #21788. It must be an issue with the stack guard setting, since the tests pass with it disabled and segfault with it enabled.

iblislin commented 7 years ago

on 11.0-RELEASE-p12 the test-pkg passed

root@:~/julia # sysctl security.bsd.stack_guard_page
security.bsd.stack_guard_page: 1
root@:~/julia # uname -a
FreeBSD  11.0-RELEASE-p12 FreeBSD 11.0-RELEASE-p12 #0: Wed Aug  9 10:03:39 UTC 2017     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

I think it's releated to r320317. It does change something besides setting stack guard as default; 11.0-RELEASE doesn't include this patch, so test-pkg passed with stack guard.

ararslan commented 7 years ago

Excellent catch!

iblislin commented 7 years ago

Ok, maybe we should create an minimal example, and send it to FreeBSD CURRENT mailing list. The problematic part is libgit2 (perhaps plus unwind) right? I'm going to start from building simple c code.

ararslan commented 7 years ago

I'm not sure the problem is libgit2; in the LLDB backtrace that contains libgit2 calls, the libgit2 calls are in a different thread than the one that hits the SIGSEGV. Do you know how to reproduce this minimally? I'm still trying to figure out what in Julia is actually triggering it.

iblislin commented 7 years ago

πŸ˜–
I managed to remove some test case in test/pkg.jl but that will make testing pass with 11.1's stack guard. The condition to triggering SIGSEGV seems quite tricky.

ararslan commented 7 years ago

Which tests did you have to remove in test/pkg.jl to get the tests passing? Can you post a diff?

iblislin commented 7 years ago

please checkout this: https://gist.github.com/iblis17/e2199735aee9673585da8aa48e5d4984

In the original pkg.jl, the explosion point of SIGSEGV is near line 300, IIRC. when I comment out some test cases like my gist, the test-pkg will pass. πŸ˜•

iblislin commented 7 years ago

For record, segfault happened on -CURRENT with stack guard enabled, also.

└─[iblis@abeing]% uname -a
FreeBSD abeing 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r323335: Sun Sep 17 00:56:35 CST 2017     root@abeing:/usr/obj/usr/src/sys/GENERIC  amd64
iblislin commented 7 years ago

another reproduce steps:

  1. git clone https://github.com/JuliaLang/julia.git repo
  2. ~cd repo && git reset --hard 67731c2c07 (It's just HEAD~500)~ edit: ignore this
  3. cat test.jl 
    for i ∈ 1:100
info(i)
LibGit2.fetch(LibGit2.GitRepo("./repo"))
end
  4. ./julia test.jl 
    └─[iblis@abeing]% ./julia test.jl
INFO: 1                           
INFO: 2                           
INFO: 3                           
INFO: 4                           
INFO: 5                           
INFO: 6                           
zsh: segmentation fault (core dumped)  ./julia test.jl
ararslan commented 7 years ago

Good sleuthing! If you write the same thing in C using the functions from libgit2, do you also get a segfault? Also, why is the git reset necessary? Is it just so that HEAD doesn't point to the latest commit or is there something specific about 67731c2c07?

iblislin commented 7 years ago

I checked it again, git reset is unnecessary. I try to invoke libgit2 api in C: https://gist.github.com/iblis17/bbc621a78fda6ffcbca077fadba8ecdd#file-git2_fetch-c but cannot get segfault.

ararslan commented 7 years ago

I was able to reduce it to a single ccall:

import Base.LibGit2: GitRepo, GitRemote, RemoteCallbacks, CredentialPayload,
                     StrArrayStruct, FetchOptions, get, credentials_cb

repo = GitRepo("./repo")
rmt = get(GitRemote, repo, "origin")
fo = FetchOptions(callbacks=RemoteCallbacks(credentials_cb(), CredentialPayload()))

for i = 1:100
    info(i)
    ccall((:git_remote_fetch, :libgit2), Cint,
          (Ptr{Void}, Ptr{StrArrayStruct}, Ptr{FetchOptions}, Cstring),
          rmt.ptr, C_NULL, Ref(fo), "hi")
end

close(rmt)
close(repo)

For me it consistently faults on the sixth call, as it is in your example output above.

StefanKarpinski commented 7 years ago

Is it possible that one of the structs has the wrong specification?

ararslan commented 7 years ago

I compared our implementations in base/libgit2/types.jl to the documentation of the corresponding types in libgit2 and they seem to match as far as I can tell, but there may be some subtle difference that I'm missing.

ararslan commented 7 years ago

I'm going to call this a Julia bug since I haven't been able to reproduce it with C or Rust's libgit2 bindings.

ararslan commented 7 years ago

@omus has noted that this seems only to occur when using multiple cores. That is, setting JULIA_CPU_CORES=1 avoids segfaulting.

ararslan commented 7 years ago

Hm, setting JULIA_CPU_CORES=1 does not avoid segfaulting for me. I guess it's only when the actual VM is set to only use one core.

wildart commented 7 years ago

My guess that something happens during the credentials_cb call. It would be hard to track (it is Julia-C-Julia call) and it would segfault without any relevant information.

ararslan commented 7 years ago

Why would something inside credentials_cb, which is outside of the loop, cause the repeated ccall to git_remote_fetch to segfault? You mean like the FetchOptions ends up being constructed incorrectly or something, and messes things up when being passed back and forth between Julia and C?

ararslan commented 7 years ago

Okay, updated LLDB output on latest master with the above git_remote_fetch in a loop: https://gist.github.com/ararslan/3eb7df6f83d21242d5e6d53719ff2efc

ararslan commented 7 years ago

LLDB backtrace on FreeBSD 12.0-CURRENT with security.bsd.stack_guard_page=1, built with DISABLE_LIBUNWIND=1: https://gist.github.com/iblis17/b9eb213150b1da48a46c460bd310187b

ararslan commented 7 years ago

Plot twist: libgit2 may be a red herring. Setting USE_SYSTEM_CURL=1 allows things to work fine without segfaulting. I tried applying all of the curl patches in the Ports tree to our curl but that didn't do it. A difference between the system curl and ours is that the system's is built with OpenSSL while ours is built with mbedTLS, so I'm inclined to think that could be related.

iblislin commented 7 years ago

I tried copying my /usr/local/lib/libcurl.so to julia/usr/lib/, but I still got segfault. :/

ararslan commented 7 years ago

Using the system curl with USE_SYSTEM_CURL=1 no longer fixes this for me.

ararslan commented 7 years ago

Never mind, it doesβ€”I just had to rebuild libgit2 and mbedTLS.

ararslan commented 6 years ago

I'm not getting a segfault with a stock build of Julia on FreeBSD 12.0-CURRENT, built from source at r326614 with the GENERIC-NODEBUG kernel and MALLOC_PRODUCTION. The stack guard page is enabled.

ararslan commented 6 years ago

I just tried current master on FreeBSD 11.1 and it didn't segfault. I'm doing a git clean -fdx and rebuilding from scratch to make sure it's reproducible.

ararslan commented 6 years ago

Neither Iblis nor I can reproduce this on 11.1 with current master, so I'm going to close this and call it resolved. I have no idea what changed in Base that made it work, though I might bisect it out of curiosity.