search

dmlc / XGBoost.jl

XGBoost Julia Package
Other
288 stars 111 forks source link

Occasional segfaults on m1 apple silicon #203

Open ericphanson opened 3 months ago

ericphanson commented 3 months ago

I was able to repeat it once with this script, but it fails to consistently reproduce:

using StableRNGs, MLJ, MLJXGBoostInterface, DataFrames
X = float.(rand(StableRNG(1), [0,0,0,0,0,1], 10000, 4));
y = rand(StableRNG(2), 0:1, 10000);
classifier = Pipeline(; standardizer=Standardizer(), classifier=XGBoostClassifier());
mach = machine(classifier, DataFrame(X, :auto), coerce(y, OrderedFactor))
fit!(mach)

Here's the time that it happened:

julia> using StableRNGs, MLJ, MLJXGBoostInterface, DataFrames

julia> X = float.(rand(StableRNG(1), [0,0,0,0,1], 10000, 4))
10000×4 Matrix{Float64}:
 0.0  1.0  0.0  1.0
 0.0  1.0  0.0  0.0
 1.0  1.0  0.0  1.0
 0.0  0.0  0.0  0.0
 0.0  1.0  0.0  0.0
 0.0  0.0  0.0  0.0
 ⋮              
 0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0
 0.0  1.0  0.0  0.0
 0.0  1.0  0.0  0.0
 0.0  0.0  0.0  0.0

julia> y = rand(StableRNG(2), 0:1, 10000);

julia> classifier = Pipeline(Standardizer(), XGBoostClassifier());

julia> mach = machine(classifier, DataFrame(X, :auto), coerce(y, OrderedFactor))
untrained Machine; does not cache data
  model: ProbabilisticPipeline(standardizer = Standardizer(features = Symbol[], …), …)
  args: 
    1:  Source @134 ⏎ Table{AbstractVector{Continuous}}
    2:  Source @800 ⏎ AbstractVector{OrderedFactor{2}}

julia> fit!(mach)
[ Info: Training machine(ProbabilisticPipeline(standardizer = Standardizer(features = Symbol[], …), …), …).
[ Info: Training machine(:standardizer, …).
[ Info: Training machine(:xg_boost_classifier, …).
[ Info: XGBoost: starting training.

[19107] signal (11.2): Segmentation fault: 11
in expression starting at REPL[53]:1
_ZN7xgboost4tree20CommonRowPartitioner14UpdatePositionIhLb0ELb0ENS0_14CPUExpandEntryEEEvPKNS_7ContextERKNS_16GHistIndexMatrixERKNS_6common12ColumnMatrixERKNSt3__16vectorIT2_NSE_9allocatorISG_EEEEPKNS_7RegTreeE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost4tree20CommonRowPartitioner14UpdatePositionINS0_14CPUExpandEntryEEEvPKNS_7ContextERKNS_16GHistIndexMatrixERKNSt3__16vectorIT_NSA_9allocatorISC_EEEEPKNS_7RegTreeE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost4tree11HistUpdater14UpdatePositionEPNS_7DMatrixEPKNS_7RegTreeERKNSt3__16vectorINS0_14CPUExpandEntryENS7_9allocatorIS9_EEEE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost4tree10UpdateTreeINS0_14CPUExpandEntryENS0_11HistUpdaterEEEvPNS_6common7MonitorENS_6linalg10TensorViewIKNS_6detail20GradientPairInternalIfEELi2EEEPT0_PNS_7DMatrixEPKNS0_10TrainParamEPNS_16HostDeviceVectorIiEEPNS_7RegTreeE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost4tree17QuantileHistMaker6UpdateEPKNS0_10TrainParamEPNS_16HostDeviceVectorINS_6detail20GradientPairInternalIfEEEEPNS_7DMatrixENS_6common4SpanINS5_IiEELm18446744073709551615EEERKNSt3__16vectorIPNS_7RegTreeENSH_9allocatorISK_EEEE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost3gbm6GBTree13BoostNewTreesEPNS_16HostDeviceVectorINS_6detail20GradientPairInternalIfEEEEPNS_7DMatrixEiPNSt3__16vectorINS2_IiEENSA_9allocatorISC_EEEEPNSB_INSA_10unique_ptrINS_7RegTreeENSA_14default_deleteISI_EEEENSD_ISL_EEEE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost3gbm6GBTree7DoBoostEPNS_7DMatrixEPNS_16HostDeviceVectorINS_6detail20GradientPairInternalIfEEEEPNS_20PredictionCacheEntryEPKNS_11ObjFunctionE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
_ZN7xgboost11LearnerImpl13UpdateOneIterEiNSt3__110shared_ptrINS_7DMatrixEEE at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
XGBoosterUpdateOneIter at /Users/eph/.julia/artifacts/0079d93a46694d4e5e45f4e0b6bd6a35e24f4346/lib/libxgboost.dylib (unknown line)
XGBoosterUpdateOneIter at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/Lib.jl:282 [inlined]
xgbcall at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/Lib.jl:25 [inlined]
#updateone!#72 at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:374
updateone! at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:369 [inlined]
#update!#77 at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:446
update! at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:429 [inlined]
#xgboost#82 at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:602
xgboost at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:579
unknown function (ip: 0x2a35bc457)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/gf.c:3076
fit at /Users/eph/.julia/packages/MLJXGBoostInterface/uFARS/src/MLJXGBoostInterface.jl:168
unknown function (ip: 0x2a34bc71f)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/./julia.h:1982 [inlined]
do_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/builtins.c:768
#fit_only!#57 at /Users/eph/.julia/packages/MLJBase/iIhiI/src/machines.jl:681
fit_only! at /Users/eph/.julia/packages/MLJBase/iIhiI/src/machines.jl:607 [inlined]
#fit_only!#62 at /Users/eph/.julia/packages/MLJBase/iIhiI/src/machines.jl:752
fit_only! at /Users/eph/.julia/packages/MLJBase/iIhiI/src/machines.jl:735 [inlined]
#80 at /Users/eph/.julia/packages/MLJBase/iIhiI/src/composition/learning_networks/nodes.jl:235
unknown function (ip: 0x2a2f4c0bb)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/./julia.h:1982 [inlined]
start_task at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/src/task.c:1238
Allocations: 41540566 (Pool: 41501312; Big: 39254); GC: 43
zsh: segmentation fault  julia --project

From that stacktrace, the last calls before we get into the C code are:

XGBoosterUpdateOneIter at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/Lib.jl:282 [inlined]
xgbcall at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/Lib.jl:25 [inlined]
#updateone!#72 at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:374
updateone! at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:369 [inlined]
#update!#77 at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:446
update! at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:429 [inlined]
#xgboost#82 at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:602
xgboost at /Users/eph/.julia/packages/XGBoost/nqMqQ/src/booster.jl:579

which is why I filed it here rather than on MLJXGBoostInterface.jl.

I'm on v2.5.1 of XGBoost.jl, Julia v1.10.2, 

julia> versioninfo()
Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, apple-m1)
Threads: 4 default, 0 interactive, 2 GC (on 4 virtual cores)
Environment:
  JULIA_NUM_THREADS = 4
  JULIA_PKG_SERVER_REGISTRY_PREFERENCE = eager
ericphanson commented 3 months ago

BTW in my actual code, I generally don't get a stacktrace, just

[18447] signal (11.2): Segmentation fault: 11
in expression starting at my_julia_code/fit.jl:42
Allocations: 243091803 (Pool: 242670917; Big: 420886); GC: 102
zsh: segmentation fault  julia --project

Edit: just got again, this time with maybe a bit more info, that there's a double-free:

julia(20662,0x325333000) malloc: Double free of object 0x2ad42abc0

[20662] signal (11.2): Segmentation fault: 11
in expression starting at more_julia_code/fit.jl:39
Allocations: 308241693 (Pool: 307688023; Big: 553670); GC: 114
julia(20662,0x325333000) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: segmentation fault  julia --project
ericphanson commented 3 months ago

I started julia under lldb and was able to repro, but I don't actually know what I'm doing, so not sure I found anything helpful.

Process 22988 stopped
* thread #22, stop reason = EXC_BAD_ACCESS (code=1, address=0x62e4bb3b0)
    frame #0: 0x00000002b2122d98 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::PushTemp() + 360
libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::PushTemp:
->  0x2b2122d98 <+360>: stur   x1, [x16, #-0x20]
    0x2b2122d9c <+364>: add    x4, x1, x15
    0x2b2122da0 <+368>: stur   x2, [x16, #-0x10]
    0x2b2122da4 <+372>: add    x1, x1, x12
Target 0: (julia) stopped.
(lldb) thread backtrace
* thread #22, stop reason = EXC_BAD_ACCESS (code=1, address=0x62e4bb3b0)
  * frame #0: 0x00000002b2122d98 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::PushTemp() + 360
    frame #1: 0x00000002b2122bb4 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::Push(float, float) + 372
    frame #2: 0x00000002b2133ab8 libxgboost.dylib`.omp_outlined..118 + 360
    frame #3: 0x0000000172ee6b0c libomp.dylib`__kmp_invoke_microtask + 156
    frame #4: 0x0000000172e85a0c libomp.dylib`__kmp_invoke_task_func + 332
    frame #5: 0x0000000172e84914 libomp.dylib`__kmp_launch_thread + 420
    frame #6: 0x0000000172ec7d58 libomp.dylib`__kmp_launch_worker(void*) + 280
    frame #7: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
(lldb) thread list
Process 22988 stopped
  thread #1: tid = 0xd2916d, 0x00000002b2123010 libxgboost.dylib`void std::__1::__sort<std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&, xgboost::common::WQSummary<float, float>::Queue::QEntry*>(xgboost::common::WQSummary<float, float>::Queue::QEntry*, xgboost::common::WQSummary<float, float>::Queue::QEntry*, std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&) + 224, queue = 'com.apple.main-thread'
  thread #2: tid = 0xd29188, 0x000000018146a060 libsystem_kernel.dylib`kevent + 8
  thread #3: tid = 0xd29189, 0x0000000181463f14 libsystem_kernel.dylib`mach_msg2_trap + 8
  thread #4: tid = 0xd2918a, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #5: tid = 0xd2918b, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #6: tid = 0xd2918c, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #7: tid = 0xd2918d, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #8: tid = 0xd2918e, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #9: tid = 0xd2918f, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #10: tid = 0xd29190, 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
  thread #13: tid = 0xd29232, 0x0000000181463f14 libsystem_kernel.dylib`mach_msg2_trap + 8
  thread #16: tid = 0xd2935f, 0x0000000181463f14 libsystem_kernel.dylib`mach_msg2_trap + 8
  thread #21: tid = 0xd2936f, 0x0000000000000000
* thread #22: tid = 0xd295b2, 0x00000002b2122d98 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::PushTemp() + 360, stop reason = EXC_BAD_ACCESS (code=1, address=0x62e4bb3b0)
  thread #23: tid = 0xd295b3, 0x00000002b212302c libxgboost.dylib`void std::__1::__sort<std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&, xgboost::common::WQSummary<float, float>::Queue::QEntry*>(xgboost::common::WQSummary<float, float>::Queue::QEntry*, xgboost::common::WQSummary<float, float>::Queue::QEntry*, std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&) + 252
  thread #24: tid = 0xd295b4, 0x00000001811a2484 dyld`dyld3::MachOLoaded::findClosestSymbol(unsigned long long, char const**, unsigned long long*) const + 552
(lldb) thread backtrace all
  thread #1, queue = 'com.apple.main-thread'
    frame #0: 0x00000002b2123010 libxgboost.dylib`void std::__1::__sort<std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&, xgboost::common::WQSummary<float, float>::Queue::QEntry*>(xgboost::common::WQSummary<float, float>::Queue::QEntry*, xgboost::common::WQSummary<float, float>::Queue::QEntry*, std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&) + 224
    frame #1: 0x00000002b2123248 libxgboost.dylib`void std::__1::__sort<std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&, xgboost::common::WQSummary<float, float>::Queue::QEntry*>(xgboost::common::WQSummary<float, float>::Queue::QEntry*, xgboost::common::WQSummary<float, float>::Queue::QEntry*, std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&) + 792
    frame #2: 0x00000002b2122b10 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::Push(float, float) + 208
    frame #3: 0x00000002b2133ab8 libxgboost.dylib`.omp_outlined..118 + 360
    frame #4: 0x0000000172ee6b0c libomp.dylib`__kmp_invoke_microtask + 156
    frame #5: 0x0000000172e85a0c libomp.dylib`__kmp_invoke_task_func + 332
    frame #6: 0x0000000172e81400 libomp.dylib`__kmp_fork_call + 7936
    frame #7: 0x0000000172e72fa4 libomp.dylib`__kmpc_fork_call + 180
    frame #8: 0x00000002b2133448 libxgboost.dylib`void xgboost::common::SketchContainerImpl<xgboost::common::WQuantileSketch<float, float> >::PushRowPageImpl<xgboost::data::SparsePageAdapterBatch, xgboost::common::SketchContainerImpl<xgboost::common::WQuantileSketch<float, float> >::PushRowPage(xgboost::SparsePage const&, xgboost::MetaInfo const&, xgboost::common::Span<float const, 18446744073709551615ul>)::'lambda'(auto)>(auto const&, unsigned long, xgboost::common::OptionalWeights, unsigned long, unsigned long, bool, xgboost::common::SketchContainerImpl<xgboost::common::WQuantileSketch<float, float> >::PushRowPage(xgboost::SparsePage const&, xgboost::MetaInfo const&, xgboost::common::Span<float const, 18446744073709551615ul>)::'lambda'(auto)) + 248
    frame #9: 0x00000002b2132b0c libxgboost.dylib`xgboost::common::SketchContainerImpl<xgboost::common::WQuantileSketch<float, float> >::PushRowPage(xgboost::SparsePage const&, xgboost::MetaInfo const&, xgboost::common::Span<float const, 18446744073709551615ul>) + 988
    frame #10: 0x00000002b20fdb24 libxgboost.dylib`xgboost::common::SketchOnDMatrix(xgboost::Context const*, xgboost::DMatrix*, int, bool, xgboost::common::Span<float const, 18446744073709551615ul>) + 3812
    frame #11: 0x00000002b2199708 libxgboost.dylib`xgboost::GHistIndexMatrix::GHistIndexMatrix(xgboost::Context const*, xgboost::DMatrix*, int, double, bool, xgboost::common::Span<float const, 18446744073709551615ul>) + 216
    frame #12: 0x00000002b21bcf30 libxgboost.dylib`xgboost::data::SimpleDMatrix::GetGradientIndex(xgboost::Context const*, xgboost::BatchParam const&) + 1168
    frame #13: 0x00000002b2350d20 libxgboost.dylib`xgboost::tree::HistUpdater::InitData(xgboost::DMatrix*, xgboost::RegTree const*) + 208
    frame #14: 0x00000002b232395c libxgboost.dylib`void xgboost::tree::UpdateTree<xgboost::tree::CPUExpandEntry, xgboost::tree::HistUpdater>(xgboost::common::Monitor*, xgboost::linalg::TensorView<xgboost::detail::GradientPairInternal<float> const, 2>, xgboost::tree::HistUpdater*, xgboost::DMatrix*, xgboost::tree::TrainParam const*, xgboost::HostDeviceVector<int>*, xgboost::RegTree*) + 156
    frame #15: 0x00000002b2322098 libxgboost.dylib`xgboost::tree::QuantileHistMaker::Update(xgboost::tree::TrainParam const*, xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*, xgboost::DMatrix*, xgboost::common::Span<xgboost::HostDeviceVector<int>, 18446744073709551615ul>, std::__1::vector<xgboost::RegTree*, std::__1::allocator<xgboost::RegTree*> > const&) + 1704
    frame #16: 0x00000002b21e83c4 libxgboost.dylib`xgboost::gbm::GBTree::BoostNewTrees(xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*, xgboost::DMatrix*, int, std::__1::vector<xgboost::HostDeviceVector<int>, std::__1::allocator<xgboost::HostDeviceVector<int> > >*, std::__1::vector<std::__1::unique_ptr<xgboost::RegTree, std::__1::default_delete<xgboost::RegTree> >, std::__1::allocator<std::__1::unique_ptr<xgboost::RegTree, std::__1::default_delete<xgboost::RegTree> > > >*) + 2356
    frame #17: 0x00000002b21e2548 libxgboost.dylib`xgboost::gbm::GBTree::DoBoost(xgboost::DMatrix*, xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*, xgboost::PredictionCacheEntry*, xgboost::ObjFunction const*) + 744
    frame #18: 0x00000002b21ff0d8 libxgboost.dylib`xgboost::LearnerImpl::UpdateOneIter(int, std::__1::shared_ptr<xgboost::DMatrix>) + 808
    frame #19: 0x00000002b20d6e98 libxgboost.dylib`XGBoosterUpdateOneIter + 152
    frame #20: 0x000000030a5e40b8
    frame #21: 0x000000030a45c504
    frame #22: 0x0000000101cc1ef4 libjulia-internal.1.10.2.dylib`do_apply [inlined] jl_apply(args=<unavailable>, nargs=<unavailable>) at julia.h:1982:12 [opt]
    frame #23: 0x0000000101cc1ee4 libjulia-internal.1.10.2.dylib`do_apply(args=0x00000003553ff9f8, nargs=<unavailable>, iterate=0x0000000127532040) at builtins.c:768:26 [opt]
    frame #24: 0x00000003215e4c34
    frame #25: 0x0000000101cd2efc libjulia-internal.1.10.2.dylib`start_task [inlined] jl_apply(args=<unavailable>, nargs=1) at julia.h:1982:12 [opt]
    frame #26: 0x0000000101cd2ef0 libjulia-internal.1.10.2.dylib`start_task at task.c:1238:19 [opt]
  thread #2
    frame #0: 0x000000018146a060 libsystem_kernel.dylib`kevent + 8
    frame #1: 0x0000000101d139e0 libjulia-internal.1.10.2.dylib`signal_listener(arg=<unavailable>) at signals-unix.c:747:27 [opt]
    frame #2: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #3
    frame #0: 0x0000000181463f14 libsystem_kernel.dylib`mach_msg2_trap + 8
    frame #1: 0x0000000181476240 libsystem_kernel.dylib`mach_msg2_internal + 80
    frame #2: 0x000000018146cb78 libsystem_kernel.dylib`mach_msg_overwrite + 604
    frame #3: 0x000000018146bf08 libsystem_kernel.dylib`mach_msg_server + 356
    frame #4: 0x0000000101d125fc libjulia-internal.1.10.2.dylib`mach_segv_listener(arg=<unavailable>) at signals-mach.c:104:15 [opt]
    frame #5: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #4
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000101d82390 libjulia-internal.1.10.2.dylib`uv_cond_wait(cond=0x00000001008cb5b8, mutex=0x00000001008cb578) at thread.c:883:7
    frame #3: 0x0000000101cfb2d0 libjulia-internal.1.10.2.dylib`ijl_task_get_next(trypoptask=0x0000000126898c50, q=0x000000010d150030, checkempty=0x00000001268a0940) at partr.c:509:17 [opt]
    frame #4: 0x0000000123f14728 sys.dylib`julia_poptask_75480 at task.jl:985
    frame #5: 0x0000000124ccfc40 sys.dylib`julia_wait_74761 at task.jl:994
    frame #6: 0x00000002b65f04f0
    frame #7: 0x0000000101cd1dfc libjulia-internal.1.10.2.dylib`jl_finish_task [inlined] jl_apply(args=<unavailable>, nargs=2) at julia.h:1982:12 [opt]
    frame #8: 0x0000000101cd1df4 libjulia-internal.1.10.2.dylib`jl_finish_task(t=0x000000034d35dc30) at task.c:320:13 [opt]
    frame #9: 0x0000000101cd2f74 libjulia-internal.1.10.2.dylib`start_task at task.c:1249:5 [opt]
  thread #5
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000101d82390 libjulia-internal.1.10.2.dylib`uv_cond_wait(cond=0x000000010400b3b8, mutex=0x000000010400b378) at thread.c:883:7
    frame #3: 0x0000000101cfb2d0 libjulia-internal.1.10.2.dylib`ijl_task_get_next(trypoptask=0x0000000126898c50, q=0x000000010d148030, checkempty=0x00000001268a0940) at partr.c:509:17 [opt]
    frame #4: 0x0000000123f14728 sys.dylib`julia_poptask_75480 at task.jl:985
    frame #5: 0x0000000124ccfc40 sys.dylib`julia_wait_74761 at task.jl:994
    frame #6: 0x00000002b65f04f0
    frame #7: 0x0000000101cd1dfc libjulia-internal.1.10.2.dylib`jl_finish_task [inlined] jl_apply(args=<unavailable>, nargs=2) at julia.h:1982:12 [opt]
    frame #8: 0x0000000101cd1df4 libjulia-internal.1.10.2.dylib`jl_finish_task(t=0x000000034d35ddc0) at task.c:320:13 [opt]
    frame #9: 0x0000000101cd2f74 libjulia-internal.1.10.2.dylib`start_task at task.c:1249:5 [opt]
  thread #6
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000101d82390 libjulia-internal.1.10.2.dylib`uv_cond_wait(cond=0x000000010480b3b8, mutex=0x000000010480b378) at thread.c:883:7
    frame #3: 0x0000000101cfb2d0 libjulia-internal.1.10.2.dylib`ijl_task_get_next(trypoptask=0x0000000126898c50, q=0x000000010d140030, checkempty=0x00000001268a0940) at partr.c:509:17 [opt]
    frame #4: 0x0000000123f14728 sys.dylib`julia_poptask_75480 at task.jl:985
    frame #5: 0x0000000124ccfc40 sys.dylib`julia_wait_74761 at task.jl:994
    frame #6: 0x00000002b65f04f0
    frame #7: 0x0000000101cd1dfc libjulia-internal.1.10.2.dylib`jl_finish_task [inlined] jl_apply(args=<unavailable>, nargs=2) at julia.h:1982:12 [opt]
    frame #8: 0x0000000101cd1df4 libjulia-internal.1.10.2.dylib`jl_finish_task(t=0x000000034d35df50) at task.c:320:13 [opt]
    frame #9: 0x0000000101cd2f74 libjulia-internal.1.10.2.dylib`start_task at task.c:1249:5 [opt]
  thread #7
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000101d82390 libjulia-internal.1.10.2.dylib`uv_cond_wait(cond=0x000000010202e1d0, mutex=0x000000010202e190) at thread.c:883:7
    frame #3: 0x0000000101cfab58 libjulia-internal.1.10.2.dylib`jl_gc_mark_threadfun(arg=<unavailable>) at partr.c:135:13 [opt]
    frame #4: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #8
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000120173804 libopenblas64_.dylib`blas_thread_server + 388
    frame #3: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #9
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000120173804 libopenblas64_.dylib`blas_thread_server + 388
    frame #3: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #10
    frame #0: 0x0000000181467710 libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x00000001814a4574 libsystem_pthread.dylib`_pthread_cond_wait + 1232
    frame #2: 0x0000000120173804 libopenblas64_.dylib`blas_thread_server + 388
    frame #3: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #13
    frame #0: 0x0000000181463f14 libsystem_kernel.dylib`mach_msg2_trap + 8
    frame #1: 0x0000000181476240 libsystem_kernel.dylib`mach_msg2_internal + 80
    frame #2: 0x000000018146cb78 libsystem_kernel.dylib`mach_msg_overwrite + 604
    frame #3: 0x0000000181464290 libsystem_kernel.dylib`mach_msg + 24
    frame #4: 0x00000001815827e4 CoreFoundation`__CFRunLoopServiceMachPort + 160
    frame #5: 0x00000001815810c4 CoreFoundation`__CFRunLoopRun + 1208
    frame #6: 0x00000001815804b8 CoreFoundation`CFRunLoopRunSpecific + 612
    frame #7: 0x000000018160557c CoreFoundation`CFRunLoopRun + 64
    frame #8: 0x0000000101d883cc libjulia-internal.1.10.2.dylib`uv__cf_loop_runner(arg=0x00000001022f04c0) at fsevents.c:734:3
    frame #9: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #16
    frame #0: 0x0000000181463f14 libsystem_kernel.dylib`mach_msg2_trap + 8
    frame #1: 0x0000000181476240 libsystem_kernel.dylib`mach_msg2_internal + 80
    frame #2: 0x000000018146cb78 libsystem_kernel.dylib`mach_msg_overwrite + 604
    frame #3: 0x0000000181464290 libsystem_kernel.dylib`mach_msg + 24
    frame #4: 0x00000001815827e4 CoreFoundation`__CFRunLoopServiceMachPort + 160
    frame #5: 0x00000001815810c4 CoreFoundation`__CFRunLoopRun + 1208
    frame #6: 0x00000001815804b8 CoreFoundation`CFRunLoopRunSpecific + 612
    frame #7: 0x000000018160557c CoreFoundation`CFRunLoopRun + 64
    frame #8: 0x0000000195da10d8 DebugSymbols`SpotlightQueryThread(void*) + 580
    frame #9: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #21
    frame #0: 0x0000000000000000
* thread #22, stop reason = EXC_BAD_ACCESS (code=1, address=0x62e4bb3b0)
  * frame #0: 0x00000002b2122d98 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::PushTemp() + 360
    frame #1: 0x00000002b2122bb4 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::Push(float, float) + 372
    frame #2: 0x00000002b2133ab8 libxgboost.dylib`.omp_outlined..118 + 360
    frame #3: 0x0000000172ee6b0c libomp.dylib`__kmp_invoke_microtask + 156
    frame #4: 0x0000000172e85a0c libomp.dylib`__kmp_invoke_task_func + 332
    frame #5: 0x0000000172e84914 libomp.dylib`__kmp_launch_thread + 420
    frame #6: 0x0000000172ec7d58 libomp.dylib`__kmp_launch_worker(void*) + 280
    frame #7: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #23
    frame #0: 0x00000002b212302c libxgboost.dylib`void std::__1::__sort<std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&, xgboost::common::WQSummary<float, float>::Queue::QEntry*>(xgboost::common::WQSummary<float, float>::Queue::QEntry*, xgboost::common::WQSummary<float, float>::Queue::QEntry*, std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&) + 252
    frame #1: 0x00000002b2123248 libxgboost.dylib`void std::__1::__sort<std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&, xgboost::common::WQSummary<float, float>::Queue::QEntry*>(xgboost::common::WQSummary<float, float>::Queue::QEntry*, xgboost::common::WQSummary<float, float>::Queue::QEntry*, std::__1::__less<xgboost::common::WQSummary<float, float>::Queue::QEntry, xgboost::common::WQSummary<float, float>::Queue::QEntry>&) + 792
    frame #2: 0x00000002b2122b10 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::Push(float, float) + 208
    frame #3: 0x00000002b2133ab8 libxgboost.dylib`.omp_outlined..118 + 360
    frame #4: 0x0000000172ee6b0c libomp.dylib`__kmp_invoke_microtask + 156
    frame #5: 0x0000000172e85a0c libomp.dylib`__kmp_invoke_task_func + 332
    frame #6: 0x0000000172e84914 libomp.dylib`__kmp_launch_thread + 420
    frame #7: 0x0000000172ec7d58 libomp.dylib`__kmp_launch_worker(void*) + 280
    frame #8: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148
  thread #24
    frame #0: 0x00000001811a2484 dyld`dyld3::MachOLoaded::findClosestSymbol(unsigned long long, char const**, unsigned long long*) const + 552
    frame #1: 0x000000018118215c dyld`dyld4::APIs::dladdr(void const*, dl_info*) + 212
    frame #2: 0x000000018134d974 libsystem_c.dylib`backtrace_symbols + 120
    frame #3: 0x00000002b20b6a24 libxgboost.dylib`dmlc::StackTrace(unsigned long, unsigned long) + 372
    frame #4: 0x00000002b20b680c libxgboost.dylib`dmlc::LogMessageFatal::~LogMessageFatal() + 124
    frame #5: 0x00000002b2124170 libxgboost.dylib`xgboost::common::WQSummary<float, float>::SetCombine(xgboost::common::WQSummary<float, float> const&, xgboost::common::WQSummary<float, float> const&) + 1296
    frame #6: 0x00000002b2122ea8 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::PushTemp() + 632
    frame #7: 0x00000002b2122bb4 libxgboost.dylib`xgboost::common::QuantileSketchTemplate<float, float, xgboost::common::WQSummary<float, float> >::Push(float, float) + 372
    frame #8: 0x00000002b2133ab8 libxgboost.dylib`.omp_outlined..118 + 360
    frame #9: 0x0000000172ee6b0c libomp.dylib`__kmp_invoke_microtask + 156
    frame #10: 0x0000000172e85a0c libomp.dylib`__kmp_invoke_task_func + 332
    frame #11: 0x0000000172e84914 libomp.dylib`__kmp_launch_thread + 420
    frame #12: 0x0000000172ec7d58 libomp.dylib`__kmp_launch_worker(void*) + 280
    frame #13: 0x00000001814a3fa8 libsystem_pthread.dylib`_pthread_start + 148