CUDA PC2 on Tesla V100S panic

Checklist

[X] This is not a security-related bug/issue. If it is, please follow please follow the security policy.
[X] This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
[X] This is not a new feature request. If it is, please file a feature request instead.
[X] This is not an enhancement request. If it is, please file a improvement suggestion instead.
[X] I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
[X] I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
[X] I did not make any code changes to lotus.

Lotus component

[ ] lotus daemon - chain sync
[ ] lotus miner - mining and block production
[X] lotus miner/worker - sealing
[ ] lotus miner - proving(WindowPoSt)
[ ] lotus miner/market - storage deal
[ ] lotus miner/market - retrieval deal
[ ] lotus miner/market - data transfer
[ ] lotus client
[ ] lotus JSON-RPC API
[ ] lotus message management (mpool)
[ ] Other

Lotus Version

lotus-worker version 1.13.0-rc2+mainnet+git.530011fd0

Describe the Bug

I've compiled lotus-worker with FIL_PROOFS_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70" to target Tesla V100. After this I'm unable to do PC2 because of panic error :)

Logging Information

Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.027 INFO filcrypto::proofs::api > seal_pre_commit_phase2: start
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO filecoin_proofs::api > validate_cache_for_precommit_phase2:start
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO filecoin_proofs::api > validate_cache_for_precommit_phase2:finish
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO filecoin_proofs::api::seal > seal_pre_commit_phase2:start
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO storage_proofs_porep::stacked::vanilla::proof > replicate_phase2
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO storage_proofs_porep::stacked::vanilla::proof > Building trees [295536 descriptors max available]
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO storage_proofs_porep::stacked::vanilla::proof > generating tree c using the GPU
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.028 INFO storage_proofs_porep::stacked::vanilla::proof > Building column hashes
Oct 12 16:41:40  lotus-worker[73229]: 2021-10-12T16:41:40.048 INFO neptune::proteus::program > Using kernel on CUDA.
Oct 12 16:41:40  lotus-worker[73229]: thread 'worker-thread-2' panicked at 'failed to create tree gpu batcher: GpuError("Cuda(NoBinaryForGpu)")', .cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-10.0.0/src/stacked/vanilla/proof.rs:596:30
Oct 12 16:41:40  lotus-worker[73229]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Oct 12 16:41:40  lotus-worker[73229]: thread '<unnamed>' panicked at 'failed to receive base_data, tree_data for tree_c: RecvError', .cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-10.0.0/src/stacked/vanilla/proof.rs:652:26
Oct 12 16:41:40  lotus-worker[73229]: thread 'worker-thread-1' panicked at 'failed to send columns: SendError { .. }', .cargo/registry/src/github.com-1ecc6299db9ec823/storage-proofs-porep-10.0.0/src/stacked/vanilla/proof.rs:588:34
Oct 12 16:41:40  lotus-worker[73229]: thread '<unnamed>' panicked at 'Worker Pool was poisoned', .cargo/registry/src/github.com-1ecc6299db9ec823/yastl-0.1.2/src/wait.rs:50:13
Oct 12 16:41:40  lotus-worker[73229]: stack backtrace:
Oct 12 16:41:40  lotus-worker[73229]:    0:          0x20bbbf0 - std::backtrace_rs::backtrace::libunwind::trace::hdcf4f90f85129e83
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
Oct 12 16:41:40  lotus-worker[73229]:    1:          0x20bbbf0 - std::backtrace_rs::backtrace::trace_unsynchronized::h2669e30cb82f6732
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
Oct 12 16:41:40  lotus-worker[73229]:    2:          0x20bbbf0 - std::sys_common::backtrace::_print_fmt::hfbda19e17f6db318
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:67:5
Oct 12 16:41:40  lotus-worker[73229]:    3:          0x20bbbf0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h1a8751bf59281272
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:46:22
Oct 12 16:41:40  lotus-worker[73229]:    4:          0x212be5f - core::fmt::write::h7aa6cd0067dca82a
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/core/src/fmt/mod.rs:1094:17
Oct 12 16:41:40  lotus-worker[73229]:    5:          0x20b0385 - std::io::Write::write_fmt::hd7dd3a1df9b6befb
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/io/mod.rs:1580:15
Oct 12 16:41:40  lotus-worker[73229]:    6:          0x20bf94b - std::sys_common::backtrace::_print::h551e9ec8a9fa8106
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:49:5
Oct 12 16:41:40  lotus-worker[73229]:    7:          0x20bf94b - std::sys_common::backtrace::print::ha4b1c5e95fa040b3
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/sys_common/backtrace.rs:36:9
Oct 12 16:41:40  lotus-worker[73229]:    8:          0x20bf94b - std::panicking::default_hook::{{closure}}::h0b34c9ab7fb9f857
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/panicking.rs:208:50
Oct 12 16:41:40  lotus-worker[73229]:    9:          0x20bf42d - std::panicking::default_hook::h3067e8318decd17a
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/panicking.rs:225:9
Oct 12 16:41:40  lotus-worker[73229]:   10:          0x20c003d - std::panicking::rust_panic_with_hook::h81b8facc50f34daa
Oct 12 16:41:40  lotus-worker[73229]:                                at /rustc/bb491ed23937aef876622e4beb68ae95938b3bf9/library/std/src/panicking.rs:591:17
Oct 12 16:41:40  lotus-worker[73229]:   11:          0x20231b6 - std::panicking::begin_panic::{{closure}}::h17aa203bdf3b53f4
Oct 12 16:41:40  lotus-worker[73229]:   12:          0x202317e - std::sys_common::backtrace::__rust_end_short_backtrace::h992fb3327e62bc2a
Oct 12 16:41:40  lotus-worker[73229]:   13:           0x5029ac - std::panicking::begin_panic::hf1f74302d08d14e1
Oct 12 16:41:40  lotus-worker[73229]:   14:          0x2025373 - yastl::wait::WaitGroup::join::h03280543330f3f44
Oct 12 16:41:40  lotus-worker[73229]:   15:          0x1b6d8fc - yastl::scope::Scope::zoom::h2dc3f87d72d5d2cf
Oct 12 16:41:40  lotus-worker[73229]:   16:          0x1a98d66 - yastl::Pool::scoped::hc33c0a9214f792c8
Oct 12 16:41:40  lotus-worker[73229]:   17:          0x19b4471 - storage_proofs_core::measurements::measure_op::h1260afd960ab575d
Oct 12 16:41:40  lotus-worker[73229]:   18:          0x2212823 - storage_proofs_porep::stacked::vanilla::proof::StackedDrg<Tree,G>::replicate_phase2::h125e70534093daef
Oct 12 16:41:40  lotus-worker[73229]:   19:          0x217218b - filecoin_proofs::api::seal::seal_pre_commit_phase2::ha3d06c2bb552dfde
Oct 12 16:41:40  lotus-worker[73229]:   20:          0x21eb349 - filecoin_proofs_api::seal::seal_pre_commit_phase2_inner::he260fc02a6d000f5
Oct 12 16:41:40  lotus-worker[73229]:   21:          0x21ae3e8 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h979bad57e9431d37
Oct 12 16:41:40  lotus-worker[73229]:   22:          0x2160d3e - ffi_toolkit::catch_panic_response::h8d0bb7f164831c2b
Oct 12 16:41:40  lotus-worker[73229]:   23:          0x1b27c8a - fil_seal_pre_commit_phase2
Oct 12 16:41:40  lotus-worker[73229]:   24:          0x18f4597 - _cgo_c8a539579dfc_Cfunc_fil_seal_pre_commit_phase2
Oct 12 16:41:40  lotus-worker[73229]:                                at /tmp/go-build/cgo-gcc-prolog:1232:11
Oct 12 16:41:40  lotus-worker[73229]:   25:           0x58bb30 - runtime.asmcgocall
Oct 12 16:41:40  lotus-worker[73229]:                                at /usr/local/go/src/runtime/asm_amd64.s:667
Oct 12 16:41:40  lotus-worker[73229]: thread panicked while panicking. aborting.
Oct 12 16:41:40  lotus-worker[73229]: SIGILL: illegal instruction
Oct 12 16:41:40  lotus-worker[73229]: PC=0x20bffbc m=27 sigcode=2
Oct 12 16:41:40  lotus-worker[73229]: instruction bytes: 0xf 0xb 0xf 0xb 0x4c 0x89 0x6c 0x24 0x10 0x4c 0x89 0x64 0x24 0x18 0x48 0x8d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 0 [idle]:
Oct 12 16:41:40  lotus-worker[73229]: runtime: unknown pc 0x20bffbc
Oct 12 16:41:40  lotus-worker[73229]: stack: frame={sp:0x7f189d7f7c50, fp:0x0} stack=[0x7f189cffa108,0x7f189d7f9d08)
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b50:  0000000004d08b38  00007f189d7f7cb8
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b60:  00007f1917ad5160  00000000020b0385
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b70:  00007f189d7f7c20  00007f189d7f7bf8
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b80:  0000000000000004  00007f187c017980
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b90:  0000000004d0e7c0  0000000000000001
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7ba0:  0000000000000000  0000000004d08d48
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bb0:  0000000003fa8360  0000000000000000
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bc0:  0000000000000000  0000000000000002
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bd0:  0000000004d08b38  00007f189d7f7cb8
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7be0:  00007f1917ad5160  00000000020be0e0
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bf0:  0000000004d08d48  00007f189d7f7bd0
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c00:  0000000004d0e7c0  0000000000000001
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c10:  0000000000000000  0000000004d08d48
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c20:  0000000003fa8360  0000000000000000
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c30:  0000000004d08b38  0000000000000002
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c40:  0000000004d08b38  00000000020bffbc
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c50: <0000000004d0e7c0  0000000000000001
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c60:  0000000000000000  0000000004d08d48
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c70:  0000000003fa8360  0000000000000000
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c80:  0000000003f986d6  00007f187c0177d0
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c90:  0000000000000001  0000000000000018
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7ca0:  00007f187c003270  00000000020231b6
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cb0:  0000000000000000  0000000003f986d6
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cc0:  0000000000000018  000000000202317e
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cd0:  0000000003f986d6  0000000000000018
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7ce0:  0000000004d08d48  00000000005029ac
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cf0:  0000000003f986d6  0000000000000018
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d00:  0000000004d08d48  00007f187c003278
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d10:  00007f187c003290  0000000002025373
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d20:  0000000004ccf7c8  00007f187c003278
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d30:  0000000000000201  00007f187c003278
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d40:  00007f189d7f7d01  00007f187c0044e0
Oct 12 16:41:40  lotus-worker[73229]: runtime: unknown pc 0x20bffbc
Oct 12 16:41:40  lotus-worker[73229]: stack: frame={sp:0x7f189d7f7c50, fp:0x0} stack=[0x7f189cffa108,0x7f189d7f9d08)
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b50:  0000000004d08b38  00007f189d7f7cb8
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b60:  00007f1917ad5160  00000000020b0385
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b70:  00007f189d7f7c20  00007f189d7f7bf8
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b80:  0000000000000004  00007f187c017980
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7b90:  0000000004d0e7c0  0000000000000001
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7ba0:  0000000000000000  0000000004d08d48
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bb0:  0000000003fa8360  0000000000000000
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bc0:  0000000000000000  0000000000000002
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bd0:  0000000004d08b38  00007f189d7f7cb8
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7be0:  00007f1917ad5160  00000000020be0e0
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7bf0:  0000000004d08d48  00007f189d7f7bd0
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c00:  0000000004d0e7c0  0000000000000001
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c10:  0000000000000000  0000000004d08d48
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c20:  0000000003fa8360  0000000000000000
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c30:  0000000004d08b38  0000000000000002
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c40:  0000000004d08b38  00000000020bffbc
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c50: <0000000004d0e7c0  0000000000000001
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c60:  0000000000000000  0000000004d08d48
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c70:  0000000003fa8360  0000000000000000
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c80:  0000000003f986d6  00007f187c0177d0
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7c90:  0000000000000001  0000000000000018
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7ca0:  00007f187c003270  00000000020231b6
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cb0:  0000000000000000  0000000003f986d6
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cc0:  0000000000000018  000000000202317e
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cd0:  0000000003f986d6  0000000000000018
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7ce0:  0000000004d08d48  00000000005029ac
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7cf0:  0000000003f986d6  0000000000000018
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d00:  0000000004d08d48  00007f187c003278
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d10:  00007f187c003290  0000000002025373
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d20:  0000000004ccf7c8  00007f187c003278
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d30:  0000000000000201  00007f187c003278
Oct 12 16:41:40  lotus-worker[73229]: 00007f189d7f7d40:  00007f189d7f7d01  00007f187c0044e0
Oct 12 16:41:40  lotus-worker[73229]: goroutine 298 [syscall]:
Oct 12 16:41:40  lotus-worker[73229]: runtime.cgocall(0x18f4570, 0xc002dd1ad0, 0x28064e0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/cgocall.go:154 +0x5b fp=0xc002dd1aa0 sp=0xc002dd1a68 pc=0x51c7fb
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/filecoin-ffi/generated._Cfunc_fil_seal_pre_commit_phase2(0x7f187c000ca0, 0x5e9, 0x7f187c0012a0, 0x7f187c0012d0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         _cgo_gotypes.go:2021 +0x49 fp=0xc002dd1ad0 sp=0xc002dd1aa0 pc=0x12d6e49
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/filecoin-ffi/generated.FilSealPreCommitPhase2(0xc0023e6600, 0x5e9, 0x5eb, 0x5e9, 0xc001c50a50, 0x22, 0xc001c50a80, 0x23, 0xc001c50960)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/filecoin-ffi/generated/generated.go:766 +0x177 fp=0xc002dd1bb0 sp=0xc002dd1ad0 pc=0x12e0f97
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/filecoin-ffi.SealPreCommitPhase2(0xc0023e6600, 0x5e9, 0x5eb, 0xc001c509f0, 0x21, 0xc001c508d0, 0x22, 0x0, 0x0, 0x0, ...)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/filecoin-ffi/proofs.go:370 +0xbe fp=0xc002dd1c50 sp=0xc002dd1bb0 pc=0x12e4efe
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).SealPreCommit2(0xc0005a1560, 0x2e65988, 0xc0011097d0, 0x5693, 0x3a1, 0x8, 0xc0023e6600, 0x5e9, 0x5eb, 0x0, ...)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/sector-storage/ffiwrapper/sealer_cgo.go:543 +0x2cd fp=0xc002dd1de8 sp=0xc002dd1c50 pc=0x12f0a0d
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/lotus/extern/sector-storage.(*LocalWorker).SealPreCommit2.func1(0x2e65988, 0xc0011097d0, 0x5693, 0x3a1, 0xc646e6988e91639b, 0x281ecc800f84d78c, 0xc0015680c0, 0xc001898f88, 0xc00043378c, 0xc001899038)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/sector-storage/worker_local.go:359 +0x91 fp=0xc002dd1ea0 sp=0xc002dd1de8 pc=0x130f591
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/lotus/extern/sector-storage.(*LocalWorker).asyncCall.func1(0xc000215050, 0x2e65528, 0xc00063b680, 0xc000e12690, 0x5693, 0x3a1, 0xc646e6988e91639b, 0x281ecc800f84d78c, 0x2a78d99, 0xe)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/sector-storage/worker_local.go:242 +0x102 fp=0xc002dd1f90 sp=0xc002dd1ea0 pc=0x130e9e2
Oct 12 16:41:40  lotus-worker[73229]: runtime.goexit()
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc002dd1f98 sp=0xc002dd1f90 pc=0x58bec1
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/filecoin-project/lotus/extern/sector-storage.(*LocalWorker).asyncCall
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/sector-storage/worker_local.go:234 +0x31d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 1 [IO wait]:
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.runtime_pollWait(0x7f18eca91c70, 0x72, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/netpoll.go:222 +0x55
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).wait(0xc001137498, 0x72, 0x0, 0x0, 0x2a2fda3)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).waitRead(...)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*FD).Accept(0xc001137480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_unix.go:401 +0x212
Oct 12 16:41:40  lotus-worker[73229]: net.(*netFD).accept(0xc001137480, 0x232a03edc5ed0af8, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/fd_unix.go:172 +0x45
Oct 12 16:41:40  lotus-worker[73229]: net.(*TCPListener).accept(0xc000d6ee58, 0x61659ea4, 0xc0022075b8, 0x5b21a6)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
Oct 12 16:41:40  lotus-worker[73229]: net.(*TCPListener).Accept(0xc000d6ee58, 0xc002207608, 0x18, 0xc000000180, 0x8b7bfb)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/tcpsock.go:261 +0x65
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*Server).Serve(0xc0003502a0, 0x2e5d570, 0xc000d6ee58, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:2981 +0x285
Oct 12 16:41:40  lotus-worker[73229]: main.glob..func5(0xc000793840, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cmd/lotus-seal-worker/main.go:550 +0x24ba
Oct 12 16:41:40  lotus-worker[73229]: github.com/urfave/cli/v2.(*Command).Run(0x4d55fc0, 0xc0007934c0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/command.go:164 +0x4dd
Oct 12 16:41:40  lotus-worker[73229]: github.com/urfave/cli/v2.(*App).RunContext(0xc000245200, 0x2e654b8, 0xc000040078, 0xc00003c090, 0x9, 0x9, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/app.go:306 +0x810
Oct 12 16:41:40  lotus-worker[73229]: github.com/urfave/cli/v2.(*App).Run(...)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/app.go:215
Oct 12 16:41:40  lotus-worker[73229]: main.main()
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cmd/lotus-seal-worker/main.go:112 +0x6ec
Oct 12 16:41:40  lotus-worker[73229]: goroutine 6 [select, 2 minutes]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc0001d9bf0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/ipfs/go-log@v1.0.5/writer/writer.go:71 +0x106
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/ipfs/go-log/writer.NewMirrorWriter
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/ipfs/go-log@v1.0.5/writer/writer.go:36 +0xb9
Oct 12 16:41:40  lotus-worker[73229]: goroutine 7 [select]:
Oct 12 16:41:40  lotus-worker[73229]: go.opencensus.io/stats/view.(*worker).start(0xc0001f8300)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xcd
Oct 12 16:41:40  lotus-worker[73229]: created by go.opencensus.io/stats/view.init.0
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x68
Oct 12 16:41:40  lotus-worker[73229]: goroutine 274 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0xc0001f6620, 0x0, 0xc0003248c0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:390 +0x266
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:371 +0x8d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 297 [IO wait]:
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.runtime_pollWait(0x7f18eca919b8, 0x72, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/netpoll.go:222 +0x55
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).wait(0xc000f26218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).waitRead(...)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*FD).Read(0xc000f26200, 0xc0023d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
Oct 12 16:41:40  lotus-worker[73229]: net.(*netFD).Read(0xc000f26200, 0xc0023d8000, 0x1000, 0x1000, 0x0, 0x7f18eca919c0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/fd_posix.go:55 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net.(*conn).Read(0xc000011ff8, 0xc0023d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/net.go:183 +0x91
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*connReader).Read(0xc00063a570, 0xc0023d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:800 +0x1b9
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).fill(0xc00121fb00)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:101 +0x108
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).Peek(0xc00121fb00, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:139 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*conn).readRequest(0xc00063d540, 0x2e65480, 0xc000f14280, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:983 +0xec5
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*conn).serve(0xc00063d540, 0x2e65528, 0xc000f14280)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:1878 +0x705
Oct 12 16:41:40  lotus-worker[73229]: created by net/http.(*Server).Serve
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:3013 +0x39b
Oct 12 16:41:40  lotus-worker[73229]: goroutine 159 [chan receive, 2 minutes]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/lotus/cli/util.ReqContext.func1(0xc000d38ea0, 0xc000319520)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cli/util/api.go:396 +0x34
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/filecoin-project/lotus/cli/util.ReqContext
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cli/util/api.go:395 +0x108
Oct 12 16:41:40  lotus-worker[73229]: goroutine 161 [syscall, 2 minutes]:
Oct 12 16:41:40  lotus-worker[73229]: os/signal.signal_recv(0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/sigqueue.go:168 +0xa5
Oct 12 16:41:40  lotus-worker[73229]: os/signal.loop()
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/os/signal/signal_unix.go:23 +0x25
Oct 12 16:41:40  lotus-worker[73229]: created by os/signal.Notify.func1.1
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/os/signal/signal.go:151 +0x45
Oct 12 16:41:40  lotus-worker[73229]: goroutine 169 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc00027a1a0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0xb1
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/syndtr/goleveldb/leveldb.openDB
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x429
Oct 12 16:41:40  lotus-worker[73229]: goroutine 258 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*DB).updateSize(0xc001716400, 0xc000648060)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:1200 +0x125
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:360 +0x9a5
Oct 12 16:41:40  lotus-worker[73229]: goroutine 275 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0xc0001f6620, 0x1, 0xc0003248c0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:390 +0x266
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:371 +0x8d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 170 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc00027a1a0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0xd7
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/syndtr/goleveldb/leveldb.openDB
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x44b
Oct 12 16:41:40  lotus-worker[73229]: goroutine 171 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc00027a1a0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x228
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/syndtr/goleveldb/leveldb.openDB
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x5a5
Oct 12 16:41:40  lotus-worker[73229]: goroutine 168 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc0003500e0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0x105
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x18c
Oct 12 16:41:40  lotus-worker[73229]: goroutine 172 [select, 2 minutes]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc00027a1a0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0x125
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/syndtr/goleveldb/leveldb.openDB
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x5c7
Oct 12 16:41:40  lotus-worker[73229]: goroutine 173 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc0011b9ce0, 0xc0001b4860)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:212 +0x2a7
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:70 +0x76
Oct 12 16:41:40  lotus-worker[73229]: goroutine 174 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc0011b9d10, 0xc0001b4860)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:212 +0x2a7
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:70 +0x76
Oct 12 16:41:40  lotus-worker[73229]: goroutine 175 [select, 2 minutes]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*DB).updateSize(0xc001556800, 0xc0001b4920)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:1200 +0x125
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:360 +0x9a5
Oct 12 16:41:40  lotus-worker[73229]: goroutine 176 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0xc0001f64d0, 0x0, 0xc0001b4c60)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:390 +0x266
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:371 +0x8d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 177 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0xc0001f64d0, 0x1, 0xc0001b4c60)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:390 +0x266
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:371 +0x8d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 226 [chan receive]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*DB).flushMemtable(0xc001556800, 0xc0001b4c80, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:1107 +0xb7
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.Open.func5(0xc000010db0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:377 +0x37
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:376 +0x1445
Oct 12 16:41:40  lotus-worker[73229]: goroutine 228 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*valueLog).flushDiscardStats(0xc0015569d0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1948 +0x15c
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.(*valueLog).open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1114 +0x6c
Oct 12 16:41:40  lotus-worker[73229]: goroutine 242 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*DB).doWrites(0xc001556800, 0xc0003b00e0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:862 +0x533
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:411 +0xf34
Oct 12 16:41:40  lotus-worker[73229]: goroutine 243 [chan receive]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*valueLog).waitOnGC(0xc0015569d0, 0xc0003b01e0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1851 +0x65
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:415 +0x1165
Oct 12 16:41:40  lotus-worker[73229]: goroutine 244 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*publisher).listenForUpdates(0xc0011b9da0, 0xc0003b05e0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/publisher.go:67 +0x145
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:419 +0x101b
Oct 12 16:41:40  lotus-worker[73229]: goroutine 245 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/ipfs/go-ds-badger2.(*Datastore).periodicGC(0xc000f08060)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:172 +0xe5
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/ipfs/go-ds-badger2.NewDatastore
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:160 +0x65d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 246 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc000f12090, 0xc0003b0780)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:212 +0x2a7
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:70 +0x76
Oct 12 16:41:40  lotus-worker[73229]: goroutine 247 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0xc000f120c0, 0xc0003b0780)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:212 +0x2a7
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:70 +0x76
Oct 12 16:41:40  lotus-worker[73229]: goroutine 276 [chan receive]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*DB).flushMemtable(0xc001716400, 0xc000324900, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:1107 +0xb7
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.Open.func5(0xc000b0c008)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:377 +0x37
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:376 +0x1445
Oct 12 16:41:40  lotus-worker[73229]: goroutine 279 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*DB).doWrites(0xc001716400, 0xc000324b00)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:862 +0x533
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:411 +0xf34
Oct 12 16:41:40  lotus-worker[73229]: goroutine 278 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*valueLog).flushDiscardStats(0xc0017165d0)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1948 +0x15c
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.(*valueLog).open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1114 +0x6c
Oct 12 16:41:40  lotus-worker[73229]: goroutine 280 [chan receive]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*valueLog).waitOnGC(0xc0017165d0, 0xc000324b20)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1851 +0x65
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:415 +0x1165
Oct 12 16:41:40  lotus-worker[73229]: goroutine 281 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/dgraph-io/badger/v2.(*publisher).listenForUpdates(0xc000f12150, 0xc000324b40)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/publisher.go:67 +0x145
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/dgraph-io/badger/v2.Open
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:419 +0x101b
Oct 12 16:41:40  lotus-worker[73229]: goroutine 282 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/ipfs/go-ds-badger2.(*Datastore).periodicGC(0xc000d68c60)
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:172 +0xe5
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/ipfs/go-ds-badger2.NewDatastore
Oct 12 16:41:40  lotus-worker[73229]:         go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:160 +0x65d
Oct 12 16:41:40  lotus-worker[73229]: goroutine 286 [select]:
Oct 12 16:41:40  lotus-worker[73229]: github.com/filecoin-project/lotus/extern/sector-storage/stores.(*Local).reportHealth(0xc001568120, 0x2e65480, 0xc000793a40)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/sector-storage/stores/local.go:330 +0x11b
Oct 12 16:41:40  lotus-worker[73229]: created by github.com/filecoin-project/lotus/extern/sector-storage/stores.(*Local).open
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/extern/sector-storage/stores/local.go:242 +0x1ae
Oct 12 16:41:40  lotus-worker[73229]: goroutine 284 [IO wait]:
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.runtime_pollWait(0x7f18eca91d58, 0x72, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/netpoll.go:222 +0x55
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).wait(0xc001202898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).waitRead(...)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*FD).Read(0xc001202880, 0xc001554000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
Oct 12 16:41:40  lotus-worker[73229]: net.(*netFD).Read(0xc001202880, 0xc001554000, 0x1000, 0x1000, 0x55433c, 0xc000a50c38, 0x5830a0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/fd_posix.go:55 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net.(*conn).Read(0xc000010b30, 0xc001554000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/net.go:183 +0x91
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*persistConn).Read(0xc000c90900, 0xc001554000, 0x1000, 0x1000, 0xc0015642a0, 0xc000a50d40, 0x51e735)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:1922 +0x77
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).fill(0xc000d68d80)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:101 +0x108
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).Peek(0xc000d68d80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:139 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*persistConn).readLoop(0xc000c90900)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:2083 +0x1a8
Oct 12 16:41:40  lotus-worker[73229]: created by net/http.(*Transport).dialConn
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:1743 +0xc77
Oct 12 16:41:40  lotus-worker[73229]: goroutine 285 [select]:
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*persistConn).writeLoop(0xc000c90900)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:2382 +0xf7
Oct 12 16:41:40  lotus-worker[73229]: created by net/http.(*Transport).dialConn
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:1744 +0xc9c
Oct 12 16:41:40  lotus-worker[73229]: goroutine 288 [chan receive]:
Oct 12 16:41:40  lotus-worker[73229]: main.glob..func5.5(0x2e65480, 0xc000793a40, 0xc0003502a0)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cmd/lotus-seal-worker/main.go:434 +0x65
Oct 12 16:41:40  lotus-worker[73229]: created by main.glob..func5
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cmd/lotus-seal-worker/main.go:433 +0x1ca5
Oct 12 16:41:40  lotus-worker[73229]: goroutine 306 [IO wait]:
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.runtime_pollWait(0x7f18eca91b88, 0x72, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/netpoll.go:222 +0x55
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).wait(0xc001137618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).waitRead(...)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*FD).Read(0xc001137600, 0xc0023b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
Oct 12 16:41:40  lotus-worker[73229]: net.(*netFD).Read(0xc001137600, 0xc0023b2000, 0x1000, 0x1000, 0x55433c, 0xc0001a7c38, 0x5830a0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/fd_posix.go:55 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net.(*conn).Read(0xc0003fa108, 0xc0023b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/net.go:183 +0x91
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*persistConn).Read(0xc0004625a0, 0xc0023b2000, 0x1000, 0x1000, 0xc000253560, 0xc0001a7d40, 0x51e735)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:1922 +0x77
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).fill(0xc001205b60)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:101 +0x108
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).Peek(0xc001205b60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:139 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*persistConn).readLoop(0xc0004625a0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:2083 +0x1a8
Oct 12 16:41:40  lotus-worker[73229]: created by net/http.(*Transport).dialConn
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:1743 +0xc77
Oct 12 16:41:40  lotus-worker[73229]: goroutine 307 [select]:
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*persistConn).writeLoop(0xc0004625a0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:2382 +0xf7
Oct 12 16:41:40  lotus-worker[73229]: created by net/http.(*Transport).dialConn
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/transport.go:1744 +0xc9c
Oct 12 16:41:40  lotus-worker[73229]: goroutine 291 [select]:
Oct 12 16:41:40  lotus-worker[73229]: main.glob..func5.7(0xc001568120, 0x2e65480, 0xc000793a40, 0xc000d938a0, 0x2e94f28, 0xc000615000, 0xc000d643f0, 0xc000044198, 0x11, 0xc000319680)
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cmd/lotus-seal-worker/main.go:527 +0x411
Oct 12 16:41:40  lotus-worker[73229]: created by main.glob..func5
Oct 12 16:41:40  lotus-worker[73229]:         github/lotus/cmd/lotus-seal-worker/main.go:487 +0x248f
Oct 12 16:41:40  lotus-worker[73229]: goroutine 308 [IO wait]:
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.runtime_pollWait(0x7f18eca91aa0, 0x72, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/runtime/netpoll.go:222 +0x55
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).wait(0xc0001f8998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*pollDesc).waitRead(...)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Oct 12 16:41:40  lotus-worker[73229]: internal/poll.(*FD).Read(0xc0001f8980, 0xc0023b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
Oct 12 16:41:40  lotus-worker[73229]: net.(*netFD).Read(0xc0001f8980, 0xc0023b8000, 0x1000, 0x1000, 0x0, 0x7f18eca91aa8, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/fd_posix.go:55 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net.(*conn).Read(0xc0003fa110, 0xc0023b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/net.go:183 +0x91
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*connReader).Read(0xc001094d80, 0xc0023b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:800 +0x1b9
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).fill(0xc001205c20)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:101 +0x108
Oct 12 16:41:40  lotus-worker[73229]: bufio.(*Reader).Peek(0xc001205c20, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/bufio/bufio.go:139 +0x4f
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*conn).readRequest(0xc0003da500, 0x2e65480, 0xc000c42840, 0x0, 0x0, 0x0)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:983 +0xec5
Oct 12 16:41:40  lotus-worker[73229]: net/http.(*conn).serve(0xc0003da500, 0x2e65528, 0xc000c42840)
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:1878 +0x705
Oct 12 16:41:40  lotus-worker[73229]: created by net/http.(*Server).Serve
Oct 12 16:41:40  lotus-worker[73229]:         /usr/local/go/src/net/http/server.go:3013 +0x39b
Oct 12 16:41:40  lotus-worker[73229]: rax    0x1004
Oct 12 16:41:40  lotus-worker[73229]: rbx    0x2
Oct 12 16:41:40  lotus-worker[73229]: rcx    0x4
Oct 12 16:41:40  lotus-worker[73229]: rdx    0x4d0dad0
Oct 12 16:41:40  lotus-worker[73229]: rdi    0x0
Oct 12 16:41:40  lotus-worker[73229]: rsi    0x3fa9eac
Oct 12 16:41:40  lotus-worker[73229]: rbp    0x7f1917ad5160
Oct 12 16:41:40  lotus-worker[73229]: rsp    0x7f189d7f7c50
Oct 12 16:41:40  lotus-worker[73229]: r8     0x0
Oct 12 16:41:40  lotus-worker[73229]: r9     0x0
Oct 12 16:41:40  lotus-worker[73229]: r10    0x7f189d7f6c60
Oct 12 16:41:40  lotus-worker[73229]: r11    0x0
Oct 12 16:41:40  lotus-worker[73229]: r12    0x4d08d48
Oct 12 16:41:40  lotus-worker[73229]: r13    0x0
Oct 12 16:41:40  lotus-worker[73229]: r14    0x4d08b38
Oct 12 16:41:40  lotus-worker[73229]: r15    0x7f189d7f7cb8
Oct 12 16:41:40  lotus-worker[73229]: rip    0x20bffbc
Oct 12 16:41:40  lotus-worker[73229]: rflags 0x10202
Oct 12 16:41:40  lotus-worker[73229]: cs     0x33
Oct 12 16:41:40  lotus-worker[73229]: fs     0x0
Oct 12 16:41:40  lotus-worker[73229]: gs     0x0

Repo Steps

Build lotus with FIL_PROOFS_CUDA_NVCC_ARGS="--fatbin --gpu-architecture=sm_70 --generate-code=arch=compute_70,code=sm_70"
export FIL_PROOFS_GPU_FRAMEWORK=cuda
Do PC2

filecoin-project / lotus