search

filecoin-project / lotus

Reference implementation of the Filecoin protocol, written in Go
https://lotus.filecoin.io/
Other
2.85k stars 1.27k forks source link

[BUG] No longer possible to start with LOTUS_ENABLE_CHAINSTORE_FALLBACK=1 #5975

Closed ribasushi closed 3 years ago

ribasushi commented 3 years ago

Describe the bug Running LOTUS_ENABLE_CHAINSTORE_FALLBACK=1 lotus daemon crashes on startup

Version (run lotus version): Stock v1.6.0

Logs

Complete log from startup to crash ``` 2021-04-06T13:36:45.537Z INFO main lotus/daemon.go:214 lotus repo: /srv/lotus/.lotus 2021-04-06T13:36:45.537Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-2-b62098629d07946e9028127e70295ed996fe3ed25b0f9f88eb610a0ab4385a3c.vk is ok 2021-04-06T13:36:45.537Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-5294475db5237a2e83c3e52fd6c2b03859a1831d45ed08c4f35dbf9a803165a9.vk is ok 2021-04-06T13:36:45.537Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0170db1f394b35d995252228ee359194b13199d259380541dc529fb0099096b0.vk is ok 2021-04-06T13:36:45.538Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-ecd683648512ab1765faa2a5f14bab48f676e633467f0aa8aad4b55dcb0652bb.vk is ok 2021-04-06T13:36:45.538Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-3ea05428c9d11689f23529cde32fd30aabd50f7d2c93657c1d3650bca3e8ea9e.vk is ok 2021-04-06T13:36:45.538Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.vk is ok 2021-04-06T13:36:45.538Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-6babf46ce344ae495d558e7770a585b2382d54f225af8ed0397b8be7c3fcd472.vk is ok 2021-04-06T13:36:45.538Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.vk is ok 2021-04-06T13:36:45.537Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-559e581f022bb4e4ec6e719e563bf0e026ad6de42e56c18714a2c692b1b88d7e.vk is ok 2021-04-06T13:36:45.538Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-7d739b8cf60f1b0709eeebee7730e297683552e4b69cab6984ec0285663c5781.vk is ok 2021-04-06T13:36:45.538Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0cfb4f178bbb71cf2ecfcd42accce558b27199ab4fb59cb78f2483fe21ef36d9.vk is ok 2021-04-06T13:36:45.538Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-2-sha256_hasher-96f1b4a04c5c51e4759bbf224bbc2ef5a42c7100f16ec0637123f16a845ddfb2.vk is ok 2021-04-06T13:36:45.537Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-50c7368dea9593ed0989e70974d28024efa9d156d585b7eea1be22b2e753f331.vk is ok 2021-04-06T13:36:45.545Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.vk is ok 2021-04-06T13:36:45.545Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138 Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-2-2627e4006b67f99cef990c0a47d5426cb7ab0a0ad58fc1061547bf2d28b09def.vk is ok 2021-04-06T13:36:45.545Z INFO build go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:162 parameter and key-fetching complete 2021-04-06T13:36:45.550Z INFO modules modules/core.go:64 memory limits initialized {"max_mem_heap": 0, "total_system_mem": 267082211328, "effective_mem_limit": 267082211328} 2021-04-06T13:36:45.551Z WARN modules modules/core.go:124 failed to initialize cgroup-driven watchdog; err: failed to load memory cgroup stats: open /sys/fs/cgroup/memory/user.slice/user-1001.slice/session-5.scope/memory.memsw.usage_in_bytes: no such file or directory 2021-04-06T13:36:45.551Z WARN modules modules/core.go:125 trying a system-driven watchdog 2021-04-06T13:36:45.551Z INFO watchdog go-watchdog@v1.0.1/watermarks.go:16 initialized watermark watchdog policy; watermarks: []; thresholds: [133541105664 160249326796 186957547929 227019879628 240373990195 247051045478 253728100761] 2021-04-06T13:36:45.551Z INFO watchdog go-watchdog@v1.0.1/watchdog.go:457 initialized heap profile capture; threshold: 0.900000; max captures: 10; dir: /srv/lotus/.lotus/heapprof 2021-04-06T13:36:45.551Z INFO modules modules/core.go:130 initialized system-driven watchdog 2021-04-06T13:36:45.559Z INFO badger v2@v2.2007.2/levels.go:183 All 0 tables opened in 0s 2021-04-06T13:36:45.559Z INFO badger v2@v2.2007.2/value.go:1158 Replaying file id: 0 at offset: 0 2021-04-06T13:36:45.559Z INFO badger v2@v2.2007.2/value.go:1178 Replay took: 2.314µs 2021-04-06T13:36:45.561Z INFO badger v2@v2.2007.2/levels.go:183 All 1 tables opened in 0s 2021-04-06T13:36:45.561Z INFO badger v2@v2.2007.2/value.go:1158 Replaying file id: 0 at offset: 49876 2021-04-06T13:36:45.561Z INFO badger v2@v2.2007.2/value.go:1178 Replay took: 18.757µs 2021-04-06T13:36:45.561Z INFO backupds backupds/log.go:123 opening log {"file": "/srv/lotus/.lotus/kvlog/metadata/1617462230.log.cbor"} 2021-04-06T13:36:45.976Z INFO backupds backupds/log.go:174 log opened {"file": "/srv/lotus/.lotus/kvlog/metadata/1617462230.log.cbor", "openCount": 2, "logValues": 85043} 2021-04-06T13:36:45.977Z INFO backupds backupds/log.go:199 new log head {"loghead": "1617462230.log.cbor;f1ccf2fe-ce23-4a0f-955b-c4f6882a2db0;1617716205"} runtime: goroutine stack exceeds 1000000000-byte limit runtime: sp=0x402e9e0420 stack=[0x402e9e0000, 0x404e9e0000] fatal error: stack overflow runtime stack: runtime.throw(0x2a2243d, 0xe) /usr/local/go/src/runtime/panic.go:1116 +0x54 runtime.newstack() /usr/local/go/src/runtime/stack.go:1067 +0x600 runtime.morestack() /usr/local/go/src/runtime/asm_arm64.s:308 +0x70 goroutine 1 [running]: go.uber.org/dig.shallowCheckDependencies.func1(0x2e75020, 0x4001207440, 0xbdcb3fd31f4609b7) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:743 +0x43c fp=0x402e9e0420 sp=0x402e9e0420 pc=0x11a8b3c go.uber.org/dig.paramVisitorFunc.Visit(0x400e429830, 0x2e75020, 0x4001207440, 0x0, 0x0) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:102 +0x38 fp=0x402e9e0450 sp=0x402e9e0420 pc=0x11a1f78 go.uber.org/dig.walkParam(0x2e75020, 0x4001207440, 0x2e434e0, 0x400e429830) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:117 +0x48 fp=0x402e9e0570 sp=0x402e9e0450 pc=0x11a2008 go.uber.org/dig.walkParam(0x2e74fa0, 0x400e429800, 0x2e434e0, 0x400e429830) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:131 +0x110 fp=0x402e9e0690 sp=0x402e9e0570 pc=0x11a20d0 go.uber.org/dig.shallowCheckDependencies(0x2e97ec0, 0x40005da3c0, 0x2e74fa0, 0x400e429800, 0x0, 0x40005da301) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:743 +0xc0 fp=0x402e9e06d0 sp=0x402e9e0690 pc=0x119dfc0 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:694 +0x60 fp=0x402e9e0850 sp=0x402e9e06d0 pc=0x119d7a0 go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e0980 sp=0x402e9e0850 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13780, 0x1, 0x400e4297d0, 0x402e9e0a98, 0x119d7a0) :1 +0x84 fp=0x402e9e0a20 sp=0x402e9e0980 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e0b18, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e0aa0 sp=0x402e9e0a20 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e0c20 sp=0x402e9e0aa0 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e0d50 sp=0x402e9e0c20 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13700, 0x1, 0x400e429770, 0x402e9e0e68, 0x119d7a0) :1 +0x84 fp=0x402e9e0df0 sp=0x402e9e0d50 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e0ee8, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e0e70 sp=0x402e9e0df0 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e0ff0 sp=0x402e9e0e70 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e1120 sp=0x402e9e0ff0 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13660, 0x1, 0x400e429710, 0x402e9e1238, 0x119d7a0) :1 +0x84 fp=0x402e9e11c0 sp=0x402e9e1120 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e12b8, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e1240 sp=0x402e9e11c0 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e13c0 sp=0x402e9e1240 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e14f0 sp=0x402e9e13c0 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a135e0, 0x1, 0x400e4296b0, 0x402e9e1608, 0x119d7a0) :1 +0x84 fp=0x402e9e1590 sp=0x402e9e14f0 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e1688, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e1610 sp=0x402e9e1590 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e1790 sp=0x402e9e1610 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e18c0 sp=0x402e9e1790 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13540, 0x1, 0x400e429650, 0x402e9e19d8, 0x119d7a0) :1 +0x84 fp=0x402e9e1960 sp=0x402e9e18c0 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e1a58, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e19e0 sp=0x402e9e1960 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e1b60 sp=0x402e9e19e0 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e1c90 sp=0x402e9e1b60 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a134c0, 0x1, 0x400e4295f0, 0x402e9e1da8, 0x119d7a0) :1 +0x84 fp=0x402e9e1d30 sp=0x402e9e1c90 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e1e28, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e1db0 sp=0x402e9e1d30 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e1f30 sp=0x402e9e1db0 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e2060 sp=0x402e9e1f30 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13420, 0x1, 0x400e429590, 0x402e9e2178, 0x119d7a0) :1 +0x84 fp=0x402e9e2100 sp=0x402e9e2060 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e21f8, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e2180 sp=0x402e9e2100 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e2300 sp=0x402e9e2180 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e2430 sp=0x402e9e2300 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a133a0, 0x1, 0x400e429530, 0x402e9e2548, 0x119d7a0) :1 +0x84 fp=0x402e9e24d0 sp=0x402e9e2430 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e25c8, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e2550 sp=0x402e9e24d0 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e26d0 sp=0x402e9e2550 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e2800 sp=0x402e9e26d0 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13300, 0x1, 0x400e4294d0, 0x402e9e2918, 0x119d7a0) :1 +0x84 fp=0x402e9e28a0 sp=0x402e9e2800 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e2998, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e2920 sp=0x402e9e28a0 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e2aa0 sp=0x402e9e2920 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e2bd0 sp=0x402e9e2aa0 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13280, 0x1, 0x400e429470, 0x402e9e2ce8, 0x119d7a0) :1 +0x84 fp=0x402e9e2c70 sp=0x402e9e2bd0 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e2d68, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e2cf0 sp=0x402e9e2c70 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e2e70 sp=0x402e9e2cf0 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e2fa0 sp=0x402e9e2e70 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a131e0, 0x1, 0x400e429410, 0x402e9e30b8, 0x119d7a0) :1 +0x84 fp=0x402e9e3040 sp=0x402e9e2fa0 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e3138, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e30c0 sp=0x402e9e3040 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e3240 sp=0x402e9e30c0 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e3370 sp=0x402e9e3240 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13160, 0x1, 0x400e4293b0, 0x402e9e3488, 0x119d7a0) :1 +0x84 fp=0x402e9e3410 sp=0x402e9e3370 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e3508, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e3490 sp=0x402e9e3410 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e3610 sp=0x402e9e3490 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e3740 sp=0x402e9e3610 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a130c0, 0x1, 0x400e429350, 0x402e9e3858, 0x119d7a0) :1 +0x84 fp=0x402e9e37e0 sp=0x402e9e3740 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e38d8, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e3860 sp=0x402e9e37e0 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e39e0 sp=0x402e9e3860 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e3b10 sp=0x402e9e39e0 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13040, 0x1, 0x400e4292f0, 0x402e9e3c28, 0x119d7a0) :1 +0x84 fp=0x402e9e3bb0 sp=0x402e9e3b10 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e3ca8, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e3c30 sp=0x402e9e3bb0 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e3db0 sp=0x402e9e3c30 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e3ee0 sp=0x402e9e3db0 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12fa0, 0x1, 0x400e429290, 0x402e9e3ff8, 0x119d7a0) :1 +0x84 fp=0x402e9e3f80 sp=0x402e9e3ee0 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e4078, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e4000 sp=0x402e9e3f80 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e4180 sp=0x402e9e4000 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e42b0 sp=0x402e9e4180 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12f20, 0x1, 0x400e429230, 0x402e9e43c8, 0x119d7a0) :1 +0x84 fp=0x402e9e4350 sp=0x402e9e42b0 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e4448, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e43d0 sp=0x402e9e4350 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e4550 sp=0x402e9e43d0 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e4680 sp=0x402e9e4550 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12e80, 0x1, 0x400e4291d0, 0x402e9e4798, 0x119d7a0) :1 +0x84 fp=0x402e9e4720 sp=0x402e9e4680 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e4818, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e47a0 sp=0x402e9e4720 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e4920 sp=0x402e9e47a0 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e4a50 sp=0x402e9e4920 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12e00, 0x1, 0x400e429170, 0x402e9e4b68, 0x119d7a0) :1 +0x84 fp=0x402e9e4af0 sp=0x402e9e4a50 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e4be8, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e4b70 sp=0x402e9e4af0 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e4cf0 sp=0x402e9e4b70 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e4e20 sp=0x402e9e4cf0 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12d60, 0x1, 0x400e429110, 0x402e9e4f38, 0x119d7a0) :1 +0x84 fp=0x402e9e4ec0 sp=0x402e9e4e20 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e4fb8, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e4f40 sp=0x402e9e4ec0 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e50c0 sp=0x402e9e4f40 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e51f0 sp=0x402e9e50c0 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12ce0, 0x1, 0x400e4290b0, 0x402e9e5308, 0x119d7a0) :1 +0x84 fp=0x402e9e5290 sp=0x402e9e51f0 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e5388, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e5310 sp=0x402e9e5290 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e5490 sp=0x402e9e5310 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e55c0 sp=0x402e9e5490 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12c40, 0x1, 0x400e429050, 0x402e9e56d8, 0x119d7a0) :1 +0x84 fp=0x402e9e5660 sp=0x402e9e55c0 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e5758, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e56e0 sp=0x402e9e5660 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e5860 sp=0x402e9e56e0 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e5990 sp=0x402e9e5860 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12bc0, 0x1, 0x400e428ff0, 0x402e9e5aa8, 0x119d7a0) :1 +0x84 fp=0x402e9e5a30 sp=0x402e9e5990 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e5b28, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e5ab0 sp=0x402e9e5a30 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e5c30 sp=0x402e9e5ab0 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e5d60 sp=0x402e9e5c30 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12b20, 0x1, 0x400e428f90, 0x402e9e5e78, 0x119d7a0) :1 +0x84 fp=0x402e9e5e00 sp=0x402e9e5d60 pc=0x11ac734 go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e5ef8, 0x119b7c8, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e5e80 sp=0x402e9e5e00 pc=0x11a2808 go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e6000 sp=0x402e9e5e80 pc=0x119d80c go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...) /srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e6130 sp=0x402e9e6000 pc=0x11a2c20 go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12aa0, 0x1, 0x400e428f30, 0x402e9e6248, 0x119d7a0) :1 +0x84 fp=0x402e9e61d0 sp=0x402e9e6130 pc=0x11ac734 ...additional frames elided... goroutine 6 [select]: go.opencensus.io/stats/view.(*worker).start(0x4000184300) /srv/lotus/go/pkg/mod/go.opencensus.io@v0.22.5/stats/view/worker.go:276 +0xb4 created by go.opencensus.io/stats/view.init.0 /srv/lotus/go/pkg/mod/go.opencensus.io@v0.22.5/stats/view/worker.go:34 +0x70 goroutine 66 [select]: github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0x40002ba0c0) /srv/lotus/go/pkg/mod/github.com/ipfs/go-log@v1.0.4/writer/writer.go:71 +0xd0 created by github.com/ipfs/go-log/writer.NewMirrorWriter /srv/lotus/go/pkg/mod/github.com/ipfs/go-log@v1.0.4/writer/writer.go:36 +0xac goroutine 98 [select]: github.com/libp2p/go-libp2p-connmgr.(*decayer).process(0x40014a1180) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-connmgr@v0.2.4/decay.go:165 +0x1a4 created by github.com/libp2p/go-libp2p-connmgr.NewDecayer /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-connmgr@v0.2.4/decay.go:95 +0x1d8 goroutine 69 [chan receive]: github.com/filecoin-project/lotus/cli/util.ReqContext.func1(0x4001048b40, 0x40010641f0) /srv/lotus/lotus/cli/util/api.go:268 +0x2c created by github.com/filecoin-project/lotus/cli/util.ReqContext /srv/lotus/lotus/cli/util/api.go:267 +0xe0 goroutine 71 [syscall]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:147 +0xc4 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x20 created by os/signal.Notify.func1.1 /usr/local/go/src/os/signal/signal.go:150 +0x44 goroutine 354 [IO wait]: internal/poll.runtime_pollWait(0xffff80a8e3b8, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000760018, 0x72, 0x0, 0x10, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).ReadFrom(0x4000760000, 0x400075c030, 0x10, 0x10, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:215 +0x15c net.(*netFD).readFrom(0x4000760000, 0x400075c030, 0x10, 0x10, 0x4000760000, 0x2c6cfe0, 0x40010a4d38, 0x682f24, 0x4000760000) /usr/local/go/src/net/fd_posix.go:61 +0x44 net.(*UDPConn).readFrom(0x4000618010, 0x400075c030, 0x10, 0x10, 0x2, 0x2, 0x2, 0x0) /usr/local/go/src/net/udpsock_posix.go:47 +0x44 net.(*UDPConn).ReadFromUDP(0x4000618010, 0x400075c030, 0x10, 0x10, 0x2, 0x0, 0x0, 0x85623f3b) /usr/local/go/src/net/udpsock.go:106 +0x48 github.com/jackpal/go-nat-pmp.(*network).call(0x40005ea060, 0x400075c000, 0x2, 0x2, 0x0, 0x400075c030, 0x10, 0x10, 0x0, 0x0) /srv/lotus/go/pkg/mod/github.com/jackpal/go-nat-pmp@v1.0.2/network.go:54 +0x208 github.com/jackpal/go-nat-pmp.(*Client).rpc(0x40005ea0c0, 0x400075c000, 0x2, 0x2, 0xc, 0x0, 0x0, 0x1, 0x4000129f48, 0x141b878) /srv/lotus/go/pkg/mod/github.com/jackpal/go-nat-pmp@v1.0.2/natpmp.go:108 +0x54 github.com/jackpal/go-nat-pmp.(*Client).GetExternalAddress(0x40005ea0c0, 0x40005ea0c0, 0x40019e2480, 0x0) /srv/lotus/go/pkg/mod/github.com/jackpal/go-nat-pmp@v1.0.2/natpmp.go:58 +0x64 github.com/libp2p/go-nat.discoverNATPMPWithAddr.func1(0x40014ca240, 0x400e8ac038, 0x4, 0x6c8) /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/natpmp.go:44 +0xd0 created by github.com/libp2p/go-nat.discoverNATPMPWithAddr /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/natpmp.go:41 +0x6c goroutine 179 [chan receive]: github.com/dgraph-io/badger/v2.(*valueLog).waitOnGC(0x40014815d0, 0x4001056020) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1851 +0x58 created by github.com/dgraph-io/badger/v2.Open /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:415 +0xd08 goroutine 93 [select]: github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0x400122c680) /srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x1a8 created by github.com/syndtr/goleveldb/leveldb.openDB /srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x440 goroutine 91 [select]: github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0x400122c680) /srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0x88 created by github.com/syndtr/goleveldb/leveldb.openDB /srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x318 goroutine 92 [select]: github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0x400122c680) /srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0xb4 created by github.com/syndtr/goleveldb/leveldb.openDB /srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x338 goroutine 33 [select]: github.com/libp2p/go-libp2p-peerstore/pstoremem.(*memoryAddrBook).background(0x4000301b00) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.6/pstoremem/addr_book.go:93 +0xe4 created by github.com/libp2p/go-libp2p-peerstore/pstoremem.NewAddrBook /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.6/pstoremem/addr_book.go:83 +0x164 goroutine 32 [select]: github.com/raulk/go-watchdog.pollingWatchdog(0x2e3fdc0, 0x400120af00, 0x12a05f200, 0x3e2f572000, 0x4000735410) /srv/lotus/go/pkg/mod/github.com/raulk/go-watchdog@v1.0.1/watchdog.go:320 +0x1ac created by github.com/raulk/go-watchdog.SystemDriven /srv/lotus/go/pkg/mod/github.com/raulk/go-watchdog@v1.0.1/watchdog.go:268 +0x1a0 goroutine 99 [select]: github.com/libp2p/go-libp2p-connmgr.(*BasicConnMgr).background(0x4000302d00) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-connmgr@v0.2.4/connmgr.go:246 +0x108 created by github.com/libp2p/go-libp2p-connmgr.NewConnManager /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-connmgr@v0.2.4/connmgr.go:140 +0x25c goroutine 101 [select]: github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0x40006e3880) /srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0xe0 created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool /srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x154 goroutine 94 [select]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0x400122c680) /srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0xe4 created by github.com/syndtr/goleveldb/leveldb.openDB /srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x460 goroutine 156 [select]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0x400123fe00, 0x400000fb80) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:212 +0x1dc created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:70 +0x78 goroutine 157 [select]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0x400123fe30, 0x400000fb80) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:212 +0x1dc created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:70 +0x78 goroutine 158 [select]: github.com/dgraph-io/badger/v2.(*DB).updateSize(0x4001481400, 0x400000fc60) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:1200 +0x108 created by github.com/dgraph-io/badger/v2.Open /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:360 +0x724 goroutine 162 [select]: github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0x4001030070, 0x0, 0x40014942c0) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:390 +0x1fc created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:371 +0x80 goroutine 163 [select]: github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0x4001030070, 0x1, 0x40014942c0) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:390 +0x1fc created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:371 +0x80 goroutine 164 [chan receive]: github.com/dgraph-io/badger/v2.(*DB).flushMemtable(0x4001481400, 0x40014942e0, 0x0, 0x0) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:1107 +0x78 github.com/dgraph-io/badger/v2.Open.func5(0x40001320e8) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:377 +0x34 created by github.com/dgraph-io/badger/v2.Open /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:376 +0xf50 goroutine 178 [select]: github.com/dgraph-io/badger/v2.(*DB).doWrites(0x4001481400, 0x4001056000) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:862 +0x2fc created by github.com/dgraph-io/badger/v2.Open /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:411 +0xb44 goroutine 166 [select]: github.com/dgraph-io/badger/v2.(*valueLog).flushDiscardStats(0x40014815d0) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1948 +0x108 created by github.com/dgraph-io/badger/v2.(*valueLog).open /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1114 +0x50 goroutine 180 [select]: github.com/dgraph-io/badger/v2.(*publisher).listenForUpdates(0x400123fec0, 0x4001056040) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/publisher.go:67 +0xfc created by github.com/dgraph-io/badger/v2.Open /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:419 +0xc04 goroutine 102 [select]: github.com/ipfs/go-ds-badger2.(*Datastore).periodicGC(0x4001962120) /srv/lotus/go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:172 +0xc0 created by github.com/ipfs/go-ds-badger2.NewDatastore /srv/lotus/go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:160 +0x4e4 goroutine 228 [select]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0x40011588a0, 0x40006fc540) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:212 +0x1dc created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:70 +0x78 goroutine 229 [select]: github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0x40011588d0, 0x40006fc540) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:212 +0x1dc created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:70 +0x78 goroutine 230 [select]: github.com/dgraph-io/badger/v2.(*DB).updateSize(0x4000700c00, 0x40006fc5e0) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:1200 +0x108 created by github.com/dgraph-io/badger/v2.Open /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:360 +0x724 goroutine 232 [select]: github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0x40000fe8c0, 0x0, 0x400074e5e0) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:390 +0x1fc created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:371 +0x80 goroutine 233 [select]: github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0x40000fe8c0, 0x1, 0x400074e5e0) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:390 +0x1fc created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:371 +0x80 goroutine 234 [chan receive]: github.com/dgraph-io/badger/v2.(*DB).flushMemtable(0x4000700c00, 0x400074e600, 0x0, 0x0) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:1107 +0x78 github.com/dgraph-io/badger/v2.Open.func5(0x40002da010) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:377 +0x34 created by github.com/dgraph-io/badger/v2.Open /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:376 +0xf50 goroutine 237 [select]: github.com/dgraph-io/badger/v2.(*DB).doWrites(0x4000700c00, 0x400074e760) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:862 +0x2fc created by github.com/dgraph-io/badger/v2.Open /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:411 +0xb44 goroutine 236 [select]: github.com/dgraph-io/badger/v2.(*valueLog).flushDiscardStats(0x4000700dd0) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1948 +0x108 created by github.com/dgraph-io/badger/v2.(*valueLog).open /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1114 +0x50 goroutine 238 [chan receive]: github.com/dgraph-io/badger/v2.(*valueLog).waitOnGC(0x4000700dd0, 0x400074e780) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1851 +0x58 created by github.com/dgraph-io/badger/v2.Open /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:415 +0xd08 goroutine 239 [select]: github.com/dgraph-io/badger/v2.(*publisher).listenForUpdates(0x4001158960, 0x400074e7a0) /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/publisher.go:67 +0xfc created by github.com/dgraph-io/badger/v2.Open /srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:419 +0xc04 goroutine 240 [select]: github.com/ipfs/go-ds-badger2.(*Datastore).periodicGC(0x40019f9380) /srv/lotus/go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:172 +0xc0 created by github.com/ipfs/go-ds-badger2.NewDatastore /srv/lotus/go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:160 +0x4e4 goroutine 289 [select]: github.com/libp2p/go-libp2p/p2p/protocol/identify.(*ObservedAddrManager).worker(0x4001f17980, 0x2e7df60, 0x4000641a40) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/protocol/identify/obsaddr.go:236 +0x1a8 created by github.com/libp2p/go-libp2p/p2p/protocol/identify.NewObservedAddrManager /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/protocol/identify/obsaddr.go:127 +0x16c goroutine 290 [select]: github.com/filecoin-project/lotus/lib/backupds.(*Datastore).runLog(0x4000640a00, 0x4001f18008) /srv/lotus/lotus/lib/backupds/log.go:76 +0xc4 created by github.com/filecoin-project/lotus/lib/backupds.(*Datastore).startLog /srv/lotus/lotus/lib/backupds/log.go:68 +0x3b8 goroutine 318 [select]: github.com/libp2p/go-nat.discoverNATPMP.func1(0x400e88b620, 0x400e8ac038, 0x4, 0x6c8, 0x2e7dfe0, 0x400e88b560) /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/natpmp.go:28 +0xdc created by github.com/libp2p/go-nat.discoverNATPMP /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/natpmp.go:23 +0xa0 goroutine 288 [select]: github.com/libp2p/go-libp2p-swarm.(*DialBackoff).background(0x4000703790, 0x2e7df60, 0x4000640cc0) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.3.1/swarm_dial.go:126 +0xd8 created by github.com/libp2p/go-libp2p-swarm.(*DialBackoff).init /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.3.1/swarm_dial.go:119 +0x54 goroutine 315 [select]: github.com/libp2p/go-nat.DiscoverNATs.func1(0x40000d94a0, 0x2e7dfe0, 0x400e88b560) /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/nat.go:54 +0x1c0 created by github.com/libp2p/go-nat.DiscoverNATs /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/nat.go:42 +0x60 goroutine 286 [select]: github.com/jbenet/goprocess/context.CloseAfterContext.func1(0x2e7df60, 0x4000640c80, 0x2e9bf40, 0x4001f17800) /srv/lotus/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/context/context.go:65 +0xa0 created by github.com/jbenet/goprocess/context.CloseAfterContext /srv/lotus/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/context/context.go:64 +0xa4 goroutine 316 [IO wait]: internal/poll.runtime_pollWait(0xffff80a8e670, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000138218, 0x72, 0x800, 0x800, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).ReadFrom(0x4000138200, 0x4000c3c000, 0x800, 0x800, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:215 +0x15c net.(*netFD).readFrom(0x4000138200, 0x4000c3c000, 0x800, 0x800, 0x4000c3c000, 0xffff800c52d0, 0x4000138201, 0x400e98e000, 0x1) /usr/local/go/src/net/fd_posix.go:61 +0x44 net.(*UDPConn).readFrom(0x40002da1b0, 0x4000c3c000, 0x800, 0x800, 0x400e9849b8, 0x13f84dc, 0x400e9849b8, 0x13f8694) /usr/local/go/src/net/udpsock_posix.go:47 +0x44 net.(*UDPConn).ReadFrom(0x40002da1b0, 0x4000c3c000, 0x800, 0x800, 0x2e659a0, 0x4001058150, 0x87, 0x0, 0x0) /usr/local/go/src/net/udpsock.go:118 +0x48 github.com/huin/goupnp/httpu.(*HTTPUClient).Do(0x40006fdd00, 0x400e988000, 0x7d2b7500, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0) /srv/lotus/go/pkg/mod/github.com/huin/goupnp@v1.0.0/httpu/httpu.go:107 +0x4bc github.com/huin/goupnp/ssdp.SSDPRawSearch(0x40006fdd00, 0x2b0d083, 0x31, 0x2, 0x3, 0x4001030a10, 0x0, 0x0, 0x0, 0x0) /srv/lotus/go/pkg/mod/github.com/huin/goupnp@v1.0.0/ssdp/ssdp.go:58 +0x3b4 github.com/huin/goupnp.DiscoverDevices(0x2b0d083, 0x31, 0x0, 0x0, 0x0, 0x0, 0x0) /srv/lotus/go/pkg/mod/github.com/huin/goupnp@v1.0.0/goupnp.go:66 +0x88 github.com/libp2p/go-nat.discoverUPNP_IG1.func1(0x40000d9500, 0x2e7dfe0, 0x400e88b560) /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/upnp.go:27 +0x5c created by github.com/libp2p/go-nat.discoverUPNP_IG1 /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/upnp.go:23 +0x60 goroutine 317 [IO wait]: internal/poll.runtime_pollWait(0xffff80a8e588, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000184018, 0x72, 0x800, 0x800, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).ReadFrom(0x4000184000, 0x400e8ce000, 0x800, 0x800, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:215 +0x15c net.(*netFD).readFrom(0x4000184000, 0x400e8ce000, 0x800, 0x800, 0x400e8ce000, 0xffff48159170, 0x4000184001, 0x400dabe000, 0x1) /usr/local/go/src/net/fd_posix.go:61 +0x44 net.(*UDPConn).readFrom(0x40005fa080, 0x400e8ce000, 0x800, 0x800, 0x400e9809b8, 0x13f84dc, 0x400e9809b8, 0x13f8694) /usr/local/go/src/net/udpsock_posix.go:47 +0x44 net.(*UDPConn).ReadFrom(0x40005fa080, 0x400e8ce000, 0x800, 0x800, 0x2e659a0, 0x400daaf020, 0x87, 0x0, 0x0) /usr/local/go/src/net/udpsock.go:118 +0x48 github.com/huin/goupnp/httpu.(*HTTPUClient).Do(0x4001494460, 0x400dab6000, 0x7d2b7500, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0) /srv/lotus/go/pkg/mod/github.com/huin/goupnp@v1.0.0/httpu/httpu.go:107 +0x4bc github.com/huin/goupnp/ssdp.SSDPRawSearch(0x4001494460, 0x2b0d0b4, 0x31, 0x2, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0) /srv/lotus/go/pkg/mod/github.com/huin/goupnp@v1.0.0/ssdp/ssdp.go:58 +0x3b4 github.com/huin/goupnp.DiscoverDevices(0x2b0d0b4, 0x31, 0x0, 0x0, 0x0, 0x0, 0x0) /srv/lotus/go/pkg/mod/github.com/huin/goupnp@v1.0.0/goupnp.go:66 +0x88 github.com/libp2p/go-nat.discoverUPNP_IG2.func1(0x40000d9560, 0x2e7dfe0, 0x400e88b560) /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/upnp.go:84 +0x5c created by github.com/libp2p/go-nat.discoverUPNP_IG2 /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/upnp.go:80 +0x60 goroutine 338 [select]: github.com/libp2p/go-libp2p/p2p/protocol/identify.(*IDService).loop(0x400033d6c0) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/protocol/identify/id.go:203 +0x39c created by github.com/libp2p/go-libp2p/p2p/protocol/identify.NewIDService /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/protocol/identify/id.go:150 +0x250 goroutine 314 [chan receive]: github.com/libp2p/go-nat.DiscoverGateway(0x0, 0x0, 0x0, 0x0) /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/nat.go:93 +0xcc github.com/libp2p/go-libp2p-nat.DiscoverNAT.func1(0x40000d93e0, 0x40006a2800, 0x40006a2810) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-nat@v0.0.6/nat.go:42 +0x48 created by github.com/libp2p/go-libp2p-nat.DiscoverNAT /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-nat@v0.0.6/nat.go:39 +0x98 goroutine 341 [select]: github.com/libp2p/go-libp2p-nat.DiscoverNAT(0x2e7df60, 0x400e872480, 0x2e9bf40, 0x4001f17a40, 0x2e7df60) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-nat@v0.0.6/nat.go:45 +0x100 github.com/libp2p/go-libp2p/p2p/host/basic.(*natManager).start.func1(0x2e9bf40, 0x4001f17a40) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/host/basic/natmgr.go:89 +0x90 github.com/jbenet/goprocess.(*process).Go.func1(0x400110d050, 0x4001f17a40, 0x4001f17aa0) /srv/lotus/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/impl-mutex.go:134 +0x3c created by github.com/jbenet/goprocess.(*process).Go /srv/lotus/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/impl-mutex.go:133 +0x1bc goroutine 319 [IO wait]: internal/poll.runtime_pollWait(0xffff80a8e4a0, 0x72, 0x40006a2d10) /usr/local/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40014a1798, 0x72, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).RawRead(0x40014a1780, 0x400e8776e0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:533 +0xe0 net.(*rawConn).Read(0x4001f181c0, 0x400e8776e0, 0x1, 0x1) /usr/local/go/src/net/rawconn.go:43 +0x60 golang.org/x/net/internal/socket.(*Conn).recvMsg(0x4001056c60, 0x40014c9c60, 0x0, 0x0, 0x40014c9c68) /srv/lotus/go/pkg/mod/golang.org/x/net@v0.0.0-20201021035429-f5854403a974/internal/socket/rawconn_msg.go:32 +0x1ac golang.org/x/net/internal/socket.(*Conn).RecvMsg(...) /srv/lotus/go/pkg/mod/golang.org/x/net@v0.0.0-20201021035429-f5854403a974/internal/socket/socket.go:247 golang.org/x/net/ipv4.(*payloadHandler).ReadFrom(0x4001f070a0, 0x400e8be000, 0xffff, 0xffff, 0x0, 0x0, 0x560b400, 0x2329, 0x400e86cb20, 0x4) /srv/lotus/go/pkg/mod/golang.org/x/net@v0.0.0-20201021035429-f5854403a974/ipv4/payload_cmsg.go:31 +0x178 github.com/koron/go-ssdp.(*multicastConn).readPackets(0x400e8775c0, 0x12a05f200, 0x40014c9dc0, 0x96, 0x2e659a0) /srv/lotus/go/pkg/mod/github.com/koron/go-ssdp@v0.0.0-20191105050749-2e1c40ed0b5d/multicast.go:103 +0x7c github.com/koron/go-ssdp.Search(0x29d9a02, 0x8, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /srv/lotus/go/pkg/mod/github.com/koron/go-ssdp@v0.0.0-20191105050749-2e1c40ed0b5d/search.go:100 +0x1a8 github.com/libp2p/go-nat.discoverUPNP_GenIGDev.func1(0x400e88b680, 0x2e7dfe0, 0x400e88b560) /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/upnp.go:154 +0x70 created by github.com/libp2p/go-nat.discoverUPNP_GenIGDev /srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/upnp.go:151 +0x64 goroutine 342 [select]: github.com/libp2p/go-libp2p-peerstore/pstoremem.(*memoryAddrBook).background(0x4000301200) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.6/pstoremem/addr_book.go:93 +0xe4 created by github.com/libp2p/go-libp2p-peerstore/pstoremem.NewAddrBook /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.6/pstoremem/addr_book.go:83 +0x164 goroutine 343 [select]: github.com/jbenet/goprocess/context.CloseAfterContext.func1(0x2e7df60, 0x4000640cc0, 0x2e9bf40, 0x4001f17c80) /srv/lotus/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/context/context.go:65 +0xa0 created by github.com/jbenet/goprocess/context.CloseAfterContext /srv/lotus/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/context/context.go:64 +0xa4 goroutine 345 [select]: github.com/libp2p/go-libp2p-swarm.(*DialBackoff).background(0x400130ab90, 0x2e7df60, 0x4000641c40) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.3.1/swarm_dial.go:126 +0xd8 created by github.com/libp2p/go-libp2p-swarm.(*DialBackoff).init /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.3.1/swarm_dial.go:119 +0x54 goroutine 346 [select]: github.com/libp2p/go-libp2p-autonat.(*autoNATService).background(0x40005dadc0, 0x2e7df60, 0x4000641e80) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-autonat@v0.4.0/svc.go:217 +0x184 created by github.com/libp2p/go-libp2p-autonat.(*autoNATService).Enable /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-autonat@v0.4.0/svc.go:197 +0xd8 goroutine 349 [select]: github.com/libp2p/go-libp2p-autonat.(*AmbientAutoNAT).background(0x400033dba0) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-autonat@v0.4.0/autonat.go:175 +0x1fc created by github.com/libp2p/go-libp2p-autonat.New /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-autonat@v0.4.0/autonat.go:125 +0x590 goroutine 350 [select]: github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).background(0x40002ca2c0) /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/host/basic/basic_host.go:520 +0x1cc created by github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).Start /srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/host/basic/basic_host.go:368 +0x58 ```
ribasushi commented 3 years ago

Commenting out this line: https://github.com/filecoin-project/lotus/blob/v1.6.0/node/builder.go#L704

Gets us to:

...
2021/04/06 13:42:21 [Fx] PROVIDE    fx.Shutdowner <= go.uber.org/fx.(*App).shutdowner-fm()
2021/04/06 13:42:21 [Fx] PROVIDE    fx.DotGraph <= go.uber.org/fx.(*App).dotGraph-fm()
2021/04/06 13:42:21 [Fx] INVOKE     github.com/filecoin-project/lotus/node/modules.MemoryWatchdog()
2021/04/06 13:42:21 [Fx] Error during "github.com/filecoin-project/lotus/node/modules.MemoryWatchdog()" invoke: cycle detected in dependency graph: dtypes.ChainBlockstore provided by "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_arm64.s:12)
    depends on dtypes.ChainBlockstore provided by "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_arm64.s:12)
ERROR: initializing node: starting node: cycle detected in dependency graph: dtypes.ChainBlockstore provided by "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_arm64.s:12)
    depends on dtypes.ChainBlockstore provided by "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_arm64.s:12)

I assume something splitstore-related?

jennijuju commented 3 years ago

@vyzo @raulk thoughts on ^ ?

vyzo commented 3 years ago

DI seems to fall into a loop; @magik6k might be the only person who really understands how the DI contraptions work.

raulk commented 3 years ago

@ribasushi does this node have the splitstore enabled?

ribasushi commented 3 years ago

@raulk No, stock config.

firesWu commented 3 years ago

same error

jennijuju commented 3 years ago

DI seems to fall into a loop; @magik6k might be the only person who really understands how the DI contraptions work.

@magik6k gentle ping^