"too many open files" - Githubissues

kevinkle commented 5 years ago

MAP 18h40m16s nquad_count:11.02G err_count:0.000 nquad_speed:163.9k/sec edge_count:22.04G edge_speed:327.8k/sec
MAP 18h40m17s nquad_count:11.02G err_count:0.000 nquad_speed:163.9k/sec edge_count:22.04G edge_speed:327.8k/sec
MAP 18h40m18s nquad_count:11.02G err_count:0.000 nquad_speed:163.9k/sec edge_count:22.04G edge_speed:327.8k/sec
badger 2019/07/11 10:32:31 INFO: Storing value log head: {Fid:4 Len:47 Offset:41694835}
MAP 18h40m19s nquad_count:11.02G err_count:0.000 nquad_speed:163.9k/sec edge_count:22.04G edge_speed:327.8k/sec
MAP 18h40m20s nquad_count:11.02G err_count:0.000 nquad_speed:163.9k/sec edge_count:22.04G edge_speed:327.8k/sec
MAP 18h40m21s nquad_count:11.02G err_count:0.000 nquad_speed:163.9k/sec edge_count:22.04G edge_speed:327.8k/sec
MAP 18h40m22s nquad_count:11.02G err_count:0.000 nquad_speed:163.9k/sec edge_count:22.04G edge_speed:327.8k/sec
badger 2019/07/11 10:32:35 INFO: Force compaction on level 0 done
MAP 18h40m23s nquad_count:11.02G err_count:0.000 nquad_speed:163.9k/sec edge_count:22.04G edge_speed:327.8k/sec
Shard tmp/shards/000 -> Reduce tmp/shards/shard_0/000
badger 2019/07/11 10:32:36 INFO: All 0 tables opened in 0s
REDUCE 18h40m24s [0.00%] edge_count:0.000 edge_speed:0.000/sec plist_count:0.000 plist_speed:0.000/sec
2019/07/11 10:32:36 open tmp/shards/shard_0/000/002646.map: too many open files

github.com/dgraph-io/dgraph/x.Wrap
        /ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:91
github.com/dgraph-io/dgraph/x.Check
        /ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:41
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.readMapOutput
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/shuffle.go:80
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1333
2019/07/11 10:32:36 open tmp/shards/shard_0/000/000728.map: too many open files

github.com/dgraph-io/dgraph/x.Wrap
        /ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:91
github.com/dgraph-io/dgraph/x.Check
        /ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:41
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.readMapOutput
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/shuffle.go:80
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1333

kevinkle commented 5 years ago

https://github.com/dgraph-io/dgraph/issues/2432 https://serverfault.com/questions/610130/how-to-set-ulimit-value-permanently

kevinkle commented 5 years ago

Set it to 100,000 and re-running

kevinkle commented 4 years ago

This issue doesn't come up after setting 100,000. But we ran into https://github.com/superphy/prairiedog/issues/113

superphy / prairiedog

"too many open files" #111