search

attic-labs / noms

The versioned, forkable, syncable database
Apache License 2.0
7.45k stars 267 forks source link

We easily hit ulimit for file handles on os x default configuration #3834

Open aboodman opened 5 years ago

aboodman commented 5 years ago

Because defaultMaxTables in nbs is 256 and the default open files on os x is also 256, we can easily hit this value and the failure is pretty cryptic. Simple example:

noms!? for counter in {1..249}; do noms list new /tmp/db1 $counter; done
#0oqtibm9uhro98kaopnv7htd068rua29
#vpoqol3jc88745ce4vh9g35n3uhubrl9
... 244 lines elided ...
#fkrrtnva3aq4jvh0et79e320ukv5nnbt
#c5iavhl36qencrdr7bt2682cqmndtsq0
        Error Trace:    try.go:99
                    try.go:44
                    file_manifest.go:78
                    file_manifest.go:126
                    file_manifest.go:125
                    manifest.go:187
                    store.go:452
                    store.go:452
                    store.go:398
                    value_store.go:361
                    value_store.go:370
                    database_common.go:229
                    database_common.go:212
                    database_common.go:195
                    database_common.go:264
                    database_common.go:195
                    database_common.go:67
                    util.go:67
                    noms_list.go:123
                    noms_list.go:59
                    noms_list.go:43
                    noms.go:100
                    proc.go:201
                    asm_amd64.s:1333
    Error:          open /tmp/db1/manifest: too many open files

goroutine 1 [running]:
github.com/attic-labs/noms/go/d.PanicIfError(0x1b17880, 0xc00009c810)
    /Users/aa/src/github.com/attic-labs/noms/go/d/try.go:44 +0x65
github.com/attic-labs/noms/go/nbs.openIfExists(0xc0003a19a0, 0x11, 0x300000002)
    /Users/aa/src/github.com/attic-labs/noms/go/nbs/file_manifest.go:78 +0x9a
github.com/attic-labs/noms/go/nbs.fileManifest.Update.func3(0xc0003a19a0, 0x11, 0x369f2a9c27ea6c7a, 0x9dfb30589c8c0659, 0xf14576f8, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/aa/src/github.com/attic-labs/noms/go/nbs/file_manifest.go:126 +0x6f
github.com/attic-labs/noms/go/nbs.fileManifest.Update(0x7ffeefbffc26, 0x8, 0x369f2a9c27ea6c7a, 0x9dfb30589c8c0659, 0xf14576f8, 0x183e82b, 0x4, 0x988d872e1b087f31, 0xc653fab98fdb9de4, 0xb2d647e8c1753072, ...)
    /Users/aa/src/github.com/attic-labs/noms/go/nbs/file_manifest.go:125 +0x26f
github.com/attic-labs/noms/go/nbs.manifestManager.Update(0x1b1ecc0, 0xc0001d6cf0, 0xc0000adcc0, 0xc000202a00, 0x369f2a9c27ea6c7a, 0x9dfb30589c8c0659, 0x988d872ef14576f8, 0x183e82b, 0x4, 0x988d872e1b087f31, ...)
    /Users/aa/src/github.com/attic-labs/noms/go/nbs/manifest.go:187 +0x250
github.com/attic-labs/noms/go/nbs.(*NomsBlockStore).updateManifest(0xc0000c18c0, 0x9a4713dab2d647e8, 0xe8eb924ce857a967, 0x4ff72d1a4329a870, 0x9f82a9cbd00007c4, 0xb20f3a3c902e441, 0x0, 0x0)
    /Users/aa/src/github.com/attic-labs/noms/go/nbs/store.go:452 +0x777
github.com/attic-labs/noms/go/nbs.(*NomsBlockStore).Commit(0xc0000c18c0, 0x9a4713dab2d647e8, 0xe8eb924ce857a967, 0x4ff72d1a4329a870, 0x9f82a9cbd00007c4, 0xb20f3a3c902e441, 0x800)
    /Users/aa/src/github.com/attic-labs/noms/go/nbs/store.go:398 +0x1a0
github.com/attic-labs/noms/go/types.(*ValueStore).Commit.func1(0xc000496000, 0x9a4713dab2d647e8, 0xe8eb924ce857a967, 0x4ff72d1a4329a870, 0x9f82a9cbd00007c4, 0xb20f3a3c902e441, 0x0)
    /Users/aa/src/github.com/attic-labs/noms/go/types/value_store.go:361 +0x4d8
github.com/attic-labs/noms/go/types.(*ValueStore).Commit(0xc000496000, 0x9a4713dab2d647e8, 0xe8eb924ce857a967, 0x4ff72d1a4329a870, 0x9f82a9cbd00007c4, 0xb20f3a3c902e441, 0x4)
    /Users/aa/src/github.com/attic-labs/noms/go/types/value_store.go:370 +0x53
github.com/attic-labs/noms/go/datas.(*database).tryCommitChunks(0xc0003a2c80, 0x1b26b40, 0xc000408c30, 0xd00007c44ff72d1a, 0xc902e4419f82a9cb, 0xc00b20f3a3, 0xc0001c7201, 0x15c5f7d)
    /Users/aa/src/github.com/attic-labs/noms/go/datas/database_common.go:229 +0x160
github.com/attic-labs/noms/go/datas.(*database).doDelete(0xc0003a2c80, 0xc0003a6cf0, 0x28, 0x1b23900, 0xc0003a8bc0)
    /Users/aa/src/github.com/attic-labs/noms/go/datas/database_common.go:212 +0x326
github.com/attic-labs/noms/go/datas.(*database).Delete.func1(0x1b25aa0, 0xc0003a2c80, 0xc0003a6cf0, 0x28, 0x1b23900, 0xc0003a8bc0, 0x0, 0x0)
    /Users/aa/src/github.com/attic-labs/noms/go/datas/database_common.go:195 +0x5f
github.com/attic-labs/noms/go/datas.(*database).doHeadUpdate(0xc0003a2c80, 0x1b25aa0, 0xc0003a2c80, 0xc0003a6cf0, 0x28, 0x1b23900, 0xc0003a8bc0, 0xc0001c7530, 0x0, 0x0, ...)
    /Users/aa/src/github.com/attic-labs/noms/go/datas/database_common.go:264 +0x7d
github.com/attic-labs/noms/go/datas.(*database).Delete(0xc0003a2c80, 0x1b25aa0, 0xc0003a2c80, 0xc0003a6cf0, 0x28, 0x1b23900, 0xc0003a8bc0, 0x0, 0x0, 0x0, ...)
    /Users/aa/src/github.com/attic-labs/noms/go/datas/database_common.go:195 +0xc6
github.com/attic-labs/noms/go/datas.(*database).Flush(0xc0003a2c80)
    /Users/aa/src/github.com/attic-labs/noms/go/datas/database_common.go:67 +0x2ee
main.appplyPatch(0x183e590, 0x3, 0x7ffeefbffc26, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/aa/src/github.com/attic-labs/noms/cmd/noms/util.go:67 +0x21b
main.applyListInserts(0x183e590, 0x3, 0x7ffeefbffc26, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/aa/src/github.com/attic-labs/noms/cmd/noms/noms_list.go:123 +0x62d
main.nomsListNew(0x7ffeefbffc26, 0x8, 0xc0001d6cc0, 0x1, 0x1, 0x1)
    /Users/aa/src/github.com/attic-labs/noms/cmd/noms/noms_list.go:59 +0x24f
main.nomsList.func1(0xc0000a3a80, 0x8, 0xc0000a3a80)
    /Users/aa/src/github.com/attic-labs/noms/cmd/noms/noms_list.go:43 +0x35c
main.main()
    /Users/aa/src/github.com/attic-labs/noms/cmd/noms/noms.go:100 +0xd78
noms!?

A principled solution would be to somehow detect max open file handles and set it based on that, but this is going to be system dependent.

Maybe a simple solution would be to drop it back to 128, which was the original value. It doesn't seem to have been updated to 256 for a particularly specific reason. Or maybe 192 :-).

https://github.com/attic-labs/noms/pull/3489/files#diff-bf374388e9c55e661083dd4d2d0a1305R21

aboodman commented 5 years ago

Note there are probably other modes where Noms is going to use even more file handles, like when talking to AWS. So it's hard to pick a good number. Maybe an alternate solution is to develop/test noms for a particular value and try to detect at startup and warn if not met.