search

wjdp / htmltest

:white_check_mark: Test generated HTML for problems
MIT License
323 stars 54 forks source link

too many open files when run in cocurrent mode #138

Open ssbarnea opened 4 years ago

ssbarnea commented 4 years ago

On MacOS when I enabed concurrent mode I got too many open files error even on a relative small blog.

panic: open public/categories/tools/page/1/index.xml/index.html: too many open files

goroutine 2641 [running]:
github.com/wjdp/htmltest/output.CheckErrorPanic(...)
/home/travis/gopath/src/github.com/wjdp/htmltest/output/error.go:22
github.com/wjdp/htmltest/htmldoc.(*Document).Parse(0xc0002e7100)
/home/travis/gopath/src/github.com/wjdp/htmltest/htmldoc/document.go:57 +0x253
github.com/wjdp/htmltest/htmltest.(*HTMLTest).testDocument(0xc0000dc6c0, 0xc0002e7100)
/home/travis/gopath/src/github.com/wjdp/htmltest/htmltest/htmltest.go:194 +0xf1
github.com/wjdp/htmltest/htmltest.(*HTMLTest).testDocuments.func1(0xc0002e35f0, 0xc0000dc6c0, 0xc0000ec0e0, 0xc0002e7100)
/home/travis/gopath/src/github.com/wjdp/htmltest/htmltest/htmltest.go:176 +0x69
created by github.com/wjdp/htmltest/htmltest.(*HTMLTest).testDocuments
/home/travis/gopath/src/githu

On MacOS max files is 256 unless you bump it, see https://wilsonmar.github.io/maximum-limits/

htmltest should detect unresonable small values and fail to start, or at least to disable a warning when max number of file handlers is too low.

I can confirm that running ulimit -n 1024 fixed the problem for my specific case but this ticket is about improving user experience.

wjdp commented 4 years ago

@ssbarnea Thanks for the report. I wouldn't recommend concurrent mode at present as it's not been tested much and there are several known major issues with it. Not much effort has gone into it as (with caching of external checks) htmltest is plenty fast enough for most applications.