search

replicon / fast-archiver

Alternative archiving tool with fast performance for huge numbers of small files
Other
179 stars 17 forks source link

Deadlock when directoryScanQueue is full #4

Closed mfenniak closed 12 years ago

mfenniak commented 12 years ago

If directoryScanQueue fills up and all directoryScanner goroutines are pending at "directoryScanQueue <- filePath", then a deadlock occurs where the workInProgress counter will never hit zero and all goroutines will get stuck shortly afterwards.

mfenniak commented 12 years ago 
throw: all goroutines are asleep - deadlock!

goroutine 1 [chan receive]:
main.archiveWriter(0xf84002ec00, 0xf840039270, 0xf84005f000, 0xf84002e690, 0xf840059300, ...)
        /home/mfenniak/fast-archiver/create-archive.go:139 +0x3b2
main.main()
        /home/mfenniak/fast-archiver/main.go:138 +0xdf5

...

Workaround when encountering this issue: increase --queue-dir to a larger value (eg. 10240), or increase --dir-readers to a larger value (128). The queue-dir increase causes a slight memory increase, but is probably the safest approach.