search

internetarchive / Zeno

State-of-the-art web crawler 🔱
GNU Affero General Public License v3.0
69 stars 8 forks source link

Send on closed channel panic #123

Open CorentinB opened 1 month ago

CorentinB commented 1 month ago

Caused when CTRL+C a crawl, it was in finishing state then this happened.

panic: send on closed channel
        panic: send on closed channel

goroutine 778 [running]:
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Crawl).Capture.func1(0xc0016dd2c0)
        /X/Zeno/internal/pkg/crawl/capture.go:233 +0x50
panic({0x13699e0?, 0x169a9f0?})
        /var/www/.go/src/runtime/panic.go:770 +0x132
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Crawl).Capture(0xc00024f888, 0xc0016dd2c0)
        /X/Zeno/internal/pkg/crawl/capture.go:364 +0x12a7
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Worker).unsafeCapture(0xc000594480, 0xc0016dd2c0)
        /X/Zeno/internal/pkg/crawl/worker.go:147 +0xa5
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Worker).Run(0xc000594480)
        /X/Zeno/internal/pkg/crawl/worker.go:129 +0x7d1
created by github.com/internetarchive/Zeno/internal/pkg/crawl.(*WorkerPool).Start in goroutine 1
        /X/Zeno/internal/pkg/crawl/worker_pool.go:35 +0x119

Happened on 2 different VMs on 2 different crawls. The second one was more verbose.

panic: send on closed channel
        panic: send on closed channel

goroutine 210 [running]:
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Crawl).Capture.func1(0xc013a64180)
        /X/Zeno/internal/pkg/crawl/capture.go:233 +0x50
panic({0x13699e0?, 0x169a9f0?})
        /var/www/.go/src/runtime/panic.go:770 +0x132
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Crawl).Capture(0xc000492008, 0xc013a64180)
        /X/Zeno/internal/pkg/crawl/capture.go:364 +0x12a7
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Worker).unsafeCapture(0xc0003c84c0, 0xc013a64180)
        /X/Zeno/internal/pkg/crawl/worker.go:147 +0xa5
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Worker).Run(0xc0003c84c0)
        /X/Zeno/internal/pkg/crawl/worker.go:129 +0x7d1
created by github.com/internetarchive/Zeno/internal/pkg/crawl.(*WorkerPool).Start in goroutine 1
        /X/Zeno/internal/pkg/crawl/worker_pool.go:35 +0x119
panic: send on closed channel
        panic: send on closed channel

goroutine 336 [running]:
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Crawl).Capture.func1(0xc013c1c000)
        /X/Zeno/internal/pkg/crawl/capture.go:233 +0x50
panic({0x13699e0?, 0x169a9f0?})
        /var/www/.go/src/runtime/panic.go:770 +0x132
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Crawl).Capture(0xc000492008, 0xc013c1c000)
        /X/Zeno/internal/pkg/crawl/capture.go:364 +0x12a7
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Worker).unsafeCapture(0xc00029e500, 0xc013c1c000)
        /X/Zeno/internal/pkg/crawl/worker.go:147 +0xa5
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Worker).Run(0xc00029e500)
        /X/Zeno/internal/pkg/crawl/worker.go:129 +0x7d1
created by github.com/internetarchive/Zeno/internal/pkg/crawl.(*WorkerPool).Start in goroutine 1
        /X/Zeno/internal/pkg/crawl/worker_pool.go:35 +0x119
panic: send on closed channel
        panic: send on closed channel

goroutine 346 [running]:
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Crawl).Capture.func1(0xc01cf08060)
        /X/Zeno/internal/pkg/crawl/capture.go:233 +0x50
panic({0x13699e0?, 0x169a9f0?})
        /var/www/.go/src/runtime/panic.go:770 +0x132
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Crawl).Capture(0xc000492008, 0xc01cf08060)
        /X/Zeno/internal/pkg/crawl/capture.go:364 +0x12a7
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Worker).unsafeCapture(0xc0012322c0, 0xc01cf08060)
        /X/Zeno/internal/pkg/crawl/worker.go:147 +0xa5
github.com/internetarchive/Zeno/internal/pkg/crawl.(*Worker).Run(0xc0012322c0)
        /X/Zeno/internal/pkg/crawl/worker.go:129 +0x7d1
created by github.com/internetarchive/Zeno/internal/pkg/crawl.(*WorkerPool).Start in goroutine 1
        /X/Zeno/internal/pkg/crawl/worker_pool.go:35 +0x119
equals215 commented 1 month ago

Which commit?

CorentinB commented 1 month ago

Which commit?

2ac2af599f020af548d67edd9e4d16c03fc41d72