search

rverton / webanalyze

Port of Wappalyzer (uncovers technologies used on websites) to automate mass scanning.
MIT License
955 stars 137 forks source link

panic: Negative WaitGroup counter #25

Closed Vishnugadupudi closed 4 years ago

Vishnugadupudi commented 4 years ago

deepakgadupudi@cloudshell:~/Vishnu/Stack$ cat hosts | wc 55195 55195 1585167 deepakgadupudi@cloudshell:~/Vishnu/Stack$ webanalyze -hosts hosts -output csv > results.csv 2020/02/06 02:38:41 Scanning with 4 workers. 2020/02/06 02:38:46 [-] Error for http://0addomain.com: Failed to retrieve panic: sync: negative WaitGroup counter

goroutine 24 [running]: sync.(WaitGroup).Add(0xc0000962f0, 0xffffffffffffffff) /usr/local/go/src/sync/waitgroup.go:74 +0x139 sync.(WaitGroup).Done(...) /usr/local/go/src/sync/waitgroup.go:99 github.com/rverton/webanalyze.process(0xc00252ccc0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/deepakgadupudi/gopath/src/github.com/rverton/webanalyze/webanalyze.go:251 +0x1a23 github.com/rverton/webanalyze.worker(0xc000070120, 0xc0000700c0, 0xc0000962e0) /home/deepakgadupudi/gopath/src/github.com/rverton/webanalyze/webanalyze.go:137 +0x73 created by github.com/rverton/webanalyze.initWorker /home/deepakgadupudi/gopath/src/github.com/rverton/webanalyze/webanalyze.go:122 +0x77

Vishnugadupudi commented 4 years ago

Hi @rverton what is the issue here??

rverton commented 4 years ago

Hi, I will try to reproduce this and then come back to you.

Vishnugadupudi commented 4 years ago

@rverton Thanks (:

rverton commented 4 years ago

I'm able to reproduce this, will let you know when I've fixed this.

$ ./webanalyze -hosts <(echo "doestnotexist.io") -output csv
2020/02/07 09:47:38 Scanning with 4 workers.
panic: sync: negative WaitGroup counter

goroutine 13 [running]:
sync.(*WaitGroup).Add(0xc00001c380, 0xffffffffffffffff)
    /usr/local/go/src/sync/waitgroup.go:74 +0x139
sync.(*WaitGroup).Done(...)
    /usr/local/go/src/sync/waitgroup.go:99
github.com/rverton/webanalyze.process(0xc000794660, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/robin/go/src/github.com/rverton/webanalyze/webanalyze.go:251 +0x1a23
github.com/rverton/webanalyze.worker(0xc000086120, 0xc0000860c0, 0xc00001c370)
    /Users/robin/go/src/github.com/rverton/webanalyze/webanalyze.go:137 +0x73
created by github.com/rverton/webanalyze.initWorker
    /Users/robin/go/src/github.com/rverton/webanalyze/webanalyze.go:122 +0x77
Vishnugadupudi commented 4 years ago

Knock Knock.

rverton commented 4 years ago

I plan to make a bigger refactor here and pulling all goroutine stuff out of the lib so the callee has to do it. This will take some time.

As a quick fix, you should be fine putting a valid URL at the beginning of your file.

theblackturtle commented 4 years ago

I have this too.

panic: sync: negative WaitGroup counter

goroutine 7 [running]:
sync.(*WaitGroup).Add(0xc0000942f0, 0xffffffffffffffff)
        /usr/lib/go-1.11/src/sync/waitgroup.go:74 +0x139
sync.(*WaitGroup).Done(...)
        /usr/lib/go-1.11/src/sync/waitgroup.go:99
github.com/rverton/webanalyze.process(0xc000baa480, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/go/src/github.com/rverton/webanalyze/webanalyze.go:251 +0x1a23
github.com/rverton/webanalyze.worker(0xc00006c180, 0xc00006c120, 0xc0000942e0)
        /root/go/src/github.com/rverton/webanalyze/webanalyze.go:137 +0x73
created by github.com/rverton/webanalyze.initWorker
        /root/go/src/github.com/rverton/webanalyze/webanalyze.go:122 +0x77
rverton commented 4 years ago

I just pushed a new version af2f9de where I refactored the concurrency mechanism. This will change some things for all people which use webanalyze as a library (because they have to do the concurrency on their own now). For all others, there should be no change except the better formatting when printing to stdout.

Please let me know if there are any problems with this version. I think I will tag this as a new release in the next days.

Vishnugadupudi commented 4 years ago

Wow 🔥️, Thanks @rverton