search

jaeles-project / gospider

Gospider - Fast web spider written in Go
MIT License
2.53k stars 306 forks source link

Empty output specifying HTTP(S) port #31

Closed electroxero closed 3 years ago

electroxero commented 3 years ago

Description

Web I try to run gospider on an URL specifying also the HTTP port, sometimes I don't know why exactly it doesn't crawl the target.

Go version

go version go1.16.2 linux/amd64

Gospider Version

1.1.5 (In the last commit of https://github.com/jaeles-project/gospider/blob/2e610b3fd79e1ac0945b694385edd88028f821ce/core/version.go the version is wrong btw)

Test case 1 - Not specifying http or https port

./gospider -q -s https://shippingmanager.bpost.be/ --debug

[0000]  INFO Start crawling: https://shippingmanager.bpost.be/
[0000]  INFO Found robots.txt: https://shippingmanager.bpost.be//robots.txt
https://shippingmanager.bpost.be/ShmFrontEnd/
[0000]  INFO Done.

Test case 2 - Specifying the port:

./gospider -q -s https://shippingmanager.bpost.be:443/ --debug

image

electroxero commented 3 years ago

Closing the issue because the problem is the web response which redirects (With location header) to a wrong URI: http to 443 (https) port