Try scraping 443 and 80, even if ports are not open

cisagov / crossfeed

External monitoring for organization assets

https://docs.crossfeed.cyber.dhs.gov

Creative Commons Zero v1.0 Universal

369 stars 54 forks source link

Try scraping 443 and 80, even if ports are not open #442

Open epicfaace opened 4 years ago

epicfaace commented 4 years ago

Right now, we have to wait for a censys scan / port scan to verify that ports 443 / 80 are open, before running wappalyzer / intrigue / web scraping. We shouldn't need to have to depend on this -- we should just make requests to ports 80 and 443 to see if they are open.

epicfaace commented 3 years ago

https://github.com/cisagov/crossfeed/blob/master/backend/src/tasks/wappalyzer.ts

https://github.com/cisagov/crossfeed/blob/18491529a42ba8bcaf33f21287bbfbcaeff25281/backend/src/tasks/helpers/getLiveWebsites.ts#L10