search

Security-Tools-Alliance / rengine-ng

reNgine-ng is an automated reconnaissance framework for web applications with a focus on highly configurable streamlined recon process via Engines, recon data correlation and organization, continuous monitoring, backed by a database, and simple yet intuitive User Interface.
GNU General Public License v3.0
19 stars 8 forks source link

bug: subdomain scan not alive stored anyway in db #13

Closed psyray closed 2 months ago

psyray commented 4 months ago

Is there an existing issue for this?

Current Behavior

Submitted by @DanyAssuid

After a subdomain scanning I receive a multiples subdomain that not alive , This subdomain stored anyway in the DB.

Expected Behavior

After a subdomain scanning I receive a multiples subdomain that not alive , This subdomain stored anyway in the DB. I search in the logs of http_crawler (httpx) and receive this error :

{"timestamp":"2024-01-12T00:53:28.420100656Z","url":"http://Meet.hilton.com","input":"Meet.hilton.com","error":"context deadline exceeded (Client.Timeout exceeded while awaiting headers)","words":0,"lines":0,"failed":true}

But the domain anyway stored in the DB Also the subdomain need to be set to lowercase first before comparing and record Capture_decran_2024-01-12_a_12 53 23

I also have another issue when execute two verifications commands with httpx : 

httpx -cl -ct -rt -location -td -websocket -cname -asn -cdn -probe -random-agent -t 1 -json -u hilton.com -x HEAD -silent -fr

{"timestamp":"2024-01-12T12:42:16.833371577+01:00","asn":{"as_number":"AS26529","as_name":"hilton-e","as_country":"US","as_range":["167.187.200.0/22"]},"hash":{"body_md5":"d41d8cd98f00b204e9800998ecf8427e","body_mmh3":"-1840324437","body_sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","body_simhash":"18446744073709551615","header_md5":"9b84c31ab4ff2cefe0991556b20e9295","header_mmh3":"-776338361","header_sha256":"bde37c9ca30d6b12a4343ea950898d567a2d5db89e02c4c5b93c5f5683a0cc29","header_simhash":"15615025676370544106"},"port":"443","url":"https://hilton.com","input":"hilton.com","scheme":"https","webserver":"AkamaiNetStorage","content_type":"text/html","method":"HEAD","host":"167.187.200.23","path":"/","final_url":"https://www.hilton.com/","time":"1.118432901s","chain_status_codes":[301,403],"a":["167.187.200.23"],"words":1,"lines":1,"status_code":403,"content_length":102518,"failed":false,"knowledgebase":{"PageType":"other","pHash":0}}
httpx -cl -ct -rt -location -td -websocket -cname -asn -cdn -probe -random-agent -t 1 -json -u hilton.com -x HEAD -silent -fr

{"timestamp":"2024-01-12T12:45:56.117176517+01:00","url":"http://hilton.com","input":"hilton.com","error":"context deadline exceeded (Client.Timeout exceeded while awaiting headers)","words":0,"lines":0,"failed":true}

it's the same url but one time it's alive another time it's not

Steps To Reproduce

  1. execute subdomain scan on hilton.com
  2. wait to log http_crawler
  3. we can find one of subdomain not alive but inserted in the DB and subdomain results

Environment

- reNgine: 2.0.2
- OS: MAC os
- Python: 
- Docker Engine: 
- Docker Compose: 
- Browser: firefox

Anything else?

No response