bug: subdomain scan not alive stored anyway in db

[DanyAssuid]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

After a subdomain scanning I receive a multiples subdomain that not alive , This subdomain stored anyway in the DB.

Expected Behavior

After a subdomain scanning I receive a multiples subdomain that not alive , This subdomain stored anyway in the DB. I search in the logs of http_crawler (httpx) and receive this error :

{"timestamp":"2024-01-12T00:53:28.420100656Z","url":"http://Meet.hilton.com","input":"Meet.hilton.com","error":"context deadline exceeded (Client.Timeout exceeded while awaiting headers)","words":0,"lines":0,"failed":true}

But the domain anyway stored in the DB Also the subdomain need to be set to lowercase first before comparing and record

I also have another issue when execute two verifications commands with httpx :

httpx -cl -ct -rt -location -td -websocket -cname -asn -cdn -probe -random-agent -t 1 -json -u hilton.com -x HEAD -silent -fr

{"timestamp":"2024-01-12T12:42:16.833371577+01:00","asn":{"as_number":"AS26529","as_name":"hilton-e","as_country":"US","as_range":["167.187.200.0/22"]},"hash":{"body_md5":"d41d8cd98f00b204e9800998ecf8427e","body_mmh3":"-1840324437","body_sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","body_simhash":"18446744073709551615","header_md5":"9b84c31ab4ff2cefe0991556b20e9295","header_mmh3":"-776338361","header_sha256":"bde37c9ca30d6b12a4343ea950898d567a2d5db89e02c4c5b93c5f5683a0cc29","header_simhash":"15615025676370544106"},"port":"443","url":"https://hilton.com","input":"hilton.com","scheme":"https","webserver":"AkamaiNetStorage","content_type":"text/html","method":"HEAD","host":"167.187.200.23","path":"/","final_url":"https://www.hilton.com/","time":"1.118432901s","chain_status_codes":[301,403],"a":["167.187.200.23"],"words":1,"lines":1,"status_code":403,"content_length":102518,"failed":false,"knowledgebase":{"PageType":"other","pHash":0}}

httpx -cl -ct -rt -location -td -websocket -cname -asn -cdn -probe -random-agent -t 1 -json -u hilton.com -x HEAD -silent -fr

{"timestamp":"2024-01-12T12:45:56.117176517+01:00","url":"http://hilton.com","input":"hilton.com","error":"context deadline exceeded (Client.Timeout exceeded while awaiting headers)","words":0,"lines":0,"failed":true}

it's the same url but one time it's alive another time it's not

Steps To Reproduce

1. execute subdomain scan on hilton.com
2. wait to log http_crawler
3. we can find one of subdomain not alive but inserted in the DB and subdomain results

Environment

- reNgine: 2.0.2
- OS: MAC os
- Python: 
- Docker Engine: 
- Docker Compose: 
- Browser: firefox

Anything else?

No response

[github-actions[bot]]

👋 Hi @DanyAssuid, Issues is only for reporting a bug/feature request. Please read documentation before raising an issue https://rengine.wiki For very limited support, questions, and discussions, please join reNgine Discord channel: https://discord.gg/azv6fzhNCE Please include all the requested and relevant information when opening a bug report. Improper reports will be closed without any response.

[yogeshojha]

As you said one time the subdomain was alive and another wasn't, it has nothing to do with reNgine. If it detects as not alive then in UI what you have shown is expected.