search

projectdiscovery / httpx

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.
https://docs.projectdiscovery.io/tools/httpx
MIT License
7.64k stars 831 forks source link

New feature: The same target is only executed once in an hour #1349

Closed hktalent closed 11 months ago

hktalent commented 1 year ago

New feature: The same target is only executed once in an hour now

cat xxx.txt|httpx -json -location -fr -o chaos-bugbounty-list-httpx.json -p 80,443

Targets may be added here

cat xxx.txt|httpx -json -location -fr -o chaos-bugbounty-list-httpx.json -p 80,443

Running the same command multiple times in this way can achieve better network optimization.

@ehsandeep It is recommended that all tools be limited to a certain time range and not be executed repeatedly, directly return the last result, and set up a cache

nuclei
subfinder
katana
httpx
naabu
proxify
uncover
dnsx
shuffledns
public-bugbounty-programs
tlsx
asnmap
wappalyzergo
hktalent commented 1 year ago

@ehsandeep Regarding caching, it is recommended to use "github.com/blevesearch/bleve/v2". I have proven in project practice that on SSD 4core CPU, 1 billion data queries are very fast.

about blevesearch/bleve example:https://github.com/hktalent/AiCSA

ehsandeep commented 11 months ago

@hktalent, thanks for the feature suggestion, this is out of scope for a specific project, we have an alternate solution for this problem that will apply on all projects, and hopefully, we will release in the future.