Update Inconsistent source implementations

[tarunKoyalwar]

Description

Some Sources are not implemented optimally this includes

• time.sleep in some sources
• Improper implementations of token manager
• Proper Pagination

Code Refactoring

• Sources should be abstracted as much as possible using struct embedding
• All Sources should use tokens without exhausting them and use within their constraints (rate limits)
• Respect and use Rate-Limits

Subfinder version latest | dev

### Tasks
- [ ] https://github.com/projectdiscovery/subfinder/issues/1174
- [ ] https://github.com/projectdiscovery/subfinder/issues/1176

[deadjdona]

Good day

Does subfinder works at all?

└─$ subfinder -d celsys.ai -v

        _     __ _         _
____  _| |__ / _(_)_ _  __| |___ _ _
(_-< || | '_ \  _| | ' \/ _  / -_) '_|
/__/\_,_|_.__/_| |_|_||_\__,_\___|_| v2

                projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[WRN] By using subfinder, you also agree to the terms of the APIs used.

[INF] Enumerating subdomains for celsys.ai
[WRN] Could not run source threatminer: Unexpected status code 500 received from https://api.threatminer.org/v2/domain.php?q=celsys.ai&rt=5
[WRN] Could not run source bufferover: Get "https://dns.bufferover.run/dns?q=.celsys.ai": dial tcp: lookup dns.bufferover.run on 172.24.144.1:53: no such host
[WRN] Could not run source shodan: Unexpected status code 503 received from https://api.shodan.io/shodan/host/search?query=hostname:celsys.ai&page=0&key=AAAAClP1bJJSRMEYJazgwhJKrggRwKA
[WRN] Could not run source ipv4info: Unexpected status code 404 received from http://ipv4info.com/search/celsys.ai
[WRN] Could not run source urlscan: Unexpected status code: 400
[WRN] Could not run source binaryedge: Unexpected status code 401 received from https://api.binaryedge.io/v2/query/domains/subdomain/celsys.ai
[WRN] Could not run source certspotterold: Unexpected status code 410 received from https://certspotter.com/api/v0/certs?domain=celsys.ai
[WRN] Could not run source github: Unexpected status code 401 received from https://api.github.com/search/code?per_page=100&q="celsys.ai"
[WRN] Could not run source archiveis: Unexpected status code 403 received from http://archive.is/*.celsys.ai
[WRN] Could not run source entrust: Unexpected status code 403 received from https://ctsearch.entrust.com/api/v1/certificates?fields=issuerCN,subjectO,issuerDN,issuerO,subjectDN,signAlg,san,publicKeyType,publicKeySize,validFrom,validTo,sn,ev,logEntries.logName,subjectCNReversed,cert&domain=celsys.ai&includeExpired=true&exactMatch=false&limit=5000
[WRN] Could not run source bufferover: Unexpected status code 403 received from https://tls.bufferover.run/dns?q=.celsys.ai
[WRN] Could not run source threatcrowd: Unexpected status code 503 received from https://www.threatcrowd.org/searchApi/v2/domain/report/?domain=celsys.ai
[WRN] Could not run source sitedossier: Unexpected status code 403 received from http://www.sitedossier.com/parentdomain/celsys.ai
[WRN] Could not run source commoncrawl: Unexpected status code 404 received from https://index.commoncrawl.org/CC-MAIN-2020-50-index?url=*.celsys.ai&output=json
[virustotal] Source took 1.7862ms for enumeration
[dnsdumpster] Source took 1.223838804s for enumeration
[dnsdb] Source took 1.7817ms for enumeration
[alienvault] Source took 272.216601ms for enumeration
[rapiddns] Source took 384.647701ms for enumeration
[bufferover] Source took 725.843802ms for enumeration
[waybackarchive] Source took 2.157906006s for enumeration
[censys] Source took 1.9241ms for enumeration
[ipv4info] Source took 251.889501ms for enumeration
[certspotter] Source took 851µs for enumeration
[certspotterold] Source took 372.803701ms for enumeration
[threatcrowd] Source took 911.953603ms for enumeration
[sitedossier] Source took 5.790483215s for enumeration
[commoncrawl] Source took 17.485675742s for enumeration
[entrust] Source took 683.894702ms for enumeration
[hackertarget] Source took 1.050866403s for enumeration
[sublist3r] Source took 30.002356968s for enumeration
[spyse] Source took 1.7746ms for enumeration
[urlscan] Source took 314.118701ms for enumeration
[github] Source took 406.195301ms for enumeration
[WRN] Could not run source sublist3r: Get "https://api.sublist3r.com/search.php?domain=celsys.ai": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[crtsh] Source took 480.309202ms for enumeration
[archiveis] Source took 568.291302ms for enumeration
[binaryedge] Source took 372.690801ms for enumeration
[intelx] Source took 625.4µs for enumeration
[zoomeye] Source took 714.9µs for enumeration
[passivetotal] Source took 1.6613ms for enumeration
[threatminer] Source took 131.582301ms for enumeration
[shodan] Source took 209.756001ms for enumeration
[securitytrails] Source took 561.8µs for enumeration

0 results