Wappalyzer code not used

[olearycrew]

Even though this PR appears to add wappalyzer support to cdncheck, I'm finding that not to be true.

Example

For example - store-ca.figma.com

This site uses Cloudflare as it's CDN but is NOT hosted on a Cloudflare IP. This appears to be because Shopify (the host) must be using the Cloudflare BYOIP (bring your own IP) program.

cdncheck -i store-ca.figma.com Returns "No results found"

The IP address (23.227.38.74) is owned by Shopify themselves

However nuclei -u https://store-ca.figma.com -id tech-detect

Returns cloudflare as the HTTP header shows it is clearly being served via Cloudflare CDN

Code

In looking at the code (other.go) there is a function CheckWappalyzer .. but that code isn't called anywhere except in other_test.go.

https://github.com/projectdiscovery/cdncheck/blob/bd3899787a68f5951e9b2ed827b52009bea8891c/other.go#L47-L61

This means when running naabu against this same target, the -exclude-cdn flag doesn't correctly identify this target as served by a CDN

[brenocss]

Exactly, when I implemented this function, the idea was to add an additional way of performing detection, mainly when cdncheck is used as a library.

I'm not sure if it's worth adding this detection mode for cdncheck, naabu, or dnsx, as it would involve making additional HTTPS requests to detect the technologies.

Moreover, the technology used by Cloudflare/Akamai may be detected as a JavaScript CDN but not as a WAF protection.

[brenocss]

Adding this can introduce some false positives.

[olearycrew]

Thanks for the context!

[olearycrew]

Closing this in favor of having httpx use this functionality instead

[brenocss]

@olearycrew i think this discussion is important, maybe you can convert this to a discussion before closing

[Mzack9999]

@brenocss As you pointed out, HTTP analysis would imply additional http requests, precisely what httpx already does with full DOM inspection and many other optimizations to handle http edge cases that fail with the standard library. On the other hand, naabu performs mainly a static membership analysis of the IP address ranges. Keep an eye on https://github.com/projectdiscovery/naabu/pull/670, which will introduce service discovery (-sV flag in nmap). Static cdn analysis on the retrieved response of http discovery would be a great addition. What do you think?

[brenocss]

As you pointed out, HTTP analysis would imply additional http requests, precisely what httpx already does with full DOM inspection and many other optimizations to handle http edge cases that fail with the standard library

For sure, if we use this with -td for httpx without adding any more http requests would be great

Adding this can introduce some false positives.

This is what I think can be problematic, because wappalyzer detection is not only about using cloudflare or akamai WAF, but also their technologies, which could be javascript CDN for example.

Add that to tech detection it's okay, but to determine if we can scan more ports or more nuclei scanning might miss a lot of results

[brenocss]

as you can see

https://store-ca.figma.com -> tech detect cloudlfare

https://store-ca.figma.com/?/etc/passwd -> 200

https://developers.cloudflare.com/?/etc/passwd -> 403

[zy9ard3]

@olearycrew @Mzack9999

Thanks for creating this issue and much needed to do something for more efficient results and avoiding false positive/unnecessary full port scans on CDNs

But unable to understand why the issue has been closed

I'm not sure whether it's possible or not, but I believe this can be fixed in way by enhancing with service discovery/service grabbing since these CDN proxied IPs always return the respective CDN on the results, so initiating service discovery on default ports 80/443 like something while scanning ports can help on this

@brenocss

The issue is not anything about tech-detection

The issue is about unneccessary full port scans for CDN hosts where naabu is unable to detect the hosts proxying through CDNs via BYOIP

Thanks & Regards, @zy9ard3

[brenocss]

The problem is how to detect if a website is using BYOIP without false positives.

Using wappalyzer to detect can result in these false positives. Ignoring sites that aren't cloudflare but use javascript cdn for example. This can lead to missing important scans.

I think we can think of something using passive port detection (shodan/discover) because cloudflare has some specific ports open. But this can also result in false positives, do you have something in mind to detect it without false positives?

$ echo store-ca.figma.com | naabu -passive
store-ca.figma.com:8880
store-ca.figma.com:80
store-ca.figma.com:443
store-ca.figma.com:2086
store-ca.figma.com:8080
store-ca.figma.com:8443
store-ca.figma.com:2053
store-ca.figma.com:2082
store-ca.figma.com:2083
store-ca.figma.com:2087
$ echo cloudflare.com | naabu -passive

                  __
  ___  ___  ___ _/ /  __ __
 / _ \/ _ \/ _ \/ _ \/ // /
/_//_/\_,_/\_,_/_.__/\_,_/ v2.0.7

                projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
[INF] Running PASSIVE scan with non root privileges
[INF] Found 10 ports on host cloudflare.com (104.16.132.229)
cloudflare.com:8880
cloudflare.com:80
cloudflare.com:443
cloudflare.com:2052
cloudflare.com:8080
cloudflare.com:8443
cloudflare.com:2082
cloudflare.com:2083
cloudflare.com:2086
cloudflare.com:2087

[zy9ard3]

@brenocss

The problem is how to detect if a site is using BYOIP without false positives

No need to use wappalyzer at anywhere

This can be easily detected without false positives by initiating service discovery on default ports 80/443

No matter whether it is BYOIP or not, CDN hosts will always returns their respective CDN on results Cloudflare/AkamaiGHost/Varnish/etc..., we can use this to grab CDN hosts regardless of BYOIP/CDN-CIDR

like nmap -sV store-ca.figma.com and you can see the results for better understanding

we can think of something using passive port detection (shodan/discover)

using shodan will results only if the host has been scanned before by shodan

[brenocss]

mmm, I think I get it, are you saying we can detect it by checking some response HTTP headers ? like:

Server: cloudflare
CF-RAY: 

We could add this check, i think would be great.

[zy9ard3]

@brenocss

No not anything related to HTTP headers or probing

I'm saying about using service grabbing like ;

• Cloudflare

nmap -sV store-ca.figma.com      
Starting Nmap 7.91 ( https://nmap.org ) at 2023-05-31 23:07 IST
Nmap scan report for store-ca.figma.com (23.227.38.74)
Host is up (0.037s latency).
rDNS record for 23.227.38.74: shops.myshopify.com
Not shown: 996 filtered ports
PORT     STATE SERVICE  VERSION
80/tcp   open  http     Cloudflare http proxy
443/tcp  open  ssl/http Cloudflare http proxy
8080/tcp open  http     Cloudflare http proxy
8443/tcp open  ssl/http Cloudflare http proxy

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 19.40 seconds

• Akamai

nmap -sV www.carrentals.com      
Starting Nmap 7.91 ( https://nmap.org ) at 2023-05-31 23:27 IST
Nmap scan report for www.carrentals.com (104.115.70.245)
Host is up (0.016s latency).
rDNS record for 104.115.70.245: a104-115-70-245.deploy.static.akamaitechnologies.com
Not shown: 998 filtered ports
PORT    STATE SERVICE  VERSION
80/tcp  open  http     AkamaiGHost (Akamai's HTTP Acceleration/Mirror service)
443/tcp open  ssl/http AkamaiGHost (Akamai's HTTP Acceleration/Mirror service)

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 20.79 seconds

will grab the respective CDNs and give the exact thing what we want

[brenocss]

@Mzack9999 so -sV should work on this, I think we need to run this in 80 and 443 before full port scan in naabu and skip waf's/cdn's.