Broken Domains Used by Detectors

[rgmz]

Description

Since @lc is doing a great job validating existing detectors, I figured it would be useful to do a high-level check of any domains that no longer resolve or are expired.

I will continue to update this list with findings.

Broken

This could indicate that the specific endpoint is broken and needs attention (e.g., Gitter) or that the domain itself is no longer registered/resolves.

• [ ] https://api.apiscience.com https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/apiscience/apiscience.go#L51
• [x] https://api.base-api.io (fixed in #1992)
• [x] https://api.datafire.io (fixed in #1995) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/datafire/datafire.go#L51
• [ ] https://api.gitter.im/ https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/gitter/gitter.go#L51
• [x] https://api.happi.dev/ (#2003) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/happi/happi.go#L50
• [x] https://api.idbus.com/ (fixed in #1996)
• [ ] https://api.lexigram.io https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/lexigram/lexigram.go#L51
• [x] https://api.flowdock.com (#2004) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/flowdock/flowdock.go#L50
• [ ] https://api.macaddress.io https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/macaddress/macaddress.go#L50
• [ ] https://api.meta-api.io/ https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/metaapi/metaapi.go#L60
• [ ] https://api.onwater.io/ (domain redirects to https://isitwater.com/; API url is apparently now https://isitwater-com.p.rapidapi.com) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/onwaterio/onwaterio.go#L51
• [ ] https://api.opengraphr.com/ (domain resolves to a blank Forge/laravel site) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/opengraphr/opengraphr.go#L50
• [ ] https://api.passbase.com/ (domain redirects to https://parallelmarkets.com/; https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/passbase/passbase.go#L50
• [ ] https://api.sentimentinvestor.com/ (subdomain no longer resolves but the domain does; it may be spam now, hard to tell)
• [ ] https://api.sherpadesk.com (redirects to https://localhost/metadata despite https://www.sherpadesk.com/ being alive??) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/sherpadesk/sherpadesk.go#L54
• [ ] https://api.unify.id/ (the base domain redirects to https://www.prove.com) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/unifyid/unifyid.go#L52
• [ ] https://app.fakejson.com/ (domain still exists/resolves? They may have retired the site) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/fakejson/fakejson.go#L53
• [ ] https://app.lendflow.io/ (domain still exists. API url may have changed) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/lendflow/lendflow.go#L54
• [x] https://qckm.io/ (no longer resolves because QuickMetrics has shut down; fixed in #1997) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/quickmetrics/quickmetrics.go#L51
• [ ] https://sandbox.impala.travel (subdomain no longer resolves, though domain still exists) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/impala/impala.go#L50
• [ ] https://scrapersite.com/ (no longer resolves) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/scrapersite/scrapersite.go#L53
• [x] https://secretscanner.ladesk.com/ (subdomain no longer resolves; fixed in #2001) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/liveagent/liveagent.go#L50
• [ ] https://www.glitterlyapi.com/ (no longer resolves) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/glitterlyapi/glitterlyapi.go#L50
• [ ] https://scrapersite.com/ (doesn't resolve) https://github.com/trufflesecurity/trufflehog/blob/6c35dcffa583b1ac86b319d3d0f4dd00758948f1/pkg/detectors/scrapersite/scrapersite.go#L53

Parked

• [ ] TBD

Other

• [ ] https://api.scraperbox.com/ (ssl cert is expired)
• [ ] https://getsandbox.com/ (SSL_ERROR_RX_RECORD_TOO_LONG - could just be me
• [ ] https://mrticktock.com/ (invalid certificate domain)
• [ ] https://api.currencybeacon.com/ (redirects to https://api.currencybeacon.com/)
• [ ] https://api.prospect.io (redirects to https://overloop.com)
• [ ] https://api.statuspage.io (redirects to https://www.atlassian.com/software/statuspage)
• [ ] https://crossbrowsertesting.com/ redirects to https://smartbear.com/product/bitbar/

[fumblehool]

@rgmz What if we add a function in Scanner{} to verify if host exists? I mean, something like s.isValidDetector() bool. Inside this function, we'll ping the api url to make sure domain exists.

This way, we can add a Github Action check to verify if any detector needs to be removed. WDYT?

[rgmz]

This could be automated to an extent: subdomains that no longer resolve or domains that have expired are easy to check, behavioral changes are a bit harder.

I am curious whether the team has an existing process to run all the TestX_FromChunk tests with live secrets and review problematic results.

[ahrav]

We have established a daily routine where an automated test suite is executed to assess the performance of our detectors, identifying any failures. However, a segment of these failures is attributed to the expiration of test tokens, which were configured during trial phases. Our current focus is on devising strategies to segregate genuine test failures from those arising due to expired credentials. Furthermore, we are in the preliminary stages of broadening our metrics around detection to garner more insights into the issue at hand.

As our detector arsenal expands, acquiring a deeper understanding of each detector's performance and validity becomes paramount. The experiences from this month alone have highlighted a noticeable count of detectors falling into obsolescence, reinforcing the necessity of this endeavor. 😅

This approach to automation, especially concerning the verification of subdomains and domain expirations, should hopefully aid in filtering out trivial issues, allowing us to prioritize addressing more complex behavioral changes.