baddns speed optimization

[liquidsec]

Baddns is occasionally experiencing performance issues which are likely linked to the references module. The references module was disabled within bbot, instead favoring bbot's innate ability to extract and process links. This required a few exceptions to some rules, and a custom filter event. We also now ingest URL events, in very specific circumstances.

[liquidsec]

for reference: https://github.com/blacklanternsecurity/bbot/issues/1470

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 96.42857% with 1 line in your changes missing coverage. Please review.

Project coverage is 92%. Comparing base (ba75bdb) to head (2477945).

Files	Patch %	Lines
bbot/modules/baddns.py	89%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## dev #1494 +/- ## ===================================== - Coverage 92% 92% -0% ===================================== Files 327 327 Lines 21105 21130 +25 ===================================== + Hits 19370 19385 +15 - Misses 1735 1745 +10 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[liquidsec]

There are some additional problems here to consider.

1. .css is currently in the blacklist. So we'd need to remove that to allow .css based reference takeovers
2. Loss of "direct" detections. Baddns has a special feature to to detect "direct" takeovers, just within the reference model. A significant amount of additional logic would need to be placed in the baddns bbot module to replicate this. We could change everything over to "direct mode", but that would cause a significant amount of false positives, because a takeover-able domain means nothing without the in-scope CNAME in your target being connected to it.
3. Loss of context. A detection of a reference-based takeover provides a lot more context when the reference module finds it, as opposed to finding it and relying on tracing back through events manually with bbot data. For example:

As opposed to:

We don't even know how it pertains to our target without further investigation in the second example.

4. We maybe adding back in delays by opening up distance-1 js/css anyway. This could be a significant slowdown, potentially EVEN WORSE than references module, because we'd be checking all distance-1 js, not just the ones guaranteed to be related to our target in some way.

With that in mind, we should at least consider abandoning this path in favor of some other measure for speeding things up.

[liquidsec]

closing in favor of https://github.com/blacklanternsecurity/bbot/pull/1502