berstend / puppeteer-extra

💯 Teach puppeteer new tricks through plugins.
https://extra.community
MIT License
6.54k stars 744 forks source link

[Idea] List of detection tests in docs #254

Open prescience-data opened 4 years ago

prescience-data commented 4 years ago

Not sure if there's any unforeseen downsides to this idea, but it would be great if there was a way for users to submit new detection tests they find so that developers can improve their Puppeteer apps by running tests against real products.

I've been working on a Plugin that runs the tests that I've identified so far but it would be good to get a more complete list of detection examples from the community.

The ones I have so far:

Any others people know of would be awesome!

Submitted

Edit 1:

Example of how to test against them

Here is a really basic demo of how you might test each detection:

https://github.com/prescience-data/puppeteer-botcheck

tenkuken commented 4 years ago

F5 Network Bot Defense https://ib.bri.co.id/ib-bri/

evading-bot-detection commented 4 years ago

This is useful. Can someone also add the tests for WhiteOps?

https://smitop.com/post/whiteops-data/

berstend commented 4 years ago

I like the idea but most of these sites/links are not bot detection tests? :-)

prescience-data commented 4 years ago

Some of the products such as Datadome have no public "demos" that I know of, so the idea is to build up your tests like this:

https://github.com/prescience-data/puppeteer-botcheck

chris124567 commented 4 years ago

PerimeterX: https://www.usa-people-search.com/names/a_1_150_0

@berstend These sites (well, most of them) are clients of bot detection companies. If you visit them with, say, window.callPhantom exposed, you will get redirected to a page that will make you do a captcha.

drzraf commented 3 years ago

The first link one does not show Distill Networks operating anymore. But a good example is given by WesternUnion Getting a non-empty change rate in #smoExchangeRate means Distill Networks is bypassed (which currently not the case with latest stealth in headless mode)

berstend commented 3 years ago

But a good example is given by WesternUnion

I'd say they're more on the extreme end of the gaussian curve of anti-bot 😄 image

francobasilico commented 1 year ago

@drzraf Can you nowadays return that #smoExchangeRate?