Update Browser Engine Data - Githubissues

unblocked-web / double-agent

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

MIT License

135 stars 10 forks source link

Update Browser Engine Data #51

Closed blakebyrnes closed 2 years ago

blakebyrnes commented 2 years ago

This PR has changes to support updated browser data for chrome 90-98. Of note:

Operating System versions is no longer reliable in UserAgent strings after chrome 90. For this reason, we needed to change how we track assignments to store the userAgentId upfront, instead of parsing out of the userAgentString.
ResourceType Xhr was case-changed to XHR, per a corresponding change from Hero
SEC-CH-UA headers needed to have some handling
Add hero assignment runner

NOTE: there remains a bug (will log) in the way we're analyzing headers that can change orders (basic headers and xhr both have this issue)