Refactor search engine stuff - Githubissues

zerohour-phishing-detection / zpd-server

Code and test data for anti-phishing tool: A decision-support tool for experimentation on zero-hour phishing detection

Creative Commons Attribution 4.0 International

2 stars 0 forks source link

Refactor search engine stuff #11

Closed TPGamesNL closed 6 months ago

TPGamesNL commented 7 months ago

progress for this is going to be in the same branch as #7, so fix-google-search (see PR #9)

TPGamesNL commented 7 months ago

some higher level goals for the cleanup

[ ] separate clearbit API stuff from google.py
[ ] dont put reverse image search and text search stuff together, should be separate
[ ] test and update non-google engine impls
[ ] documentation
[ ] reduce imports + dependencies where possible
[ ] make impls adhere to superclass method signatures

TPGamesNL commented 7 months ago

tbh it looks like most of the bing engine was just copy pasted code from google, i highly doubt it actually works.

same CSS classes for extracting search result URLs
same code for detecting rate limits (based on specific warning message from the engine)
uses www.google., https://www.google.com/imgres?, translate.google. and webcache.googleusercontent. to verify URLs (I doubt bing uses those names)

TPGamesNL commented 7 months ago

Can confirm by comparing source is that the bing source is almost entirely copied from google

TPGamesNL commented 7 months ago

Yandex text search no work: it has better robot detection, after changing from HTTP to HTTPS requests (without change it defo doesnt work) every request simply gets a response with a captcha (Please confirm that you and not a robot are sending requests)