search

funilrys / PyFunceble

The tool to check the availability or syntax of domain, IP or URL.
https://pyfunceble.github.io
Apache License 2.0
289 stars 44 forks source link

FEATURE: user_agent: random #368

Open spirillen opened 2 weeks ago

spirillen commented 2 weeks ago

Description

As I'm working my way through the new PyFunceble.yaml I stumbled on the fact we hare hard coding the use of browser and OS, Somehow this just seems wrong, as we know, then the big5 and there wannabees are logging the IP/Brpwser/OS etc and sharing to fetch bots and gives us FP-responses on http codes.

Possible Solution

One of two

  1. Randomize browser and OS per -n records (Optimal)
  2. Randomize browser and OS per test starts

Considered Alternative

Manual labor, bad, the system is MY slave, not the other way around.

Additional context

funilrys commented 1 week ago

Yeah, but what does random means for you?

Our user_agents are getting rotated every week - when not every day... And we choose a random one from one of the latest / most used ones:

https://github.com/funilrys/PyFunceble/blob/fd2ce92208336300df07b00ce3995956b81ad62e/PyFunceble/dataset/user_agent.py#L225-L226

spirillen commented 1 week ago

And we choose a random one from one of the latest / most used ones:

I see tree questions here

  1. Selection among browser + OS

  2. How to control

  3. You(me) ask because

  4. Automatically rotate among all the available ones from the list (forgot the url), but you probably remember it. to the list of options

  5. by a switch of Random vs chrome,Linux in the pyf.owerwrite.yml

  6. Because as of now, 1 I didn't know it was rotated by default, 2. in pyf.owerwrite.yml. you can only set a form value

spirillen commented 1 week ago

Come to think of... and YES it is a though only, can you spoof the mac addresses used for querying?

IF == True

Would it be an idea then to periodically with hourly randomness generate a set of "Clients, running its own MAC + Browser + OS"? ... IF == TRUE Could this help making it looks like a random number of users on the same network vs one client from the same network?