Random User-Agent - Githubissues

philippta / flyscrape

Flyscrape is a command-line web scraping tool designed for those without advanced programming skills.

https://flyscrape.com

Mozilla Public License 2.0

1.02k stars 29 forks source link

Random User-Agent #35

Open philippta opened 7 months ago

philippta commented 7 months ago

The headers module should get a feature that allows for randomized user-agents per request.

The feature should be enabled when the following configuration is provided.

headers: {
    "User-Agent": "random",
}

Ref:

https://github.com/gocolly/colly/blob/master/extensions/random_user_agent.go

rafiramadhana commented 7 months ago

can I work on this enhancement @philippta ? thanks

philippta commented 7 months ago

@rafiramadhana Please go ahead. Two things:

The list doesn’t seem to have been updated since 2023/05. Make sure to check for new versions and add them to the list.
I think just the desktop versions of Chrome, Firefox, Edge and Opera will be enough. Bonus points will be given if you can add Safari. Ignore the pixel/mobile devices.

I am also thinking if it would be beneficial to enable random user agents by default, and only make it static if configured in the config. Any opinion?

philippta commented 6 months ago

Hey @rafiramadhana, any updates on this?

rafiramadhana commented 6 months ago

@philippta Sry I haven't put any progress on this.

Feel free to assign this to someone else if needed. Thanks.