automatically get new UserAgents

[MadAim123]

Is your feature request related to a problem? Please describe. The UserAgents will be old after some days. An automatically way to update them could be good

Describe the solution you'd like Crawl the newest 4 needed useragents without manually work

Describe alternatives you've considered Something like this could be used in "utils":

try:
    # JSON URL containing user agents
    json_url = "https://jnrbsn.github.io/user-agents/user-agents.json"

    # Fetch the JSON data from the URL
    response = requests.get(json_url)
    user_agents_json = response.json()

    # Extract the first 4 user agents
    first_4_user_agents = user_agents_json[:4]

    # Save the user agents into the USERAGENTS list
    USERAGENTS = first_4_user_agents

except Exception as e:
    USERAGENTS = [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
    ]

[deedy5]

Hello! Thanks for the suggestion, but there's no point in doing it.

1) Api requests will work with headers of previous browser versions as well. 2) User can specify any headers he wants.

At least for the moment.