Change user agent of Chrome and send custom CDP commands - see alternative

[Sam621]

Hi Ken, I want to change the agent of chrome and few commands with RPA-Python. Otherwise the site doesn't allow to load as detect as bots. With Python and selenium, I am running below commands and site is loading perfectly. Is there any way to run achieve this RPA-Python.

driver = webdriver.Chrome(options=options)
driver.execute_cdp_cmd(
            "Page.addScriptToEvaluateOnNewDocument",
            {
                "source": """
        Object.defineProperty(window, 'navigator', {
            value: new Proxy(navigator, {
              has: (target, key) => (key === 'webdriver' ? false : key in target),
              get: (target, key) =>
                key === 'webdriver'
                  ? undefined
                  : typeof target[key] === 'function'
                  ? target[key].bind(target)
                  : target[key]
            })
        })
                  """
            },
        )
original_user_agent_string = driver.execute_script(
            "return navigator.userAgent"
        )

Thanks in advance. Rgds, Sam

[kensoh]

Hi RPA for Python users, 🅗🅐🅟🅟🅨 ②⓪②①!

It has been many months since I last worked on this personal side project. Just want to shout out that this project is very much alive, and I hope to look into the 30+ issues raised since July (and bug-fixes if required) soon when I get a moment again.

Below are details if you are interested, why I'm away past few months and why I believe my darkest days are over 🍀🤞🏻🙏🏻

---

Since May last year, I've stopped all work to juggle with my mum's critical illness, taking care of my toddler, and doing housework chores like cleaning and cooking. TBH, my wife and I felt that taking care of a baby and giving her the best nutrition (nursing and home-cooked organic food) and care requires at least 1.5 full-time headcount. There is no life to speak of, other than trying to stay afloat with baby care needs, and keep finding alternative ways to treat my mum, where there're some wins.

Eg importing soy-based tube feeds from Netherlands / Germany / Australia --> after switching away from milk-based tube feeds for 2 weeks she stops requiring dialysis. She started requiring 3x a week dialysis for more than 3 months, after admitting to the hospital. Also, applying herbs like mashed garlic or diced onions to her feet somehow consistently improve her breathing and lung function. Have to keep trying alternative healing methods when her recovery keeps hitting plateau in hospital.

She has been discharged from the hospital a month ago and recovering at home, though very slowly. A longterm medication seems to be one of the contributing causes for her condition. After stopping it, there seems to be improvement to her breathing capacity. She is still bed-bound, but over the course of the year I hope she'll regain strength to move about, and hopefully lungs to recover sufficiently to not require oxygen tank support through a tracheal mask (a hole through her throat).

---

Since November, I've gone back to work at AI Singapore, a government-funded programme to accelerate AI in Singapore. I'm with Product Engineering team, focusing on RPA and TagUI (open-source RPA tool), which RPA for Python is a wrapper for. There are pretty interesting stuffs on TagUI 2021 roadmap. 2020 was the year which my family was going to move to the UK and work, after I receive its tier 1 exceptional tech talent visa. Then everything changed with the virus outbreak, and my mum's illness. The best thing that happened to me in 2020 was being able to go back to work at AI Singapore - I'm grateful.

I've just hired someone to help with housework, cooking and supporting my wife with baby care. I hope this new arrangement will allow me to deliver on my work at AI Singapore, and soon be able to reply countless friends' messages which circumstances leave me with no choice but go missing for months, and then look into the issues raised here!

Eg of the cost of best-in-class nutrition standard researched and set by wife. Baby just felt restless and broke her plate. Even Corelle's break-resistant plate breaks. Why not use plastic plate? Cuz plastic can leech toxins to the food on it. Why not use Silicon plate? Cuz there is no white silicon materials for plate. Why white? Cuz that helps to improve visibility of the food instead of contrasting against some pink colour or some other colours. I would be spending much time clearing up the mess of 70 over pieces of plate fragments, while baby gets nursed to calm down, if not for our newfound helper.

That's why I feel like my darkest days are over and now seeing glimpses of light at the end of the tunnel.

---

My vision for RPA for Python remains - I would like it to be the go-to Python package for RPA.

A best-in-class tool in Python users toolbox which you combine with other cool packages to build amazing stuffs.

[fi-do]

Hey Sam621, to change the user agent you can edit your tagui configuration. Open your src/tagui config file and add the user-agent flag to your chrome switches. For example:

chrome_started="$(uname -s)"; chrome_switches="--user-agent=TEST --user-data-dir=chrome/tagui_user_profile --remote-debugging-port=9222 about:blank" A little bit hacky, but I hope it will help you.

[Sam621]

Cool. Thanks. I will give a try and back to you

[kensoh]

Thanks @fi-do ! Hi @Sam621, this tool has a different architecture from Selenium and thus does not have options to change user agent. However it might be possible to send custom commands to Chrome through TagUI's chrome_step() function.

Alternatively, see if your use case can it be done using visual automation and keyboard() function. Eg start with r.init(True, False). In that mode, there is no backend connection to Chrome, so the tool will not be detected as a robot. However, it is likely to be harder to automate the web process using image snapshots instead of web identifiers.

Also, this RPA for Python package is based on a forked version of TagUI open-source RPA tool. Feel free to join Telegram community group chat to post any questions - https://t.me/rpa_chat

[omegastripes]

Hey Sam621, to change the user agent you can edit your tagui configuration. Open your src/tagui config file and add the user-agent flag to your chrome switches. For example:

chrome_started="$(uname -s)"; chrome_switches="--user-agent=TEST --user-data-dir=chrome/tagui_user_profile --remote-debugging-port=9222 about:blank" A little bit hacky, but I hope it will help you.

Hi to all! Any clue on how to add space char to user agent?

Update

The answer is:

• use separate variable,
• single quote the entire user agent switch string or escape spaces and special chars like ;, (, ), etc. with backslashes,
• double quote variable.

Here is shortened example based on tagui script:

#!/usr/bin/env bash
chrome_command="google-chrome"
chrome_switches="--no-sandbox --headless --disable-gpu --remote-debugging-port=9222"
user_agent='--user-agent=test 12345'
$chrome_command $chrome_switches "$user_agent" https://webhook.site/1d2f6cd3-ced8-4473-9882-31e826a3daf2

Going that way it's possible to specify user agent switch by setting env variable in python script:

os.environ['user_agent'] = r'--user-agent=test 1234555'

And in addition to user agent change I would add --disable-features=UserAgentClientHint switch.

[Samlearn621]

HI @fi-do @kensoh , Thanks all of your great responses. I will give a shot and let you know how things goes. :)