Redeeming humble keys via selenium

[jarjee]

I've based this PR off the excellent initial work by KaiserBh (#26), and we now can redeem keys using selenium.

The issue with not being able to redeem keys was with cloudfront blocking the requests - I'm wondering if there's some extra fingerprinting at play which means selenium keeps working, but going back to cloudscraper breaks it somehow.

What still isn't working is the humble chooser mode, it's not finding months in my case and not entirely sure why. I'll have a poke around at some point, but now I've got a massive backlog to go through.

The code should be able to handle failures (the only game I had issues with was Husk, which had key exhaustion). I haven't gotten it to try that key a second time, so I claim it's resilient at least.

I've also added a flake.nix & flake.lock to the project - this should hopefully make installing all the new dependencies a little easier. I had it just work with the gecko driver, but I'd love someone on another machine (MacOS) to give this a shot and tell me if it works or not.

[FailSpy]

Hey, thanks for doing this @KaiserBh and @jarjee. Selenium is probably the best way going forward for the redeemer.

Here are some of my thoughts/concerns: This is really just for @KaiserBh: Tor routing makes me a little nervous. I'd rather offer the ability to simply proxy your requests through a SOCKS proxy, which the user could have as Tor. A user who wants to use Tor should be knowledgeable enough to do this, and have their reasons for doing so. I'm a little weary of this otherwise as it's likely this can raise some flags going forwards. Additionally, I don't think it should be a requirement that gets downloaded.

I'd maybe also like a little more detailed how-to section on Selenium configuration -- though I understand this is a little fiddly. I'm not sure how much autoconfiguration Selenium does out-of-the-box these days, as I think it now installs a webdriver on its own, but haven't tested this yet.

I also noticed you're using the actual web interface (clicking on actual page elements) to fire off requests with some selectors. I think it would be better to go back to using the API as much as possible, but simply using Selenium as the medium for those requests while the browser is on the "correct" webpage, via injected JS, making the requests as the website would.

Some excerpt of a hacky login method I modified the old code to do via Selenium:

fetch_cmd = '''
var done = arguments[arguments.length - 1];
var formData = new FormData();

formData.append('access_token','{access_token}');
formData.append('access_token_provider_id','');
formData.append('goto','/');
formData.append('qs','');
formData.append('username','{username}');
formData.append('password','{password}');
formData.append('guard','{guard}');
formData.append('code','{code}');

fetch("https://www.humblebundle.com/processlogin", {{
  "headers": {{
    "csrf-prevention-token": "{csrf}"
    }},
  "body": formData,
  "method": "POST",
}}).then(r => {{ r.json().then( v=>{{done([r.status,v])}} ) }} );
'''

        username = input("Humble Email: ")
        password = getpass.getpass("Password: ")

        # go to login screen
        driver.get(HUMBLE_LOGIN_PAGE)

        payload = {
            "access_token": "",
            "access_token_provider_id": "",
            "goto": "/",
            "qs": "",
            "username": username,
            "password": password.replace("\\","\\\\").replace("'","\\'"), # escape ' and \ chars in passwords
            "guard":'',
            "code":'',
            "csrf": driver.get_cookie('csrf_cookie')['value']
        }

        auth,login_json = driver.execute_async_script(fetch_cmd.format(**payload))

All that scary wall of thoughts aside, sincerely thanks again to both of you though for all the work on this on cracking this open!

[rupert404]

I've stumbled upon this fork by @Nionor (who hasn't submitted a PR though) https://github.com/Nionor/humble-steam-key-redeemer and fiddled around with it for a bit.

Instead of using a proxy etc. just for signing in (since using Tor just for this seemed a bit much to me as well), why isn't getting cookie data from browsers (e.g. via the library browser_cookie3 - not necessarily manually) an option? I've rewritten a bit of his stuff to attempt it using browser_cookie3 but I couldn't get it to run (attempts still failing). I just replaced the parts where it expects users to manually input exported cookies - which didn't work for me either.

I do know just exporting and using cookies from a browser can work in various applications (e.g. for Twitter or Instagram scraping or automation), I just don't know if there is anything here specifically preventing us from just doing that - or if that's a worse option (vs. proxy).

What still isn't working is the humble chooser mode, it's not finding months in my case and not entirely sure why.

This might also be adressed in the fork by @Nionor https://github.com/Nionor/humble-steam-key-redeemer See commit https://github.com/FailSpy/humble-steam-key-redeemer/commit/a683341cfbe7d4d3215f8762109e6cbc11ac90c2

[kaiserbh]

Hey, thanks for doing this @KaiserBh and @jarjee. Selenium is probably the best way going forward for the redeemer.

Here are some of my thoughts/concerns: This is really just for @KaiserBh: Tor routing makes me a little nervous. I'd rather offer the ability to simply proxy your requests through a SOCKS proxy, which the user could have as Tor. A user who wants to use Tor should be knowledgeable enough to do this, and have their reasons for doing so. I'm a little weary of this otherwise as it's likely this can raise some flags going forwards. Additionally, I don't think it should be a requirement that gets downloaded.

I'd maybe also like a little more detailed how-to section on Selenium configuration -- though I understand this is a little fiddly. I'm not sure how much autoconfiguration Selenium does out-of-the-box these days, as I think it now installs a webdriver on its own, but haven't tested this yet.

I also noticed you're using the actual web interface (clicking on actual page elements) to fire off requests with some selectors. I think it would be better to go back to using the API as much as possible, but simply using Selenium as the medium for those requests while the browser is on the "correct" webpage, via injected JS, making the requests as the website would.

Some excerpt of a hacky login method I modified the old code to do via Selenium:

fetch_cmd = '''
var done = arguments[arguments.length - 1];
var formData = new FormData();

formData.append('access_token','{access_token}');
formData.append('access_token_provider_id','');
formData.append('goto','/');
formData.append('qs','');
formData.append('username','{username}');
formData.append('password','{password}');
formData.append('guard','{guard}');
formData.append('code','{code}');

fetch("https://www.humblebundle.com/processlogin", {{
  "headers": {{
    "csrf-prevention-token": "{csrf}"
    }},
  "body": formData,
  "method": "POST",
}}).then(r => {{ r.json().then( v=>{{done([r.status,v])}} ) }} );
'''

        username = input("Humble Email: ")
        password = getpass.getpass("Password: ")

        # go to login screen
        driver.get(HUMBLE_LOGIN_PAGE)

        payload = {
            "access_token": "",
            "access_token_provider_id": "",
            "goto": "/",
            "qs": "",
            "username": username,
            "password": password.replace("\\","\\\\").replace("'","\\'"), # escape ' and \ chars in passwords
            "guard":'',
            "code":'',
            "csrf": driver.get_cookie('csrf_cookie')['value']
        }

        auth,login_json = driver.execute_async_script(fetch_cmd.format(**payload))

All that scary wall of thoughts aside, sincerely thanks again to both of you though for all the work on this on cracking this open!

No worries, also I will just put this here regarding the TOR network feel free to remove it, it's not being used for anything else except logging into humble bundle that being said it only uses it if the user wants too basically a y/n question which we kill the TOR network after login, and it was a misunderstanding in the first place which got me to implement it, if you read the issue I don't remember which one though it should give more context. I agree with using the API more than using the actual web interface.

Also, great job, @jarjee 👏

[FailSpy]

I've stumbled upon this fork by @Nionor (who hasn't submitted a PR though) https://github.com/Nionor/humble-steam-key-redeemer and fiddled around with it for a bit.

Instead of using a proxy etc. just for signing in (since using Tor just for this seemed a bit much to me as well), why isn't getting cookie data from browsers (e.g. via the library browser_cookie3 - not necessarily manually) an option? I've rewritten a bit of his stuff to attempt it using browser_cookie3 but I couldn't get it to run (attempts still failing). I just replaced the parts where it expects users to manually input exported cookies - which didn't work for me either.

I do know just exporting and using cookies from a browser can work in various applications (e.g. for Twitter or Instagram scraping or automation), I just don't know if there is anything here specifically preventing us from just doing that - or if that's a worse option (vs. proxy).

What still isn't working is the humble chooser mode, it's not finding months in my case and not entirely sure why.

This might also be adressed in the fork by @Nionor https://github.com/Nionor/humble-steam-key-redeemer See commit a683341

Interesting, good spot on the fork @rupert404. I've really not had much time to spend on trying to circumvent Cloudflare myself. Obviously, if there is a way to reliably get past Cloudflare's system without using Selenium that is vastly preferable, though it may turn into a harder game of cat-and-mouse, being easier to spot. Thus maintaining that would become a bit of a pain.

However, you mentioned that you can't get Nionor's fork working yourself, if I'm reading your comment correctly. Which I can't say I've had any greater success. I tried replicating Nionor's work by using Selenium for the initial login, and then copying the 3 cookies Nionor monitors from Selenium into a Requests session. That was unsuccessful, Cloudflare still took issue with that.

Even going a step further:

for cookie in driver.get_cookies():                                                         
    # convert cookies to requests
    session.cookies.set(cookie['name'],cookie['value'],domain=cookie['domain'],path=cookie['path'])

which indeed does copy the 3 cookies AND any additional ones generated in the login process, and Cloudflare still denied the requests.

If you do get it working, I'll be very curious to hear, but so far I've had no luck.

[Nionor]

Hi, saw I got mentioned here so I thought I say something. First I didn't submit a PR because I thought it was a kinda hacky way to do it, and since you had a elegant solution working before it was of no interest. Second for me my fork with the manual cookie input works, actually just had to do it because I guess the session on the cookie had timed out. I have made a few other changes on my local code but nothing that should effect the login part I think. What I do is just copy paste the values exactly as they are from the cookie storage in Firefox.

[FailSpy]

Aha, that's very useful to know, thanks @Nionor! Seems like with an authorized session, Cloudflare allows any requests to bypass the bot filter.

Worth noting that in a Firefox session, at least for me, if you simply open up the Humble Bundle homepage, use cookies from that, it will not work. If you perform an unsuccessful login on top of that (Humble Guard or incorrect password), it still won't work. Only after logging in will the cookies be authorized to perform requests. And notably, it will only be for that logged-in session. If I then log out, and use the same cookies, a manual login request will fail to go through from Cloudflare's protection.

[FailSpy]

I've just pushed an update that uses Selenium and the API calls. I've also fixed the Humble Monthly chooser, I think. (hard to test old monthlies as I've claimed those)

Thanks all for cracking this open.