Ulathar commented 5 years ago

Hey there,

i just tried to update my local copy of my gog library with the latest games bought during the current xmas sales but no matter what i try, i always get:

23:58:06 | attempting gog login as 'redacted@mail.fubar' ... 23:58:08 | login successful! 23:58:08 | exiting...

Z:\Software\Games\GOG.com>gogrepo.py update -os windows -lang en de -skipknown 23:58:25 | loading local manifest... 23:58:25 | fetching game product data (page 1)... 23:58:26 | request failed: HTTP Error 403: Forbidden. will not retry. 23:58:26 | fatal... Traceback (most recent call last): File "Z:\Software\Games\GOG.com\gogrepo.py", line 1167, in main(process_argv(sys.argv)) File "Z:\Software\Games\GOG.com\gogrepo.py", line 1142, in main cmd_update(args.os, args.lang, args.skipknown, args.updateonly, args.id) File "Z:\Software\Games\GOG.com\gogrepo.py", line 627, in cmd_update 'page': str(i)}, delay=0) as data_request: File "Z:\Software\Games\GOG.com\gogrepo.py", line 170, in request page = opener.open(req) File "C:\Python36\lib\urllib\request.py", line 532, in open response = meth(req, response) File "C:\Python36\lib\urllib\request.py", line 642, in http_response 'http', request, response, code, msg, hdrs) File "C:\Python36\lib\urllib\request.py", line 570, in error return self._call_chain(args) File "C:\Python36\lib\urllib\request.py", line 504, in _call_chain result = func(args) File "C:\Python36\lib\urllib\request.py", line 650, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden

Looks like GOG has changed it's API. Any chance to get this working again? Login and even 2 way auth works just fine, can also verify local files but am unable to fetch my library and/or download new games / updates.

Edited: substituted email with pseudo mail adress ... just in case ...

willrun4fun commented 5 years ago

same problem

Shiryou commented 5 years ago

The issue here is (probably) that your session has expired and GOG requires completing a reCAPTCHA, which gogrepo doesn't support. You can get around this by copying your GOG session cookies from your browser as described in https://github.com/eddie3/gogrepo/issues/17#issuecomment-248807909

That solved it for me.

WisdomCode commented 5 years ago

I use a freshly created cookie from the script the same error occurs here unfortunately (See #38). Under this issue there seem to be some people facing this issue. I'm not sure whether its about the recaptcha, my account never got one before on any system, including the one using the gogrepy.py login request. If there a quick way to test/verify the cookie?

willrun4fun commented 5 years ago

Working here for me. Thanks!

WisdomCode commented 5 years ago

After copy pasting the values from my cookies.txt file into the gog-cookies.dat, the error 403 remains. Copying was necessary because the script somehow deleted my values from the firefox cookies.txt plugin. Instead, I manually looked up what values were captured in the original gog-cookies.dat, searched them in the downloaded cookies and copied the cookies value (the scrambled number/text thingie) into the gog-cookies.dat, replacing the old one. The following cookie values were present, maybe there is something missing? gog-al gog_lc gog_set galaxy-login-al galaxy-login-s

DerPit commented 5 years ago

I have two additional ones, gog_us and csrf. Not sure if they are important. So I created a fresh gog-cookies.dat from my (Vivaldi) Cookies. Had to convert the expiry date that the cookies.txt extension reports from Unix timestamp to a normal date, but now it indeed does work again.

Probably the easiest solution might be to go to GOG settings and disable the two-step login, so it should not use/send a captcha/confirmation. Have not tried it though, now that it works. Maybe next time the cookes get invalid (9-12 months or so?)

Ulathar commented 5 years ago

Hmm just tried to do the same as proposed:

logged out in gog (browser)
relogged (no captchas here!)
exported the cookie.txt with the cookies.txt addon (firefox)
compared both files (cookie.txt and that gog-cookies.dat file) Noticing that both files are totally different both in structure and content.

When i replace that files as described here i get a "AttributeError: 'str' object has no attribute 'search'" which was expectd because the file contents are completly different.

Shiryou commented 5 years ago

@WisdomeCode also used Firefox, so I'm guessing there's a difference in the way it represents the cookies between browsers.

DerPit commented 5 years ago

I had (to be sure) first deleted all cookies of gog, then logged in. I used the menioned cookies.txt extension to get the cookies for copy'n'paste. Indeed, that format is different from what the python code uses. But it has the same info. E.g., the displayed ones are .gog.com TRUE / FALSE 1576850035 gog_set some_cookie_value 1576850035 is the unix timestamp code, corresponding to 20.12.2019 13:53:55UT. I converted those to the format gogrepo.py wants, Set-Cookie3: gog_set="some_cookie_value"; path="/"; domain=".gog.com"; path_spec; domain_dot; expires="2019-12-20 13:53:55Z"; HttpOnly=None; version=0 Doing that for all gog cookies allows the script to run...

WisdomCode commented 5 years ago

@DerPit The Idea of converting the timestamp is interesting, how did you do that in a quicker fashion than typing it into some webinterface and back? Beside of that, I did about what @DerPit did, with the exception that I didn't care for anything other than the "some_cookie_value" section. The other parts should either be identical or represent a sent cookie by gog, either way there are no reasons to assume that there should be any problem there (as long as the expiry date is far enough away). Please correct me if I'm wrong here. @Shiryou That may be an explanation, however one that I cannot base on any technical basis. A cookie is just a String of letters and numbers, no reason to assume that Firefox somehow gets different cookies than Vivaldi or Chrome, especially if you know that the script basically has no browser typical behavior beside cookies at all. I will try to add the gog_us and csfr cooky later and report back if that made it work.

DerPit commented 5 years ago

@WisdomCode: I had a local tool (from some other project). But just as good (if you're in Unix environment) is date -u --date='@1576850035' +%Y-%m-%d\ %TZ

Lebostein commented 5 years ago

Any hints to solve this bug? I have the same problem...

disi commented 5 years ago

Probably the easiest solution might be to go to GOG settings and disable the two-step login, so it should not use/send a captcha/confirmation. Have not tried it though, now that it works. Maybe next time the cookes get invalid (9-12 months or so?) Unfortunately this does not work :(

Lebostein commented 5 years ago

I've always deactivated the two-step thing... I have deleted the old cookie file and created a new. But nothing helps...

disi commented 5 years ago

The two factor authentication is not the problem, this is already in the script. It is the google recaptcha, where you click "I am not a Robot".

Lebostein commented 5 years ago

I have never seen a "I am not a Robot" window while login... It seems all is turned off here.

WisdomCode commented 5 years ago

Even after setting the other cookie values, error 403 persists. As the cookies captured by the script contain all the important values (I assume gog-al and gog_set to be the important ones, however it doesn't matter), I assume that the login was successful, despite error 403. Like @Lebostein also said, Google captcha was never a thing for me and should probably end with no cookie sent at all, the login procedure itself would probably fail. Cokkies ARE send, so it would be best to test them in some way. Is there a way to apply the cookies from the .dat file into the browser (basically the opposite of what cookies.txt does), so that we can see whether the opening of gog will result in being already logged in (assuming that there were no cookies before of course, a cleaned up browser is needed)? Will this even work when this error does not occur with the gog-cookies.dat?

disi commented 5 years ago

OK!!!111elevenoneone Use Chrome + EditThisCookie Extension Logon to GOG.com and keep the page open, in EditThisCookie -> Options -> change the format to Netscape Then export the Cookie to Clipboard with the Extension Then Open your gog-cookies.dat and replace all with the copied cookie (paste) and save. Open gogrepo.py, (line 233 fff) comment the complete cookie load procedure and save, so it fails over to the Netscape cookie load attempt: def load_cookies():

try to load as default lwp format

#try:
#    global_cookies.load()
#    return
#except IOError:
#    pass

# try to import as mozilla 'cookies.txt' format
try:
    tmp_jar = cookiejar.MozillaCookieJar(global_cookies.filename)
    tmp_jar.load()
    for c in tmp_jar:
        global_cookies.set_cookie(c)
    global_cookies.save()
    return
except IOError:
    pass

error('failed to load cookies, did you login first?')
raise SystemExit(1)

After the first successful run/load of gogrepo.py, it will regenerate a cookie it can handle, so remove the # to un-comment the lines again. Save and be happy

Lebostein commented 5 years ago

@disi: YES! That's ist. It works. Thansk for your investigation!

WisdomCode commented 5 years ago

Wow, so it did for me, thanks! Would you mind to explain what exactly is going on here? Is it something about the cookies format?

disi commented 5 years ago

I blame the Recaptcha, it is really hard to see the images, detect cars and such and then tick the box. Btw. if you just remove the IOError bit from the code, it then runs boths and you can easily import netscape cookies:

def load_cookies():
    # try to load as default lwp format
    try:
        global_cookies.load()
        return
    except:
        pass

    # try to import as mozilla 'cookies.txt' format
    try:
        tmp_jar = cookiejar.MozillaCookieJar(global_cookies.filename)
        tmp_jar.load()
        for c in tmp_jar:
            global_cookies.set_cookie(c)
        global_cookies.save()
        return
    except:
        pass

    error('failed to load cookies, did you login first?')
    raise SystemExit(1)

Lebostein commented 5 years ago

gloated too soon... it seems it works only for round about 3 hours, then it seems the cookie expired and the error comes back...

WisdomCode commented 5 years ago

At least its a consistent method of making it work. However, to close this issue, we should identify the source of the problem. I am not sure how the google captcha is implemented into the login process (never got one), therefore I am not sure how to find it. Maybe a specific error throwing on login when google captcha is requested would be a good start.

DerPit commented 5 years ago

I also have never seen a captcha when loggin in via browser, so I think it's something different. Nevertheless, I found two tools. First the browser extension Export Cookies 0.3.0, and then a small perl script from https://stackoverflow.com/questions/27886991/convert-netscape-cookies-to-lwp-perl. I adapted the filenames in the script, so now it's only a three-step thing: Log in in browser, click the 'export cookies' for gog.com, and run the perl script. With that, gogrepo.py works without changes.
Have not yet checked how long it stays valid - there's two cookies (csrf and gog_us) that have a close-by expiry date. And a new one I hadn't seen before, galaxy-login-tsa from login.gog.com.

Ulathar commented 5 years ago

Doesnt seem to work for me (getting "failed to load product data (are you still logged in?)") now when using your workaround (can't execute perl scripts so i used the Chrome Plugin Edit This Cookie which is able to export cookies in LWP format).

Edit: compared to the .dat file generated by gogrepo.py there are at least 2 gog-galaxy related cookies missing using the "browser method".

DerPit commented 5 years ago

Some more observations. After some time I had tried connecting to gog again using the cookie file created as described in the previous post. Would get the 403 again :( So I re-ran the described sequence(*) and with the new generated gog-cookies.dat connection worked again. So I compared the two (old and new) cookie files. 4 cookies were unchanged: gog_set, galaxy-login-al, galaxy-login-s and galaxy-login-tsa. gog_lc and csrf have same values, but new expiry dates (but for gog_lc that is 1 year in the future) gog-al and gog_us have new values (and expiry date).

side note: The cookies come from a different computer. I run browser and conversion script on the desktop machine, and copy the generated gog-cookies.dat to my laptop where I run gogrepo.py

WisdomCode commented 5 years ago

Now the proposed method with chrome which was working previously is not working anymore either... Is this repository still maintained?

DerPit commented 5 years ago

You mean via the cookie export? For me it does still work (Vivaldi with Export Cookies 0.3.0).

Looks someone with knowledge in both Python and HTML is needed - I suck at both :((

WisdomCode commented 5 years ago

I could do it, however, I am not skilled in the Libraries used. For the moment, I will try to reimplement the login with different libraries (requests, BeatifulSoup, eventually robobrowser). I will give feedback when I have reached something, maybe I'll create a fork then.

disi commented 5 years ago

This would be great.

On Tue, 29 Jan 2019, 11:01 WisdomCode <notifications@github.com wrote:

I could do it, however, I am not skilled in the Libraries used. For the moment, I will try to reimplement the login with different libraries (requests, BeatifulSoup, eventually robobrowser). I will give feedback when I have reached something, maybe I'll create a fork then.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/eddie3/gogrepo/issues/39#issuecomment-458498719, or mute the thread https://github.com/notifications/unsubscribe-auth/AA1h9iIoe6QLM8o3VeRLWG79fx0gRiuAks5vICoEgaJpZM4ZXSgp .

WisdomCode commented 5 years ago

Well, I rewrote the Login code, I investigated every step and every loaded site. Everything, however, looks absolutely fine. The cookies still are not usable for the script. There were a handful of oddities though:

There is already a recaptcha error implemented,there is an error message when it occurs. I made this search even a bit thorougher, risking false positives to absolutely be sure.
There is a login_flow in the login process, which is set default, but the login value is empty. Corrected that.
The Login request is a post request, not sure if gogrepo is doing that.

None of the code contains a captcha, the last url indicates to a successful URL, and as we already discovered, the cookies look fine too. This makes me sure by now that its not a problem with the login process, but with the usage of the cookies. I may post the Code if anyone wants to investigate further

WisdomCode commented 5 years ago

Now I made another interesting discovery that may actually explain it: If you login under https://login.gog.com/login (which is basically the url of the login which is embedded into the actual GOG page). The login is entirely done via the login.gog.com subdomain, this is also true for the script. Logging in there however leaves you not logged in on the original page (of course given your not logged in already). Maybe somebody could confirm this phenomenom? If this is the case, the cookie is probably simply not handed to the main webpage, rendering them useless.

version2013 commented 5 years ago

After logging in via https://login.gog.com/login , opening another tab at https://www.gog.com/ , results in 'SIGN IN' still visible at the top. This happened the majority of the login attempts. So far, only three times has that login page (after inputting credentials) redirected to a logged in gog.com main page.

DerPit commented 5 years ago

version2013 wrote:

After logging in via https://login.gog.com/login , opening another tab at https://www.gog.com/ , results in 'SIGN IN' still visible at the top. This happened the majority of the login attempts. So far, only three times has that login page (after inputting credentials) redirected to a logged in gog.com main page.

Just an idea:

It seems that (some of the) cookies from gog.com have short validity dates. Could it be that on connect (to gog.com) it checks with login.gog.com and issues a new cookie or something like that? and that gogrepo.py somehow isn't doing that?

If I find time today I'll try playing with the various logins, too....

WisdomCode commented 5 years ago

Well, I am no real expert to cookies, but I can tell that pythons internal structure on acquiring the cookies is no different than the one the browser uses. No Cookies are directly dropped, and the login process is even quicker than the manual one, so no reason to assume they time out somehow.

I believe (and this is the part where I am not sure) that the embedding of the login page into the main page results in a different kind of cookie. Maybe the same values are given, but the domain of the cookie is different or something like that. Maybe somebody has experience with the netscape format of the cookies and these values in general? Is something like that possible.

Ulathar commented 5 years ago

Anything new on this case? Just read, that gog seems to have financial problems.... Would make an "offline backup" of my current (huge!) lib even more usefull but sadly I can't get this script to work again with the proposed workarounds...

Strubbl commented 5 years ago

I have the same problem.

@Ulathar where did you hear about gog having problems? Any link?

stesser commented 5 years ago

Am 26.02.19 um 21:20 schrieb Strubbl:

I have the same problem.

@Ulathar https://github.com/Ulathar where did you hear about gog having problems? Any link?

They have announced that they'll stop the "fair price" scheme offered to customers in certain countries because of a too low revenue:

https://www.gog.com/newsletter/20190226_Updates_Legal_en

(The reasoning was found at a page linked from that update, but I could not find it when I looked for it, now ...)

BTW: Has anybody had a look at what Lutris (https://lutris.net) does to have access to GOG games?

Maybe they have solved this download problem ...

WisdomCode commented 5 years ago

A good idea to look up how lutris is doing it. Never actually tried it that way, but now after trying, I can say 2 things:

It doesn't work for me either. I enter username and password, enter two factor auth, and... nothing. The window disappears the window previously downloading wine is still stuck there at 100%, no network activity on the device. I fear the same problems are faced there, although I could not find an issue in their tracker. Unfortunately I cannot find the location of the log files to harden this suspicion.
The system used differs at least partly. I think it uses electron to embed the website into the local app run in python, which is quite different from what is done here. The result will however probably be the same: Cookies get extracted and used for downloading the application.

Maybe somebody can tell me where to find the log file? Google Fu fails me here...

DerPit commented 5 years ago

No idea about lutris, I failed to get 0.5 working. But another possibility to look at might be lgogdownloader. I just compiled and tested the latest git version, and I'm able to log in and download things(*). But I think it uses the gogdownloader/galaxy api, not sure if that is something different.

(*)Then again, for me also the transfer-cookies-from-browser solution with gogrepo.py does work.

Ulathar commented 5 years ago

I have the same problem.

@Ulathar where did you hear about gog having problems? Any link?

I read an article which said that they had to lay off 10% of their employes alongside with the already mentioned cut in their "fair price" program...

The metioned lgogdownloader is used by one of my colleagues and seems to be working. But it only runs on linux so i can't use it myself.

Strubbl commented 5 years ago

I can confirm, the lgogdownloader works for me (on Linux). Maybe one can create a Dockerfile to make use of it on a different platform?

I only installed it. Then i launched it simply with lgogdownloader and logged in with my data. Afterwards lgogdownloader --directory ./backup/ --download downloads all the files.

disi commented 5 years ago

Since this method works for a while: https://github.com/eddie3/gogrepo/issues/39#issuecomment-449516889 I think we need to find a way to keep the cookie alive longer. Kalanyrs fork uses session classes and seems to run longer, but also fails to login and generate a usable cookie, also it has some other design decision I don't like.

Shiryou commented 5 years ago

I was able update my library by using find/replace of expires="2019 with expires="2020 on my gog-cookies.dat file.

That won't work for cookies from Firefox, though, because those use the Unix epoch instead of a human readable date. In that case, you would have to calculate a new timestamp.

PescheHelfer commented 5 years ago

When creating the gog-cookies.dat using the EditThisCookie Extension and editing gogrepo.py as mentioned above, I get the following error:

fatal...
Traceback (most recent call last):
  File "F:\Libraries\Pesche\Docs_Pesche\Coding\Python\GOGRepo\gogrepo.py", line 1167, in <module>
    main(process_argv(sys.argv))
  File "F:\Libraries\Pesche\Docs_Pesche\Coding\Python\GOGRepo\gogrepo.py", line 1142, in main
    cmd_update(args.os, args.lang, args.skipknown, args.updateonly, args.id)
  File "F:\Libraries\Pesche\Docs_Pesche\Coding\Python\GOGRepo\gogrepo.py", line 605, in cmd_update
    load_cookies()
  File "F:\Libraries\Pesche\Docs_Pesche\Coding\Python\GOGRepo\gogrepo.py", line 242, in load_cookies
    tmp_jar.load()
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Anaconda3_64\lib\http\cookiejar.py", line 1784, in load
    self._really_load(f, filename, ignore_discard, ignore_expires)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Anaconda3_64\lib\http\cookiejar.py", line 2004, in _really_load
    if not self.magic_re.search(magic):
AttributeError: 'str' object has no attribute 'search'

Is anybody else getting this error?

edit: forget it, I realized there is a newer fork by Kalanyr. Saving the cookies as cookies.txt (not gog-cookies.dat) and using his version did the trick :)

Ulathar commented 5 years ago

edit: forget it, I realized there is a newer fork by Kalanyr. Saving the cookies as cookies.txt (not gog-cookies.dat) and using his version did the trick :)

Hm? Does that mean that the Kalanyr fork is working? Can you explain what you did exactly?

Ulathar commented 5 years ago

Did some tests with that Kalanyr fork but it doesn't seem to work either. Although the errors in that fork are different: The login and cookie handling seems to work just fine (in contrast to this one). But the downloading and updating also fails but with a different reason:

line 262, in getattr return self[key] KeyError: 'galaxyDownloads'

Edit: ok this was caused by the gog-manifest.dat file (the one from this gogrepo.py is not compatible with the one from the kalanyr fork). Looking good so far, fetching stuff again. Hope that the DL will work, too. Edit2: Downloads are working, too. So switching over to the Kalanyr fork is indeed a working solution! Nice find spitfireCH, thx!

PauloGDPeixoto commented 3 years ago

I started having this issue today for some games, tried the suggested Kalanyr fork and it gives the same error.

eddie3 / gogrepo

HTTP Error 403: Forbidden after update -os windows -lang en de -skipknown #39

try to load as default lwp format