Automatic cookie retrieval for thx_guid fails

[felciano]

I am getting a "Mint.com login failed" error when get_session_cookies attempts to retrieve cookies automatically. I've traced it and the failure occurs in the last stage when trying to retrieve thx_guid from the https://pf.intuit.com URL.

ius_session appears to be retrieved correctly, and if I insert a couple of time.sleep() calls in the flow I see the regular Mint UI show up correctly so the initial auto-login process is working. However when retrieving the pf.intuit.com URL with ius_session as a parameter, the resulting page does not seem to include the thx_guid cookie (at least as far as I can tell when I try to go to the page using Chrome myself and then inspecting cookies using dev tools).

Any suggestions for how to diagnose further?

[jprouty]

I just noticed this as well! I haven't looked as deep, but my suspicion is that I'm hitting this same issue.

[jprouty]

Well, I've played around with this, and it looks like the two required session cookies are now: 'incap_ses_726_966665' # This is what I get now for 2 different accounts. It sounds like the numbers can change. 'ius_session'

From a quick search, it sounds like 'incap_ses' is a DoS service by Incapsula (see https://ico.org.uk/global/cookies/).

Two possible fixes: a) Replace thx_guid with incapses*. Since the incap numbers can change, this requires changing the API to allow the client to specify the current name. b) Retrieve all cookies from webdriver and set this to requests.Session.cookies. Requires changing API to accept a dict (or raw string) to support client supplied cookies.

(b) Seems much more forgiving for future changes in Mint's auth and acts more like the browser would. Furthermore, passing through all the cookies (like the real 1st party app) will make it harder to discern 3rd party requests.

I'm happy to whip up either.

[jprouty]

^ I thought I had it figured out. Not quite it seems (no longer getting a ius_session back!)

[mitch-lindsay]

@jprouty Were you able to test with solution b? I agree that this seems like the most future proof approach.

[jprouty]

Yes, B was working (mostly). Let me cleanup the PR real quick and hopefully you can help verify?

[jprouty]

Please give this a shot! https://github.com/mrooney/mintapi/pull/120

[mrooney]

Going to review #120 shortly, would love any yays or nays from others on if it worked for them!

On Fri, Sep 29, 2017 at 2:45 PM, Jeff Prouty notifications@github.com wrote:

Please give this a shot!

120 https://github.com/mrooney/mintapi/pull/120

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mrooney/mintapi/issues/118#issuecomment-333221149, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHjRaFpxr2EiYhixQv4GwwJ-l53raUVks5snUjBgaJpZM4PiDF7 .

[felciano]

@mrooney @jprouty I'd love to help test this since I submitted the bug, but we've strayed into unfamiliar gitland, and I'm having trouble finding a succinct explanation of how to test a proposed pull request. Do I just merge in directly from jprouty:master (which I assume would merge in all the changes on master) or should I only test this patch, in which case, how do I selectively merge in just this pull request?

I know this is OT but I'd love to help so would appreciate any pointers. I'm on a Mac using Github Desktop and Tower clients, in case it matters.

[jprouty]

I appreciate your effort. If you want the simplest way, it's rather trivial to download the one src file at the commit you care about, and then simply change your import statement from import mintapi to import api. Here's a link to the raw file in that PR

[felciano]

OK I did a clean install of @jprouty's branch into a fresh virtualenv with only chromium, pandas, and keyring, and am seeing several issues. In both cases, the login seems to be working, at least insofar as I can visually debug by seeing main UI show up in the browser:

Retrieving accounts works, but returns null:

mintapi --keyring $(cat /path/to/.config/.mintusername) --accounts
null

If I add some debugging statements to api.py it looks like the response code in get_accounts is returning <error><code>1</code><description>Session has expired.</description><name></name><type></type></error> (here: https://github.com/mrooney/mintapi/blob/82892422ba2de55b27d50bff44e334ad9149f843/mintapi/api.py#L244)

Second, retrieving transactions produces a MIME type mismatch error:

mintapi --keyring $(cat /path/to/.config/.mintusername) --transactions --filename output.csv
Traceback (most recent call last):
  File "/path/to/.Virtualenvs/minttest/bin/mintapi", line 11, in <module>
    load_entry_point('mintapi==1.28.1', 'console_scripts', 'mintapi')()
  File "/path/to/.Virtualenvs/minttest/lib/python2.7/site-packages/mintapi/api.py", line 766, in main
    data = mint.get_transactions(include_investment=options.include_investment)
  File "/path/to/.Virtualenvs/minttest/lib/python2.7/site-packages/mintapi/api.py", line 435, in get_transactions
    s = StringIO(self.get_transactions_csv(include_investment=include_investment))
  File "/path/to/.Virtualenvs/minttest/lib/python2.7/site-packages/mintapi/api.py", line 408, in get_transactions_csv
    expected_content_type='text/csv')
  File "/path/to/.Virtualenvs/minttest/lib/python2.7/site-packages/mintapi/api.py", line 121, in request_and_check
    (url, content_type, expected_content_type))
RuntimeError: Error requesting 'https://mint.intuit.com/transactionDownload.event', content type 'text/html; charset=UTF-8' does not match 'text/csv'

[mitch-lindsay]

Just to isolate any idiosyncrasies, I pulled down jprouty:master clean and my results are mostly consistent with above. Transactions consistently return the error specified above but accounts has occasionally returned results. I can't figure out what the differentiator is between a populated response and a null response.

[jprouty]

Thank you both; looking at these now. I think I'm onto something. Stay tuned

[jprouty]

@felciano @mitch-lindsay What version of python are you guys using? And OS?

It seems that I'm getting [nearly] perfect results on 3.5/3.6 on ubuntu/mac respectively. 2.7.13 on either platform is giving me trouble. I've even tried capturing the session cookies from 3.X and use those over in 2.7 land. I have a feeling it's some sort of encoding error that py3 handles fine and 2 is having a hard time.

The culprit (on py27) is that the /getUserPod.xevent is returning a: 400 Bad Request. Since this 400's, all the actual API calls seem to fail as well, typically resulting in a "Session is expired." response.

Ugh. I think I'm close.

[felciano]

@jprouty I'm running 2.7 on OSX 10.12.6, so that might indeed be the issue

[zosocanuck]

I'm also getting a similar error on 2.7.14 on Windows 10 x64.

[mitch-lindsay]

I was also running 2.7 on OSX 10.12.6. Upgrading to 3.6.2 changed the behavior a bit but getting accounts is still sporadically successful. Instead of returning null immediately, it appears that the request hangs indefinitely while waiting for a response.

^CTraceback (most recent call last):
  File "mintapi/api.py", line 783, in <module>
    main()
  File "mintapi/api.py", line 722, in main
    mint = Mint.create(email, password, session_cookies=session_cookies)
  File "mintapi/api.py", line 79, in create
    mint.login_and_get_token(email, password, session_cookies)
  File "mintapi/api.py", line 140, in login_and_get_token
    session_cookies = self.get_session_cookies(**data)
  File "mintapi/api.py", line 198, in get_session_cookies
    driver.get("http://accounts.intuit.com")
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 320, in get
    self.execute(Command.GET, {'url': url})
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 306, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 464, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 488, in _request
    resp = self._conn.getresponse()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
KeyboardInterrupt

[jprouty]

Very interesting. I unfortunately was unable to crack the case. I'm unable to work on it for the next week, so please don't wait on me if someone else has the resources to check it out.

Jeff

On Wed, Oct 4, 2017, 9:08 AM Mitchel Lindsay notifications@github.com wrote:

I was also running 2.7 on OSX 10.12.6. Upgrading to 3.6.2 changed the behavior a bit but getting accounts is still sporadically successful. Instead of returning null immediately, it appears that the request hangs indefinitely while waiting for a response.

^CTraceback (most recent call last): File "mintapi/api.py", line 783, in main() File "mintapi/api.py", line 722, in main mint = Mint.create(email, password, session_cookies=session_cookies) File "mintapi/api.py", line 79, in create mint.login_and_get_token(email, password, session_cookies) File "mintapi/api.py", line 140, in login_and_get_token session_cookies = self.get_session_cookies(**data) File "mintapi/api.py", line 198, in get_session_cookies driver.get("http://accounts.intuit.com") File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 320, in get self.execute(Command.GET, {'url': url}) File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 306, in execute response = self.command_executor.execute(driver_command, params) File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 464, in execute return self._request(command_info[0], url, body=data) File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 488, in _request resp = self._conn.getresponse() File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1331, in getresponse response.begin() File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 297, in begin version, status, reason = self._read_status() File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 258, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 586, in readinto return self._sock.recv_into(b) KeyboardInterrupt

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mrooney/mintapi/issues/118#issuecomment-334206907, or mute the thread https://github.com/notifications/unsubscribe-auth/AADJarSrcg4-xpdP7GoUEmxDmQjtBfj2ks5so61ygaJpZM4PiDF7 .

[mitch-lindsay]

The more I've been thinking about this problem, the more it seems like we're fighting the wrong battle. To make this tool work, we have to make assumptions about how mint is handling their authentication. What I propose instead is making all data requests in the context of the selenium driver. I wrote a native version of pulling the transaction data from just the driver in the Ruby app I was integrating this tool into and everything works great. I imagine the same could be true for the other data sources as well. This library seems to solve the problem of not being able to make POST requests from selenium as well. https://pypi.python.org/pypi/selenium-requests/

[nkirsch]

I'm presuming this is well know, but on macOS 10.13.1, using python2 (2.7.14) or python3 (3.6.3), I receive the same error when attempting to login:

  File "mintapi/api.py", line 783, in <module>
    main()
  File "mintapi/api.py", line 722, in main
    mint = Mint.create(email, password, session_cookies=session_cookies)
  File "mintapi/api.py", line 79, in create
    mint.login_and_get_token(email, password, session_cookies)
  File "mintapi/api.py", line 155, in login_and_get_token
    data = {'clientType': 'Mint', 'authid': json_response['iamTicket']['userId']}
KeyError: 'iamTicket'

I've attempted both the @jprouty branch as well as mainline. Note that Chrome seemed to do what it should, logging into Mint.com with credentials, which caused a two-factor auth code to be sent.

[jprouty]

I'm no longer able to reproduce either the original error reported @felciano or run into @nkirsch error using mrooney/master. I wonder if Mint API reverted some of the API...?

[felciano]

@jprouty I am still seeing this error, including when trying with python3. Are you still able to get basic mintapi command line calls to work correctly?

[bmasterc]

@mitch-lindsay, thanks, that simplifies things quite a bit. After selenium login, just get token from the /overview.event page:

jsn = self.browser.find_element_by_name('javascript-user').get_attribute('value') self.token = json.loads(jsn)['token']

Make the selenium-requests web driver last for however long you need and add:

def get(self, url, headers=None):
    return self.webdriver.request('GET', url)
def post(self, url, data=None, headers=None):
    return self.webdriver.request('POST', url, data=data)

[jprouty]

@mitch-lindsay I'll likely give this approach a shot for my project! Great idea; considering that mint gives all sorts of auth challenges, I see this approach as being the most robust to future changes.

[jprouty]

120 now merged.