search

csingley / ofxtools

Python OFX Library
Other
301 stars 68 forks source link

ofxget scan fails for etrade #63

Closed antipenultimate closed 5 years ago

antipenultimate commented 5 years ago

Getting OFX from e*trade (brokerage) is generally troublesome. Hoped ofxtools would work.

Did "ofxget scan etrade" and "ofxget scan https://ofx.etrade.com/cgi-ofx/etradeofx" and get the following error:

(rebalance) C:\Users\antip>ofxget scan https://ofx.etrade.com/cgi-ofx/etradeofx
Traceback (most recent call last):
  File "d:\anaconda3\envs\rebalance\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "d:\anaconda3\envs\rebalance\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "d:\Anaconda3\envs\rebalance\Scripts\ofxget.exe\__main__.py", line 9, in <module>
  File "d:\anaconda3\envs\rebalance\lib\site-packages\ofxtools\scripts\ofxget.py", line 1502, in main
    REQUEST_HANDLERS[args["request"]](args)
  File "d:\anaconda3\envs\rebalance\lib\site-packages\ofxtools\scripts\ofxget.py", line 458, in scan_profile
    scan_results = _scan_profile(url, org, fid)
  File "d:\anaconda3\envs\rebalance\lib\site-packages\ofxtools\scripts\ofxget.py", line 1137, in _scan_profile
    for fmt in v2_result["formats"]:
KeyError: 'formats'

Windows 10, Anaconda, Python 3.7.3. ofxtools 0.8.18

Other sites work fine.

csingley commented 5 years ago

Ah crap, this was just working recently. I've got accounts with ETFC, let me investigate.

csingley commented 5 years ago

All right, I believe I fixed what I broke. Scan, profile, acctinfo, stmt all working for me @ ETFC.

Upgrade to 0.8.20 and let me know if it works for you.

antipenultimate commented 5 years ago

Thanks!

Scan and stmt work. Acctinfo doesn't seem to:

(rebalance) C:\Users\antip>ofxget acctinfo etrade -u xxxx
Password:
Traceback (most recent call last):
  File "d:\anaconda3\envs\rebalance\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "d:\anaconda3\envs\rebalance\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "d:\Anaconda3\envs\rebalance\Scripts\ofxget.exe\__main__.py", line 9, in <module>
  File "d:\anaconda3\envs\rebalance\lib\site-packages\ofxtools\scripts\ofxget.py", line 1499, in main
    REQUEST_HANDLERS[args["request"]](args)
  File "d:\anaconda3\envs\rebalance\lib\site-packages\ofxtools\scripts\ofxget.py", line 555, in request_acctinfo
    acctinfo = _request_acctinfo(args, password)
  File "d:\anaconda3\envs\rebalance\lib\site-packages\ofxtools\scripts\ofxget.py", line 573, in _request_acctinfo
    password, dtacctup, dryrun=args["dryrun"], verify_ssl=not args["unsafe"]
  File "d:\anaconda3\envs\rebalance\lib\site-packages\ofxtools\Client.py", line 390, in request_accounts
    return self.download(ofx, dryrun=dryrun, verify_ssl=verify_ssl, timeout=timeout)
  File "d:\anaconda3\envs\rebalance\lib\site-packages\ofxtools\Client.py", line 570, in download
    response = urllib_request.urlopen(req, timeout=timeout, context=ssl_context)
  File "d:\anaconda3\envs\rebalance\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "d:\anaconda3\envs\rebalance\lib\urllib\request.py", line 531, in open
    response = meth(req, response)
  File "d:\anaconda3\envs\rebalance\lib\urllib\request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "d:\anaconda3\envs\rebalance\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)
  File "d:\anaconda3\envs\rebalance\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "d:\anaconda3\envs\rebalance\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: OFX Request Parsing Error
csingley commented 5 years ago

This is actually a separate root issue, I believe caused by ETFC rejecting requests with CLIENTUID set in the signon block. I thought I had addressed it in the update too.

Do me a favor and help characterize this bug, because I'm not replicating it on my setup.

Please verify that there's a blank CLIENTUID in the [etrade] block of /path/to/ofxtools/config/fi.cfg.

Assuming you've generated a config file, open that up (I think it's <home>\AppData\Roaming\ofxtools\ofxget.cfg under Windows). Under your own [etrade] block, the CLIENTUID parameter should be absent. There's likely a [DEFAULTS] block up top with a CLIENTUID set.

This is how it's supposed to look, and it works for me like this. Is this how yours looks?

If so, delete the whole [DEFAULTS] block so there's no source of CLIENTUID in the entire config. Try the acctinfo request again (making sure not to append a --clientuid option).

antipenultimate commented 5 years ago

I've verified that fi.cfg has a blank CLIENTUID in the [etrade] block, and nothing in the ofxget.cfg [etrade] block, and there was a CLIENTUID in the defaults block. I then deleted the entire [DEFAULTS] block, and tried acctinfo again. Same error message as above.

csingley commented 5 years ago

Can you provide the output of ofxget list etrade ?

csingley commented 5 years ago

Ahh... with acct#/userid redacted, naturally

antipenultimate commented 5 years ago

Result from ofxget list etrade:

E*TRADE
ofxhome = 446
clientuid =
url = https://ofx.etrade.com/cgi-ofx/etradeofx
version = 102
org = fldProv_mId
fid = fldProv_mProvBankId
brokerid = etrade.com

(I didn't write acct / userid to cfg)

But now I can no longer replicate getting "stmt" to work:

(py37_ofxtools) C:\Users\antip>ofxget stmt etrade -u xxxx --all
Password:
Traceback (most recent call last):
  File "d:\anaconda3\envs\py37_ofxtools\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "d:\anaconda3\envs\py37_ofxtools\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "d:\Anaconda3\envs\py37_ofxtools\Scripts\ofxget.exe\__main__.py", line 9, in <module>
  File "d:\anaconda3\envs\py37_ofxtools\lib\site-packages\ofxtools\scripts\ofxget.py", line 1499, in main
    REQUEST_HANDLERS[args["request"]](args)
  File "d:\anaconda3\envs\py37_ofxtools\lib\site-packages\ofxtools\scripts\ofxget.py", line 612, in request_stmt
    acctinfo = _request_acctinfo(args, password)
  File "d:\anaconda3\envs\py37_ofxtools\lib\site-packages\ofxtools\scripts\ofxget.py", line 573, in _request_acctinfo
    password, dtacctup, dryrun=args["dryrun"], verify_ssl=not args["unsafe"]
  File "d:\anaconda3\envs\py37_ofxtools\lib\site-packages\ofxtools\Client.py", line 390, in request_accounts
    return self.download(ofx, dryrun=dryrun, verify_ssl=verify_ssl, timeout=timeout)
  File "d:\anaconda3\envs\py37_ofxtools\lib\site-packages\ofxtools\Client.py", line 570, in download
    response = urllib_request.urlopen(req, timeout=timeout, context=ssl_context)
  File "d:\anaconda3\envs\py37_ofxtools\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "d:\anaconda3\envs\py37_ofxtools\lib\urllib\request.py", line 531, in open
    response = meth(req, response)
  File "d:\anaconda3\envs\py37_ofxtools\lib\urllib\request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "d:\anaconda3\envs\py37_ofxtools\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)
  File "d:\anaconda3\envs\py37_ofxtools\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "d:\anaconda3\envs\py37_ofxtools\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: OFX Request Parsing Error

Note that I set up a new conda environment (and repip'ed ofxtools) to try it in, but the original environment gives the same error.

antipenultimate commented 5 years ago

Hmmm. And now stmt works. I had to delete the Roaming directory where ofxget.cfg was. Now there is no ofxget.cfg.

antipenultimate commented 5 years ago

And acctinfo works. Still with no ofxget.cfg

csingley commented 5 years ago

Curious.

Those configs for etrade are right; that's exactly what you need to connect. Try writing the config to file (same way, no userid/acct#) just as in the old environment and see if that's what screws it up.

The error you're getting is from ETFC's web server; their OFX server pushes down "OFX Request Parsing Error" and the web server packs it in an HTTP 400 error. Sending OFXv102, all tags closed, no pretty-printing, with BROKERID=etrade.com and their weird-ass ORG/FID (which look like variable names they forget to substitute, and I believe are actually optional), I was receiving this error if I tried to send them a CLIENTUID, and it cleared up when I stopped doing that.

The inconsistency is irksome. Is it the presence/absence of an ofxget.cfg that's triggering this? Any condition to pinpoint that toggles the failure on/off?

You might try playing with the -n/--dryrun switch, which dumps the constructed request to stdout instead of sending it to the server. Doing a diff of those outputs, between your old venv and the new one, may possibly provide illumination.

csingley commented 5 years ago

Let me know if you need any more help with this.