csingley
commented
2 years ago Did you try matching Fund Manager's config?
e.g. version=102 vs. version=220
Also worth trying different configs for unclosedelements and prettyprint
Closed vitaliknet closed 2 years ago
csingley
commented
2 years ago Did you try matching Fund Manager's config?
e.g. version=102 vs. version=220
Also worth trying different configs for unclosedelements and prettyprint
vitaliknet
commented
2 years ago Did you try matching Fund Manager's config?
e.g.
version=102vs.version=220Also worth trying different configs for
unclosedelementsandprettyprint
Yes - tried below config. unclosedelements = true comes back with the same 403 Forbidden, while unclosedelements = false returns "bad request". Behavior of https://ofx.fidelity.com/ftgw/OFX/clients/download and https://ofx.fidelity.com:443/ftgw/OFX/clients/download seem to be the same.
Looking through output with " --verbose --verbose --verbose", ofxget request differs from quicken and Fund Manager in two fields - CHARSET: NONE and NEWFILEUID having a value. Not sure if they matter. Adding gen_newfileuid = false to ofxget.cfg does not seem to have any effect (I did not go carefully through your code to see if it is supposed to be picked up from there).
[fidelity] url = https://ofx.fidelity.com/ftgw/OFX/clients/download fid = 7776 brokerid = fidelity.com org = fidelity.com version = 102 appid = QWIN appver = 2700 language = ENG pretty = false unclosedelements = true
vitaliknet
commented
2 years ago After killing almost 24 active hours on troubleshooting, the mystery is solved. Fidelity seems to require ":443" in URL again. Specifically, this has to be the URL set in urllib.Request. My guess they ultimately check it in Host http header.
The issue is that by the time url value given to OFXClient() makes it to download()'s Request composition in lines 846-848 of Client.py, the ":443" part disappears. Changing line 847 from url to self.url fixes the problem for my use case, but am not sure about unintended consequences.
req = urllib_request.Request(
self.url, method="POST", data=request, headers=self.http_headers
)I'm glad you figured it out.
Changing line 847 from url to self.url fixes the problem for my use case, but am not sure about unintended consequences.
The unintended consequences are in fact significant. What's happening here is that url is in fact extracted from the OFX PROFRS... the flow is to request an OFX profile, extract from it the URL reported to send requests, and use that for the statement request.
There is likely something in here that can be improved for the weird stuff FIdelity is doing.
What do you get back for RqCls2url in line 341 of Client.py?
vitaliknet
commented
2 years ago What do you get back for
RqCls2urlin line 341 ofClient.py?.venv\lib\site-packages\ofxtools\client.py(350)request_statements() -> assert len(urls) == 1 (Pdb) >? p RqCls2url {<class 'ofxtools.Client.InvStmtRq'>: 'https://ofx.fidelity.com/ftgw/OFX/clients/download'} (Pdb) >? p urls {'https://ofx.fidelity.com/ftgw/OFX/clients/download'} (Pdb) >? p self.url 'https://ofx.fidelity.com:443/ftgw/OFX/clients/download' (Pdb)
Some details from Quicken's OFXLOG. After doing initial setup of Fidelity on 11/19, they:
For 11/20 updated, just an INVSTMTTRNRQ to https://ofx.fidelity.com/ftgw/OFX/clients/download
So it looks like they do the same flow as you - request profile, get URL, use it for the statement request.
Generally, it feels like some kind of a breakdown on Fidelity end and not a purposeful change. But also Quicken and Fund Manager, somehow, manage to work around. Similar symptoms have been observed before and went away after a few days. Let's see what ofxhome status check shows next week.
Another thought - I noticed ofxhome reports "SSL failed validation" since November 12th, 2020 for Fidelity. urllib request has a bunch of optional SSL parameters. Maybe Windows-based apps are including something SSL-related in http request (or libraries just do that for them), helping to get through?
csingley
commented
2 years ago It's frustrating that a request to the URL that they publish will fail. However the same URL (without the TCP port#) succeeds for Quicken... and appending the TCP port# fixes it for ofxtools. That is odd, and I don't know quite what to make of it.
SSL issues sounds like a reasonable hypothesis; ofxtools implementation at the transport level could charitably be described as "very basic". My understanding here is kind of weak.
I'm happy to wait a little and hope Fidelity fixes the issue themselves!
mbafford
commented
2 years ago I've found two different ways of making this work, not using @vitaliknet 's approach, but by changing the actual engine making the HTTPS request. This reinforces the idea of the issue being TLS/SSL-related vs. protocol related.
I haven't figured out what the actual issue is, or why my solutions below work - but maybe this will help.
BTW, since the failure happens during the anonymous phase, you can easily test with a fake username/password. 403 Forbidden is the issue at hand, and ValueError: SONRS: Request failed, code=15500, severity=ERROR, message='Error occurred logging in' indicates successfully getting to the authentication step.
I've also confirmed "netbenefits" (also a Fidelity OFX server) fails with the 403 Forbidden error with default code, but also fails with a 200 Access Denied (HTML) response when using Requests.
urllib_request.Request with one with :443 embeddedDoes not work for me.
Forcing to use self.url (with URL overridden to https://ofx.fidelity.com:443/ftgw/OFX/clients/download) as suggested by @vitaliknet does not work for me. I get the 403 Forbidden response.
Does not work for me.
Changing the --version to 102 as suggested by @csingley does not work for me - I get a400 BAD REQUEST response.
Two different approaches did make the requests succeed for me:
mitmproxy intercepting the requestsWorks for me.
Change Client.py to:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
response = url_opener(req, **kwargs)
return BytesIO(response.read())Then launch with an HTTPS_PROXY (mitmproxy):
WORKS:
HTTPS_PROXY="0:9000" .env/bin/ofxget stmt -v fidelity -u "$UN" --write --password "$PW" --allThe above also works with a configured URL of https://ofx.fidelity.com:443/ftgw/OFX/clients/download or https://ofx.fidelity.com/ftgw/OFX/clients/download.
requests libraryWorks for me.
Replacing the following inClient.py with a requests version:
req = urllib_request.Request(
[...]
return BytesIO(response.read())becomes:
import requests
resp = requests.post(url, data=request, headers=self.http_headers)
return BytesIO(resp.content)This then works without a proxy intercepting:
WORKS:
.env/bin/ofxget stmt fidelity -u "$UN" --write --password "$PW" --allUnfortunately, I assume just switching to requests may affect unit tests and/or other integrations.
For example, using Requests doesn't fix netbenefits (changes the error), and it breaks Vanguard. Chase seems to be ok with the change, though.
csingley
commented
2 years ago @mbafford thanks for chiming in. The thing to do is to figure out what mitmproxy and requests are doing right, but I'm doing wrong.
If you've got mitmproxy already set up, would you be able to dump out for me the HTTP/TCP headers, and a recording of the SSL connection setup (handshake)? If so, that would be most helpful. Of course, redact public IP address & any other identifying info.
Thank you.
vitaliknet
commented
2 years ago @mbafford - confirmed that self.url workaround stopped working on Monday and anonymous profile requests are failing for me too, as you observe.
A new detail - I have an environment where my script connects successfully to Fidelity - default installation of ofxtools 0.9.4 & urllib3 1.26.7, and a different public IP from the environment where the same script fails. I do use 102 version of OFX requests just to minimize differences with Quicken and Fund Manager, but most likely this is irrelevant. My troubleshooting is now focused on two hypothesis: 1) IP address blocking (something like anomaly/multiple failures detection and blocking, or restriction of IP ranges - maybe non-US). 2) Software stack below urllib, maybe openssl libraries, CA certificates, etc.
mbafford
commented
2 years ago See the bottom - but I think this comes down to something in the Python build. It works on two linux machines, under docker on macOS, and on the macOS system build of 3.7.4, but not any build I have from pyenv or homebrew.
@csingley I've done some logging with ssldump and mitmproxy, and not seeing any meaningful differences in the output. The problem being that if I use the proxy, then I'm debugging the proxy connection (which works fine), and I don't have Fidelity's certificates to snoop that actual connection.
I've tried both using a direct proxy configuration and using pf to redirect the output to the proxy, thinking maybe there are differences in how urllib sends the request when using a proxy, but I'm not seeing anything obvious.
When going through the proxy, the HTTP headers being sent by ofxclient are pretty much the same between requests and urllib:
requests:
POST /ftgw/OFX/clients/download HTTP/1.1
Host: ofx.fidelity.com
User-Agent: InetClntApp/3.0
Accept-Encoding: gzip, deflate
Accept: */*, application/x-ofx, application/xml;q=0.9
Connection: keep-alive
Content-type: application/x-ofx
Content-Length: 761urllib:
POST /ftgw/OFX/clients/download HTTP/1.1
Accept-Encoding: identity
Content-Length: 761
Host: ofx.fidelity.com:443
User-Agent: InetClntApp/3.0
Content-Type: application/x-ofx
Accept: */*, application/x-ofx, application/xml;q=0.9
Connection: closeHowever, from using curl, I can see that the Fidelity server is fairly unconcerned about most of the request - it can be simplified down to this curl, which works fine (the rest of the headers don't affect anything that I've seen).
curl -v 'https://ofx.fidelity.com/ftgw/OFX/clients/download' \
-H 'Content-Type: application/x-ofx' \
--data '<?xml version="1.0" encoding="UTF-8" standalone="no"?><?OFX OFXHEADER="200" VERSION="220" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="FFFC37CC-7BD6-4E35-843B-95810A63CB2D"?><OFX><SIGNONMSGSRQV1><SONRQ><DTCLIENT>20211125012859.612[+0:UTC]</DTCLIENT><USERID>anonymous00000000000000000000000</USERID><USERPASS>anonymous00000000000000000000000</USERPASS><LANGUAGE>ENG</LANGUAGE><FI><ORG>fidelity.com</ORG><FID>7776</FID></FI><APPID>QWIN</APPID><APPVER>2700</APPVER><CLIENTUID>683C9212-97F9-4008-BBF1-C58FCAEDA267</CLIENTUID></SONRQ></SIGNONMSGSRQV1><PROFMSGSRQV1><PROFTRNRQ><TRNUID>EC0E703C-DA88-4D2D-B0C4-C78D3A498BBC</TRNUID><PROFRQ><CLIENTROUTING>NONE</CLIENTROUTING><DTPROFUP>20031119184217.000[+0:UTC]</DTPROFUP></PROFRQ></PROFTRNRQ></PROFMSGSRQV1></OFX>'@vitaliknet your comment made me try another machine, same IP address, and that machine works fine:
All of the following were done with a fresh venv (python3 -mvenv .env/), and the pip installed ofxtools. All were considered WORKS if I received an OFX output when running ofxtools stmt with authentication, and FAILS for any error before the OFX output.
| Status | Machine | Uname | Python | Ofxtools | OpenSSL |
|---|---|---|---|---|---|
| WORKS | Linux | Linux fridgenas 5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | Python 3.8.10 (Ubuntu / apt-get) | ofxtools 0.9.4 | OpenSSL 1.1.1d 10 Sep 2019 |
| WORKS | raspberry Pi | Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux | Python 3.7.3 (Raspbian / apt-get) | ofxtools 0.8.22 | OpenSSL 1.1.1c 28 May 2019 |
| FAILS | macOS | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 i386 MacBookPro15,1 Darwin | Python 3.9.8 (Homebrew) | ofxtools 0.8.22 | OpenSSL 1.1.1l 24 Aug 2021 |
| WORKS | macOS | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 i386 MacBookPro15,1 Darwin | Python 3.7.3 (macOS system image) | ofxtools 0.8.22 | LibreSSL 2.8.3 |
| FAILS | macOS | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 i386 MacBookPro15,1 Darwin | Python 3.7.4 (pyenv build) | ofxtools 0.8.22 | 'OpenSSL 1.1.1l 24 Aug 2021' |
| FAILS | macOS | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 i386 MacBookPro15,1 Darwin | Python 3.9.8 (Homebrew) | ofxtools 0.9.4 | OpenSSL 1.1.1l 24 Aug 2021 |
| FAILS | macOS | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 i386 MacBookPro15,1 Darwin | Python 3.8.8 (pyenv build) | ofxtools 0.9.4 | OpenSSL 1.1.1l 24 Aug 2021 |
| WORKS | Docker on macOS | Linux 272dc8f06d22 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 x86_64 GNU/Linux | Python 3.8.12 (dockerhub official Python build) | ofxtools 0.9.4 | OpenSSL 1.1.1l 24 Aug 2021 |
| FAILS | macOS | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 i386 MacBookPro15,1 Darwin | Python 3.9.7 (pyenv build) | ofxtools 0.9.4 | OpenSSL 1.1.1l 24 Aug 2021 |
Docker container running on the macOS box (virtualized Linux, of course):
FROM python:3.8-buster
WORKDIR /opt/ofxtools
RUN python3 -mvenv .env/
RUN .env/bin/pip install ofxtools
ENTRYPOINT [".env/bin/ofxget"]docker run --rm ofxtools_test stmt fidelity -u "$UN" --password "$PW" --all@mbafford - it is failing for me in anaconda dist 3.8 and 3.9 on Linux and Windows. Tried upgrading all conda and python packages to latest version but that did not help.
Everything worked just fine on a fresh AWS Lightsail instance with Amazon Linux 2.0 and default python3.8 from amazon-linux-extras. In the end, I built python 3.10 with OpenSSL 1.1.1l from sources and moved my scripts from anaconda to the new 3.10 build. It would be interesting to understand the root cause for academic reasons, but for now I am moving on.
The main lesson here is that I have to find a broker who provides a well-supported API. Fidelity is stuck in early 2000s with what is available through web interface, making it necessary to build external portfolio analysis tools. Which would've been acceptable with official API to download holdings and transactions. As it stands now, they can shutdown ofx interface for everyone except Quicken at any moment and there is nothing individual clients can do. This is unacceptable in 2021 with asset management industry consolidating every day and new competition coming. The days of sleepy money are gone, if they did not figure out that much yet.
dustinfarris
commented
2 years ago Thanks for the awesome writeup @mbafford — I was able to get brokerage and netbenefits working:
macOS 12.0.1 (Monterey)
Python 3.9.9 [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
OpenSSL 3.0.0 7 sep 2021 (Library: OpenSSL 3.0.0 7 sep 2021)
ofxtools 0.8.22Failed attempts:
ofxtools 0.9.4 did not work at allpython:3.8-buster worked for brokerage but not for netbenefitscsingley
commented
2 years ago @mbafford indeed your work here is the dog's bollocks, thanks very much.
Unfortunately my grasp of networking stuff at the application layer is weak. What I can't understand here is this: if the problem is at the SSL layer (failure to negotiate an acceptable cipher, etc.) then why are we seeing an HTTP 403 error? TLS is supposed to encapsulate HTTP. But if there's nothing to discriminate at the HTTP layer, then there's really nowhere else to look.
I have vague memories of changes in SSL protocols causing issues with its Python integration... it isn't necessarily the case that Python is using the system SSL. For a while there around one of these transitions, I just threw up my hands and switched over to using requests because too many users were experiencing problems. I never did get to the bottom of the problem properly. Then after a while the ecosystem stabilized, and everybody's boxes had SSL that would get picked up by Python & play nicely with all the servers... I took out requests again, because I hated the heavyweight dependency for something so trivial. Everything's been just fine for several years.
Well here we are again, apparently. I guess I should probably offer back the requests integration, unless I can scrape out the resources to actually get my arms around the root problem.
It's just kind of an odd situation because to me, ofxtools is library code that has no business integrating web transport dependencies... that's the business of application developers, and I don't think anybody would confuse the trivial utility that is ofxget with any sort of proper application.
Maybe I should split ofxget out into its own separate package? Meh.
mbafford
commented
2 years ago tl;dr - Whatever the issue is, 0.8.22 has a better success rate (according to my test) than 0.9.4 - when considering both fidelity and netbenefits. netbenefits and fidelity behave differently, and netbenefits is even more finicky than fidelity is.
I have no real idea what's going on with the various permutations above and the SSL interactions (if that's even what's going on) (I was hoping someone else would be able to look at my table and go "AHA!").
I don't have a good set of code using requests to fully test that - do you know the last version requests was being used?
For me, ofxget is the main object of interest from ofxtools, and I don't mind the inclusion of requests, although I don't know for sure if it's actually "better" for all permutations.
Here's a full test matrix - this time I used the SSH server docker image from here and installed python3 using apk add python3 so I could use the same test script for all environments.
Test results matrix:
(setup fails are are just due to the installed Python being too old for 0.9.4)
| Type | Host | OFXVer | Setup | ofxget | OpenSSL | PyVER | Uname | PYPath |
| ----------- | -------------- | ------ | ------- | ------- | --------------------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- |
; raspbian repositories
| fidelity | 10.10.70.51:22 | 0.8.22 | SUCCESS | SUCCESS | OpenSSL 1.1.1c 28 May 2019 | Python 3.7.3 | Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux | python3 |
| netbenefits | 10.10.70.51:22 | 0.8.22 | SUCCESS | SUCCESS | OpenSSL 1.1.1c 28 May 2019 | Python 3.7.3 | Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux | python3 |
| fidelity | 10.10.70.51:22 | 0.9.4 | FAILED | | OpenSSL 1.1.1c 28 May 2019 | Python 3.7.3 | Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux | python3 |
| netbenefits | 10.10.70.51:22 | 0.9.4 | FAILED | | OpenSSL 1.1.1c 28 May 2019 | Python 3.7.3 | Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux | python3 |
; Ubuntu repositories
| fidelity | fridgenas:22 | 0.8.22 | SUCCESS | SUCCESS | OpenSSL 1.1.1f 31 Mar 2020 | Python 3.8.10 | Linux fridgenas 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | python3 |
| netbenefits | fridgenas:22 | 0.8.22 | SUCCESS | SUCCESS | OpenSSL 1.1.1f 31 Mar 2020 | Python 3.8.10 | Linux fridgenas 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | python3 |
| netbenefits | fridgenas:22 | 0.9.4 | SUCCESS | FAILED | OpenSSL 1.1.1f 31 Mar 2020 | Python 3.8.10 | Linux fridgenas 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | python3 |
| fidelity | fridgenas:22 | 0.9.4 | SUCCESS | SUCCESS | OpenSSL 1.1.1f 31 Mar 2020 | Python 3.8.10 | Linux fridgenas 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | python3 |
; macOS system
| fidelity | localhost:22 | 0.8.22 | SUCCESS | SUCCESS | LibreSSL 2.8.3 | Python 3.7.3 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /usr/bin/python3 |
| netbenefits | localhost:22 | 0.8.22 | SUCCESS | SUCCESS | LibreSSL 2.8.3 | Python 3.7.3 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /usr/bin/python3 |
| fidelity | localhost:22 | 0.9.4 | FAILED | | LibreSSL 2.8.3 | Python 3.7.3 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /usr/bin/python3 |
| netbenefits | localhost:22 | 0.9.4 | FAILED | | LibreSSL 2.8.3 | Python 3.7.3 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /usr/bin/python3 |
; pyenv
| fidelity | localhost:22 | 0.8.22 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.7.4 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /Users/mbafford/.pyenv/versions/3.7.4/bin/python3 |
| netbenefits | localhost:22 | 0.8.22 | SUCCESS | SUCCESS | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.7.4 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /Users/mbafford/.pyenv/versions/3.7.4/bin/python3 |
| fidelity | localhost:22 | 0.9.4 | FAILED | | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.7.4 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /Users/mbafford/.pyenv/versions/3.7.4/bin/python3 |
| netbenefits | localhost:22 | 0.9.4 | FAILED | | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.7.4 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /Users/mbafford/.pyenv/versions/3.7.4/bin/python3 |
; pyenv
| fidelity | localhost:22 | 0.8.22 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.8.8 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /Users/mbafford/.pyenv/versions/3.8.8/bin/python3 |
| netbenefits | localhost:22 | 0.8.22 | SUCCESS | SUCCESS | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.8.8 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /Users/mbafford/.pyenv/versions/3.8.8/bin/python3 |
| fidelity | localhost:22 | 0.9.4 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.8.8 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /Users/mbafford/.pyenv/versions/3.8.8/bin/python3 |
| netbenefits | localhost:22 | 0.9.4 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.8.8 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /Users/mbafford/.pyenv/versions/3.8.8/bin/python3 |
; pyenv
| fidelity | localhost:22 | 0.8.22 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.9.7 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /Users/mbafford/.pyenv/versions/3.9.7/bin/python3 |
| netbenefits | localhost:22 | 0.8.22 | SUCCESS | SUCCESS | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.9.7 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /Users/mbafford/.pyenv/versions/3.9.7/bin/python3 |
| fidelity | localhost:22 | 0.9.4 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.9.7 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /Users/mbafford/.pyenv/versions/3.9.7/bin/python3 |
| netbenefits | localhost:22 | 0.9.4 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.9.7 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /Users/mbafford/.pyenv/versions/3.9.7/bin/python3 |
; homebrew
| fidelity | localhost:22 | 0.8.22 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.9.8 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /usr/local/bin/python3 |
| netbenefits | localhost:22 | 0.8.22 | SUCCESS | SUCCESS | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.9.8 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /usr/local/bin/python3 |
| fidelity | localhost:22 | 0.9.4 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.9.8 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /usr/local/bin/python3 |
| netbenefits | localhost:22 | 0.9.4 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.9.8 | Darwin marvin.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 | /usr/local/bin/python3 |
; docker
| fidelity | localhost:2222 | 0.8.22 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.9.5 | Linux eb500c9c4c3e 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 x86_64 GNU/Linux | python3 |
| netbenefits | localhost:2222 | 0.8.22 | SUCCESS | SUCCESS | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.9.5 | Linux eb500c9c4c3e 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 x86_64 GNU/Linux | python3 |
| fidelity | localhost:2222 | 0.9.4 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.9.5 | Linux eb500c9c4c3e 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 x86_64 GNU/Linux | python3 |
| netbenefits | localhost:2222 | 0.9.4 | SUCCESS | FAILED | OpenSSL 1.1.1l 24 Aug 2021 | Python 3.9.5 | Linux eb500c9c4c3e 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 x86_64 GNU/Linux | python3 |
Test script:
test script moved to gist https://gist.github.com/mbafford/f14d9648b3694034c2a08d56855abdd7
csingley
commented
2 years ago I don't have a good set of code using requests to fully test that - do you know the last version requests was being used?
I think 0.6.3 was the last such release; 0.7 dropped requests as a dependency.
I'm starting to snap out of my Romilar jag a little bit and recover some of the details. I think some of the damage may have had to do with expired root certificate authorities. IIRC requests depends on certifi specifically to avoid problems caused by stale CAs in the system SSL. In some cases (maybe on Mac Windows), installing Python may have caused certifi to get installed for its own benefit (i.e. some snag in integrating system SSL libs), but then users never knew to update it... so then certifi itself became stale... and then people were puzzled when system SSL was up to date but Python was still failing TLS connections.
Just kind of a shit show all around, and really nothing I have any interest at all in staying on top of.
For me, ofxget is the main object of interest from ofxtools
I suppose I shouldn't be surprised that the world derives the broadest utility from that script... but I don't even use it. I find no great hardship in manually downloading OFX files from FI websites, since I already pretty much have a hard requirement to be in there anyway downloading PDF statements as backup documentation for audit.
The real purpose of ofxtools is to enable rapid development of bespoke financial applications... for those cases when Quicken/Peachtree/GnuCash/etc. just aren't cutting it for you. Needless to say, if you're writing that kind of code, you're not interested in SSL connections.
I suppose it would make a certain amount of sense to carve out ofxget into its own package, but that seems kind of mean to the people who are currently using it.
Maybe a better approach is just to have ofxget introspect whether or not requests is installed in the path, and use it if it is. Then we could hopefully just close bugs like this one simply by advising the afflicted user to install requests, and everybody's happy.
aclindsa
commented
2 years ago @mbafford The seeming arbitrariness of the results of your testing makes me wonder if it is something really silly: like whitespace differences or the ordering of HTTP headers....
For those systems where you've been able to find that ofxtools 0.8.22 works and 0.9.4 does not... one strategy might be to git bisect between the two versions to attempt to pinpoint a particular commit.
mbafford
commented
2 years ago I don't have a good set of code using requests to fully test that - do you know the last version requests was being used?
@csingley I think 0.6.3 was the last such release; 0.7 dropped requests as a dependency.
Confirmed. I've now tested with those two versions (see below).
@aclindsa The seeming arbitrariness of the results of your testing makes me wonder if it is something really silly: like whitespace differences or the ordering of HTTP headers....
My simplified curl tests and proxy tests suggested otherwise. It feels like the TLS negotiation is the culprit.
@aclindsa For those systems where you've been able to find that ofxtools 0.8.22 works and 0.9.4 does not... one strategy might be to git bisect between the two versions to attempt to pinpoint a particular commit.
I rewrote my testing script to use the library directly in Python vs. using ofxget, which made it easier to test multiple versions - the CLI has been less stable than the API.
Testing framework: https://gist.github.com/mbafford/f14d9648b3694034c2a08d56855abdd7
Testing results matrix:
| Type | Success | OFXTools | OpenSSL | PyVer | PyImpl || fidelity | True | 0.6.3 | OpenSSL 1.1.1c 28 May 2019 | 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0] | namespace(_multiarch='arm-linux-gnueabihf', cache_tag='cpython-37', hexversion=50791408, name='cpython', version=sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)) |
| ----------- | ------------------------------------ | ------ | --------------------------- | --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
; raspbian
| netbenefits | True | 0.6.3 | OpenSSL 1.1.1c 28 May 2019 | 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0] | namespace(_multiarch='arm-linux-gnueabihf', cache_tag='cpython-37', hexversion=50791408, name='cpython', version=sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)) |
| fidelity | True | 0.8.0 | OpenSSL 1.1.1c 28 May 2019 | 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0] | namespace(_multiarch='arm-linux-gnueabihf', cache_tag='cpython-37', hexversion=50791408, name='cpython', version=sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)) |
| netbenefits | True | 0.8.0 | OpenSSL 1.1.1c 28 May 2019 | 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0] | namespace(_multiarch='arm-linux-gnueabihf', cache_tag='cpython-37', hexversion=50791408, name='cpython', version=sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)) |
| fidelity | True | 0.8.22 | OpenSSL 1.1.1c 28 May 2019 | 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0] | namespace(_multiarch='arm-linux-gnueabihf', cache_tag='cpython-37', hexversion=50791408, name='cpython', version=sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)) |
| netbenefits | True | 0.8.22 | OpenSSL 1.1.1c 28 May 2019 | 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0] | namespace(_multiarch='arm-linux-gnueabihf', cache_tag='cpython-37', hexversion=50791408, name='cpython', version=sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)) |
; macOS System
| fidelity | True | 0.6.3 | LibreSSL 2.8.3 | 3.7.3 (default, Apr 24 2020, 18:51:23) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791408, name='cpython', version=sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)) |
| netbenefits | True | 0.6.3 | LibreSSL 2.8.3 | 3.7.3 (default, Apr 24 2020, 18:51:23) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791408, name='cpython', version=sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)) |
| fidelity | True | 0.8.0 | LibreSSL 2.8.3 | 3.7.3 (default, Apr 24 2020, 18:51:23) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791408, name='cpython', version=sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)) |
| netbenefits | True | 0.8.0 | LibreSSL 2.8.3 | 3.7.3 (default, Apr 24 2020, 18:51:23) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791408, name='cpython', version=sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)) |
| fidelity | True | 0.8.22 | LibreSSL 2.8.3 | 3.7.3 (default, Apr 24 2020, 18:51:23) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791408, name='cpython', version=sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)) |
| netbenefits | True | 0.8.22 | LibreSSL 2.8.3 | 3.7.3 (default, Apr 24 2020, 18:51:23) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791408, name='cpython', version=sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)) |
; pyenv
| fidelity | True | 0.6.3 | OpenSSL 1.1.1l 24 Aug 2021 | 3.7.4 (default, Dec 5 2019, 08:32:43) [Clang 10.0.1 (clang-1001.0.46.4)] | namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791664, name='cpython', version=sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)) |
| netbenefits | True | 0.6.3 | OpenSSL 1.1.1l 24 Aug 2021 | 3.7.4 (default, Dec 5 2019, 08:32:43) [Clang 10.0.1 (clang-1001.0.46.4)] | namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791664, name='cpython', version=sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)) |
| fidelity | True | 0.8.0 | OpenSSL 1.1.1l 24 Aug 2021 | 3.7.4 (default, Dec 5 2019, 08:32:43) [Clang 10.0.1 (clang-1001.0.46.4)] | namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791664, name='cpython', version=sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)) |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.8.0 | OpenSSL 1.1.1l 24 Aug 2021 | 3.7.4 (default, Dec 5 2019, 08:32:43) [Clang 10.0.1 (clang-1001.0.46.4)] | namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791664, name='cpython', version=sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)) |
| fidelity | True | 0.8.22 | OpenSSL 1.1.1l 24 Aug 2021 | 3.7.4 (default, Dec 5 2019, 08:32:43) [Clang 10.0.1 (clang-1001.0.46.4)] | namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791664, name='cpython', version=sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)) |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.8.22 | OpenSSL 1.1.1l 24 Aug 2021 | 3.7.4 (default, Dec 5 2019, 08:32:43) [Clang 10.0.1 (clang-1001.0.46.4)] | namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791664, name='cpython', version=sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)) |
; pyenv
| fidelity | True | 0.6.3 | OpenSSL 1.1.1l 24 Aug 2021 | 3.8.8 (default, Jun 18 2021, 20:43:12) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-38', hexversion=50858224, name='cpython', version=sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)) |
| netbenefits | True | 0.6.3 | OpenSSL 1.1.1l 24 Aug 2021 | 3.8.8 (default, Jun 18 2021, 20:43:12) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-38', hexversion=50858224, name='cpython', version=sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)) |
| fidelity | True | 0.8.0 | OpenSSL 1.1.1l 24 Aug 2021 | 3.8.8 (default, Jun 18 2021, 20:43:12) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-38', hexversion=50858224, name='cpython', version=sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)) |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.8.0 | OpenSSL 1.1.1l 24 Aug 2021 | 3.8.8 (default, Jun 18 2021, 20:43:12) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-38', hexversion=50858224, name='cpython', version=sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)) |
| fidelity | True | 0.8.22 | OpenSSL 1.1.1l 24 Aug 2021 | 3.8.8 (default, Jun 18 2021, 20:43:12) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-38', hexversion=50858224, name='cpython', version=sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)) |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.8.22 | OpenSSL 1.1.1l 24 Aug 2021 | 3.8.8 (default, Jun 18 2021, 20:43:12) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-38', hexversion=50858224, name='cpython', version=sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)) |
| fidelity | True | 0.9 | OpenSSL 1.1.1l 24 Aug 2021 | 3.8.8 (default, Jun 18 2021, 20:43:12) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-38', hexversion=50858224, name='cpython', version=sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)) |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.9 | OpenSSL 1.1.1l 24 Aug 2021 | 3.8.8 (default, Jun 18 2021, 20:43:12) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-38', hexversion=50858224, name='cpython', version=sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)) |
| fidelity | EXCEPTION: HTTP Error 403: Forbidden | 0.9.4 | OpenSSL 1.1.1l 24 Aug 2021 | 3.8.8 (default, Jun 18 2021, 20:43:12) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-38', hexversion=50858224, name='cpython', version=sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)) |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.9.4 | OpenSSL 1.1.1l 24 Aug 2021 | 3.8.8 (default, Jun 18 2021, 20:43:12) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(_multiarch='darwin', cache_tag='cpython-38', hexversion=50858224, name='cpython', version=sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)) |
; Ubuntu
| fidelity | True | 0.6.3 | OpenSSL 1.1.1f 31 Mar 2020 | 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] | namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50858736, name='cpython', version=sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)) |
| netbenefits | True | 0.6.3 | OpenSSL 1.1.1f 31 Mar 2020 | 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] | namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50858736, name='cpython', version=sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)) |
| fidelity | True | 0.8.0 | OpenSSL 1.1.1f 31 Mar 2020 | 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] | namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50858736, name='cpython', version=sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)) |
| netbenefits | True | 0.8.0 | OpenSSL 1.1.1f 31 Mar 2020 | 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] | namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50858736, name='cpython', version=sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)) |
| fidelity | True | 0.8.22 | OpenSSL 1.1.1f 31 Mar 2020 | 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] | namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50858736, name='cpython', version=sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)) |
| netbenefits | True | 0.8.22 | OpenSSL 1.1.1f 31 Mar 2020 | 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] | namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50858736, name='cpython', version=sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)) |
| fidelity | True | 0.9 | OpenSSL 1.1.1f 31 Mar 2020 | 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] | namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50858736, name='cpython', version=sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)) |
| netbenefits | True | 0.9 | OpenSSL 1.1.1f 31 Mar 2020 | 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] | namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50858736, name='cpython', version=sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)) |
| fidelity | EXCEPTION: HTTP Error 403: Forbidden | 0.9.4 | OpenSSL 1.1.1f 31 Mar 2020 | 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] | namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50858736, name='cpython', version=sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)) |
| netbenefits | True | 0.9.4 | OpenSSL 1.1.1f 31 Mar 2020 | 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] | namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50858736, name='cpython', version=sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)) |
; pyenv
| fidelity | True | 0.6.3 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.7 (default, Nov 25 2021, 12:28:30) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0), hexversion=50923504, _multiarch='darwin') |
| netbenefits | True | 0.6.3 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.7 (default, Nov 25 2021, 12:28:30) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0), hexversion=50923504, _multiarch='darwin') |
| fidelity | True | 0.8.0 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.7 (default, Nov 25 2021, 12:28:30) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0), hexversion=50923504, _multiarch='darwin') |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.8.0 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.7 (default, Nov 25 2021, 12:28:30) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0), hexversion=50923504, _multiarch='darwin') |
| fidelity | True | 0.8.22 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.7 (default, Nov 25 2021, 12:28:30) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0), hexversion=50923504, _multiarch='darwin') |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.8.22 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.7 (default, Nov 25 2021, 12:28:30) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0), hexversion=50923504, _multiarch='darwin') |
| fidelity | True | 0.9 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.7 (default, Nov 25 2021, 12:28:30) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0), hexversion=50923504, _multiarch='darwin') |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.9 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.7 (default, Nov 25 2021, 12:28:30) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0), hexversion=50923504, _multiarch='darwin') |
| fidelity | EXCEPTION: HTTP Error 403: Forbidden | 0.9.4 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.7 (default, Nov 25 2021, 12:28:30) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0), hexversion=50923504, _multiarch='darwin') |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.9.4 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.7 (default, Nov 25 2021, 12:28:30) [Clang 11.0.3 (clang-1103.0.32.62)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0), hexversion=50923504, _multiarch='darwin') |
; pyenv
| fidelity | True | 0.6.3 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.8 (main, Nov 10 2021, 06:03:50) [Clang 12.0.0 (clang-1200.0.32.29)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=8, releaselevel='final', serial=0), hexversion=50923760, _multiarch='darwin') |
| netbenefits | True | 0.6.3 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.8 (main, Nov 10 2021, 06:03:50) [Clang 12.0.0 (clang-1200.0.32.29)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=8, releaselevel='final', serial=0), hexversion=50923760, _multiarch='darwin') |
| fidelity | True | 0.8.0 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.8 (main, Nov 10 2021, 06:03:50) [Clang 12.0.0 (clang-1200.0.32.29)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=8, releaselevel='final', serial=0), hexversion=50923760, _multiarch='darwin') |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.8.0 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.8 (main, Nov 10 2021, 06:03:50) [Clang 12.0.0 (clang-1200.0.32.29)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=8, releaselevel='final', serial=0), hexversion=50923760, _multiarch='darwin') |
| fidelity | True | 0.8.22 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.8 (main, Nov 10 2021, 06:03:50) [Clang 12.0.0 (clang-1200.0.32.29)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=8, releaselevel='final', serial=0), hexversion=50923760, _multiarch='darwin') |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.8.22 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.8 (main, Nov 10 2021, 06:03:50) [Clang 12.0.0 (clang-1200.0.32.29)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=8, releaselevel='final', serial=0), hexversion=50923760, _multiarch='darwin') |
| fidelity | True | 0.9 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.8 (main, Nov 10 2021, 06:03:50) [Clang 12.0.0 (clang-1200.0.32.29)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=8, releaselevel='final', serial=0), hexversion=50923760, _multiarch='darwin') |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.9 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.8 (main, Nov 10 2021, 06:03:50) [Clang 12.0.0 (clang-1200.0.32.29)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=8, releaselevel='final', serial=0), hexversion=50923760, _multiarch='darwin') |
| fidelity | EXCEPTION: HTTP Error 403: Forbidden | 0.9.4 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.8 (main, Nov 10 2021, 06:03:50) [Clang 12.0.0 (clang-1200.0.32.29)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=8, releaselevel='final', serial=0), hexversion=50923760, _multiarch='darwin') |
| netbenefits | EXCEPTION: HTTP Error 403: Forbidden | 0.9.4 | OpenSSL 1.1.1l 24 Aug 2021 | 3.9.8 (main, Nov 10 2021, 06:03:50) [Clang 12.0.0 (clang-1200.0.32.29)] | namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=8, releaselevel='final', serial=0), hexversion=50923760, _multiarch='darwin') |
I really have no idea what's a valuable use of time in tracking this down further. It does seem like 0.6.3 is the most reliable - which is also the last version to use requests. So I guess this proves there could be some value in switching to requests (or offering it as an optional dependency).
csingley
commented
2 years ago Thanks again @mbafford, confirming my suspicions.
I really have no idea what's a valuable use of time in tracking this down further.
Some fairly low-level SSL debugging I'm afraid... largely above my pay grade and beyond my interest at present.
So I guess this proves there could be some value in switching to requests (or offering it as an optional dependency).
That's what I plan to do, rather than dive in & reinvent this wheel. Just need to eke out some bandwidth to do the work. It was pretty trivial the last time I did this; just need to make sure requests plays nice with the new HTTP cookie jar code for the service URL lookups.
patbakdev
commented
2 years ago FYI. I was switching from using ofxget to using the library directly and I noticed that I can now download from Fidelity, but not Fidelity Net Benefits (still 403).
My code is roughly as follows:
client = OFXClient(
url='https://ofx.fidelity.com/ftgw/OFX/clients/download',
userid='redacted',
clientuid='GUIDUPPERCASENODASHES',
org='fidelity.com',
fid='7776',
version=220,
brokerid='fidelity.com')Presumably using defaults:
I tried ofxget scan fidelity, but it hung. Maybe something changed on Fidelity's side?
csingley
commented
2 years ago Why don't y'all install requests, upgrade to 863fb93, and let me know how that works for you?
aclindsa
commented
2 years ago Why don't y'all install requests, upgrade to 863fb93, and let me know how that works for you?
Works like a charm for me - thanks! And for what it's worth, I'm testing with a configuration I have otherwise been unable to fetch my non-netbenefits Fidelity stuff for quite a while.
mbafford
commented
2 years ago Why don't y'all install requests, upgrade to 863fb93, and let me know how that works for you?
FYI to anyone trying this, requests isn't added as a dependency - the code checks at runtime to see if it should use it, so make sure you also install requests in your (virtual)environment before doing the test.
Tests with [863fb93] with MacOS, PyVer installed Python 3.9.5
| Requests Installed | Bank | Status |
|---|---|---|
| no | Fidelity (Brokerage) | Failed |
| no | Fidelity (Net Benefits) | Failed |
| no | Vanguard (Brokerage) | Success |
| no | Chase (Credit Card) | Success |
| yes | Fidelity (Brokerage) | Success |
| yes | Fidelity (Net Benefits) | Failed |
| yes | Vanguard (Brokerage) | Success |
| yes | Chase (Credit Card) | Success |
Fidelity netbenefits is still failing for me with:
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
You don't have permission to access "http://nbofx.fidelity.com/netbenefits/ofx/download" on this server.<P>
Reference #18.3f00c045.1641475226.bf70f49
</BODY>
</HTML>Since the last time I posted / tested, I've been successfully running MacOS built-in Python 3.7.3 on ofxtools==0.8.22 with all of the above banks.
I can't test this new patch on that version, since ofxtools now requires Python >=3.8
csingley
commented
2 years ago I've been successfully running MacOS built-in Python 3.7.3 on ofxtools==0.8.22 with all of the above banks.
Including netbenefits? Is there anything old ofxtools can do that can't be done with HEAD?
I want Py3.8 mainly so I can use functools.singledispatchmethod in ofxtools.Types. That's way better than the mess in 0.8.22.
Is it difficult to install & run non-system Python on Mac? I'm supposed to be getting my hands on one soon.
mbafford
commented
2 years ago I've been successfully running MacOS built-in Python 3.7.3 on ofxtools==0.8.22 with all of the above banks.
Including netbenefits?
Correct. On my mac, the only one which can download Fidelity netbenefits is 0.8.22 with the system Python.
Is it difficult to install & run non-system Python on Mac? I'm supposed to be getting my hands on one soon.
Not at all.
A lot of people use homebrew, which can do it for you.
I prefer to use pyenv, because homebrew has a nasty habit of upgrading Python when I install something unrelated and breaking my virtual environments (of which I use a lot). So I use pyenv to make sure older versions don't randomly disappear.
Is there anything old ofxtools can do that can't be done with HEAD?
For me it looks like the only novel thing old ofxtools does compared to HEAD is reliably download from Fidelity netbenefits.
Oh, and not allow password on the command line. 😬
@aclindsa - when you say HEAD works for you, do you mean just regular fidelity, or both regular fidelity and fidelity netbenefits?
csingley
commented
2 years ago Correct. On my mac, the only one which can download Fidelity netbenefits is 0.8.22 with the system Python.
When you get a chance, would you do me a favor and run ofxget --dryrun for the identical netbenefits config under 0.8.22 vs ofxtools HEAD(with requests installed), then inspect the delta?
Can we narrow the problem space to the transport layer? This bug hasn't got many corners left to hide in.
patbakdev
commented
2 years ago I am also seeing the same issue with Fidleity Net Benefits, but I also get "ofxtools.header.OFXHeaderError: OFX header is malformed". Sounds like this might just be a bad response after the actual error, but I thought I would throw it in anyways.
Running ofxget scan netbenefits -vv gives a lot of info, but ends with
Traceback (most recent call last):
File "/home/patrick/.local/share/pyenv/versions/Venti/bin/ofxget", line 33, in <module>
sys.exit(load_entry_point('ofxtools', 'console_scripts', 'ofxget')())
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/scripts/ofxget.py", line 1614, in main
REQUEST_HANDLERS[args["request"]](args)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/scripts/ofxget.py", line 499, in scan_profile
scan_results = _scan_profile(
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/scripts/ofxget.py", line 1188, in _scan_profile
valid, signoninfo_ = _read_scan_response(future, not signoninfo)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/scripts/ofxget.py", line 1264, in _read_scan_response
response = future.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 438, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 390, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Client.py", line 514, in request_profile
parser.parse(response)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Parser.py", line 82, in parse
self.header, message = self._read(source)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Parser.py", line 115, in _read
header, message = parse_header(source)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/header.py", line 294, in parse_header
header, header_end_index = OFXHeaderV1.parse(rawheader)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/header.py", line 81, in parse
raise OFXHeaderError(f"OFX header is malformed:\n{rawheader}")
ofxtools.header.OFXHeaderError: OFX header is malformed:
<HTML><HEAD><TITLE>Bad request</TITLE></HEAD><BODY>The element type "SONRQ" must be terminated by the matching end-tag "</SONRQ>".</BODY></HTML>I also get "ofxtools.header.OFXHeaderError: OFX header is malformed"
Well sure, the parser is not set up to parse random HTML, which is what Fidelity is sending you in lieu of an OFX error.
The element type "SONRQ" must be terminated by the matching end-tag ""
This is why I'm soliciting from you guys the structure of OFX requests generated by ofxtools.Client.OFXClient that are being sent to Fido.
If this error message is accurate, it indicates a pretty whopping bug in the OFXClient.
patbakdev
commented
2 years ago Ok, got it. Not sure which subcommand to use, but I started with prof since scan didn't seem to have a dryrun
commit 7e8ac02163cc12cf8bbac41ec100f427be79034a (tag: 0.8.22)
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?OFX OFXHEADER="200" VERSION="220" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="be5f329b-0698-440a-a67d-0500d028b620"?>
<OFX><SIGNONMSGSRQV1><SONRQ><DTCLIENT>20220106195504.902[0:GMT]</DTCLIENT><USERID>anonymous00000000000000000000000</USERID><USERPASS>anonymous00000000000000000000000</USERPASS><LANGUAGE>ENG</LANGUAGE><FI><ORG>nbofx.fidelity.com</ORG><FID>8288</FID></FI><APPID>QWIN</APPID><APPVER>2700</APPVER></SONRQ></SIGNONMSGSRQV1><PROFMSGSRQV1><PROFTRNRQ><TRNUID>a9af9aeb-bd0e-47bc-8a00-620282ad145e</TRNUID><PROFRQ><CLIENTROUTING>NONE</CLIENTROUTING><DTPROFUP>19900101000000.000[0:GMT]</DTPROFUP></PROFRQ></PROFTRNRQ></PROFMSGSRQV1></OFX>HEAD
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?OFX OFXHEADER="200" VERSION="220" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="640591DC-5592-4FAE-9928-7EA0D952BA95"?>
<OFX><SIGNONMSGSRQV1><SONRQ><DTCLIENT>20220106195214.192[+0:UTC]</DTCLIENT><USERID>anonymous00000000000000000000000</USERID><USERPASS>anonymous00000000000000000000000</USERPASS><LANGUAGE>ENG</LANGUAGE><FI><ORG>nbofx.fidelity.com</ORG><FID>8288</FID></FI><APPID>QWIN</APPID><APPVER>2700</APPVER></SONRQ></SIGNONMSGSRQV1><PROFMSGSRQV1><PROFTRNRQ><TRNUID>E611F12F-39F1-419E-B533-89519ED26F2D</TRNUID><PROFRQ><CLIENTROUTING>NONE</CLIENTROUTING><DTPROFUP>20030820223030.000[+0:UTC]</DTPROFUP></PROFRQ></PROFTRNRQ></PROFMSGSRQV1></OFX>Main difference seems to be capitalization of guids and timezone changes.
csingley
commented
2 years ago Well a PROFRQ won't work here, because it doesn't do a sign-on. You need to match whatever you were doing that Fido was complaining about... presumably something incorporating a SONRQ, e.g. a STMTRQ.
Of course, you should redact any personal information from the dry-run dump.... acct#, username, what have you.
Note that you are using OFXv2, which is unusual. That is not really the right way to stand up 3 monkeys in a trenchcoat and pass them off as Quicken.
patbakdev
commented
2 years ago So I tried ofxget stmt netbenefits -u USER --all with 0.8.22 and got back valid xml with good data. When I reset to HEAD and tried the same command I get the malformed OFX Header. So I am not sure what to diff here. It seems like for whatever reason, I am getting denied access and the response it is sending back is just plain HTML which is of course not correct xml.
Traceback (most recent call last):
File "/home/patrick/.local/share/pyenv/versions/Venti/bin/ofxget", line 33, in <module>
sys.exit(load_entry_point('ofxtools', 'console_scripts', 'ofxget')())
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/scripts/ofxget.py", line 1614, in main
REQUEST_HANDLERS[args["request"]](args)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/scripts/ofxget.py", line 665, in request_stmt
acctinfo = _request_acctinfo(args, password)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/scripts/ofxget.py", line 622, in _request_acctinfo
with client.request_accounts(
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Client.py", line 600, in request_accounts
RqCls2url = self._get_service_urls(
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Client.py", line 427, in _get_service_urls
profile = self.request_profile(
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Client.py", line 514, in request_profile
parser.parse(response)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Parser.py", line 82, in parse
self.header, message = self._read(source)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Parser.py", line 115, in _read
header, message = parse_header(source)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/header.py", line 294, in parse_header
header, header_end_index = OFXHeaderV1.parse(rawheader)
File "/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/header.py", line 81, in parse
raise OFXHeaderError(f"OFX header is malformed:\n{rawheader}")
ofxtools.header.OFXHeaderError: OFX header is malformed:
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
You don't have permission to access "http://nbofx.fidelity.com/netbenefits/ofx/download" on this server.<P>
Reference #18.1533dd17.1641500029.48085918
</BODY>
</HTML>As to OFXv2 usage, if I try ofxget stmt netbenefits -u PatrickCheriB --all --version 1 then I get: ofxtools.header.OFXHeaderError: OFX version 1 not version 1 or version 2
patbakdev
commented
2 years ago Another thought. When I reverted to 0.8.22 and tried to run my code (not ofxget) I noticed that OFClient was missing an argument for useragent. Presumably added later. Maybe the default useragent string is different?
csingley
commented
2 years ago What's really needed here is a diff of the (successful) request sent by 0.8.22 vs. the (failed) request sent by HEAD. That will allow us to fix the breakage at the application protocol layer, or else rule it out & focus on the transport layer.
As to OFXv2 usage, if I try ofxget stmt netbenefits -u PatrickCheriB --all --version 1 then I get: ofxtools.header.OFXHeaderError: OFX version 1 not version 1 or version 2
Try --version=103, which is what Quicken uses (i.e. OFXv1.0.3)
patbakdev
commented
2 years ago This is weird. Using ofxget stmt netbenefits -u USER --all with 0.8.22 I get:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?OFX OFXHEADER="200" VERSION="220" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="db6d779c-bd06-477f-9014-01d8bdc6a00b"?>
<OFX><SIGNONMSGSRQV1><SONRQ><DTCLIENT>20220106222241.366[0:GMT]</DTCLIENT><USERID>REDACT</USERID><USERPASS>REDACT</USERPASS><LANGUAGE>ENG</LANGUAGE><FI><ORG>nbofx.fidelity.com</ORG><FID>8288</FID></FI><APPID>QWIN</APPID><APPVER>2700</APPVER></SONRQ></SIGNONMSGSRQV1><INVSTMTMSGSRQV1><INVSTMTTRNRQ><TRNUID>6fbdf30e-cdd8-4b73-8d86-9548cacd6aa5</TRNUID><INVSTMTRQ><INVACCTFROM><BROKERID>nbofx.fidelity.com</BROKERID><ACCTID>REDACT</ACCTID></INVACCTFROM><INCTRAN><INCLUDE>Y</INCLUDE></INCTRAN><INCOO>N</INCOO><INCPOS><INCLUDE>Y</INCLUDE></INCPOS><INCBAL>Y</INCBAL></INVSTMTRQ></INVSTMTTRNRQ><INVSTMTTRNRQ><TRNUID>93125b14-bfc1-4018-9bf2-757718cffe76</TRNUID><INVSTMTRQ><INVACCTFROM><BROKERID>nbofx.fidelity.com</BROKERID><ACCTID>REDACT</ACCTID></INVACCTFROM><INCTRAN><INCLUDE>Y</INCLUDE></INCTRAN><INCOO>N</INCOO><INCPOS><INCLUDE>Y</INCLUDE></INCPOS><INCBAL>Y</INCBAL></INVSTMTRQ></INVSTMTTRNRQ></INVSTMTMSGSRQV1></OFX>But with HEAD I get:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?OFX OFXHEADER="200" VERSION="220" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="C3E8D8EF-65BC-4167-9941-97A85FD759C9"?>
<OFX><SIGNONMSGSRQV1><SONRQ><DTCLIENT>20220106222514.777[+0:UTC]</DTCLIENT><USERID>anonymous00000000000000000000000</USERID><USERPASS>anonymous00000000000000000000000</USERPASS><LANGUAGE>ENG</LANGUAGE><FI><ORG>nbofx.fidelity.com</ORG><FID>8288</FID></FI><APPID>QWIN</APPID><APPVER>2700</APPVER></SONRQ></SIGNONMSGSRQV1><PROFMSGSRQV1><PROFTRNRQ><TRNUID>B6804F73-63AC-44A5-A9EB-3ED52D413D4C</TRNUID><PROFRQ><CLIENTROUTING>NONE</CLIENTROUTING><DTPROFUP>20030820223030.000[+0:UTC]</DTPROFUP></PROFRQ></PROFTRNRQ></PROFMSGSRQV1></OFX>Its not setting USER and PASS.
I got this by putting print(request.decode()) before the response = self.post_request(url, request, timeout) in self.download.
I am now trying to debug and see why the user is not set.
csingley
commented
2 years ago You can just throw a --dryrun arg on the ofxget call; you shouldn't need to patch the source to get debug output.
The first one is an INVSTMTRQ (i.e. a request to download an OFX investment statement, which requires authentication) and the second is a PROFRQ (i.e. a request to get an OFX profile - no login required)
HEAD is doing the routine of sending a PROFRQ to get the service URL for statement requests, then sending the STMTRQ there. 0.8.22 just sent the STMTRQ directly.
Need to figure out if Fido doesn't like the PROFRQ, or it doesn't like the INVSTMTRQ.
You know, there's language in the spec about requiring USERID/USERPASS for all subsequent PROFRQ after enrollment has been established. We don't do that at all; we always send anonymous logins for profile requests. Perhaps the netbenefits endpoint is strict about this?
patbakdev
commented
2 years ago It does seem like its throwing while attempting to download the profile.
Debugging to here:
/home/patrick/.local/share/pyenv/versions/Venti/bin/ofxget(33)<module>()
-> sys.exit(load_entry_point('ofxtools', 'console_scripts', 'ofxget')())
/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/scripts/ofxget.py(1615)main()
-> REQUEST_HANDLERS[args["request"]](args)
/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/scripts/ofxget.py(666)request_stmt()
-> acctinfo = _request_acctinfo(args, password)
/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/scripts/ofxget.py(623)_request_acctinfo()
-> with client.request_accounts(
/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Client.py(601)request_accounts()
-> RqCls2url = self._get_service_urls(
/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Client.py(427)_get_service_urls()
-> profile = self.request_profile(
/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Client.py(500)request_profile()
-> response = self._request_profile(
/home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Client.py(575)_request_profile()
-> return self.download(
> /home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/Client.py(857)download()
-> return BytesIO(response)The request object is:
(Pdb) p request
b'<?xml version="1.0" encoding="UTF-8" standalone="no"?>\r\n<?OFX OFXHEADER="200" VERSION="220" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="0F1AD396-AFC5-42F3-A2B3-0CB731B38F0E"?>\r\n<OFX><SIGNONMSGSRQV1><SONRQ><DTCLIENT>20220106231924.490[+0:UTC]</DTCLIENT><USERID>anonymous00000000000000000000000</USERID><USERPASS>anonymous00000000000000000000000</USERPASS><LANGUAGE>ENG</LANGUAGE><FI><ORG>nbofx.fidelity.com</ORG><FID>8288</FID></FI><APPID>QWIN</APPID><APPVER>2700</APPVER></SONRQ></SIGNONMSGSRQV1><PROFMSGSRQV1><PROFTRNRQ><TRNUID>6BDCB765-3646-45EE-92A3-B53F6AB35144</TRNUID><PROFRQ><CLIENTROUTING>NONE</CLIENTROUTING><DTPROFUP>20030820223030.000[+0:UTC]</DTPROFUP></PROFRQ></PROFTRNRQ></PROFMSGSRQV1></OFX>'and the response object is
(Pdb) p BytesIO(response).read()
b'<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don\'t have permission to access "http://nbofx.fidelity.com/netbenefits/ofx/download" on this server.<P>\nReference #18.933dd17.1641511295.2426fb06\n</BODY>\n</HTML>\n'And from what I can tell it doesn't look like we call request_profile in 0.8.22.
Edit: Right, you already mentioned this. But this probably explains why it worked in 0.8.22 and not now.
Edit2: Unfortunately hard coding my credentials into the request_profile call didn't help. I confirmed the request contained the correct USERID and USERPASS fields, but I still got the Access Denied response.
csingley
commented
2 years ago What do you get for ofxget stmt --dryrun netbenefits for 0.8.22 vs. HEAD? That really is the right place to begin.
patbakdev
commented
2 years ago HEAD
❯ ofxget stmt --dryrun netbenefits
py.warnings [WARNING] - /home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/scripts/ofxget.py:717: SyntaxWarning: No accounts specified; configure at least one of ['checking', 'savings', 'moneymrkt', 'creditline', 'creditcard', 'investment']
warnings.warn(msg, category=SyntaxWarning)
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?OFX OFXHEADER="200" VERSION="220" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="3676130B-356C-40FF-A40F-8A7A066EA988"?>
<OFX><SIGNONMSGSRQV1><SONRQ><DTCLIENT>20220106235834.230[+0:UTC]</DTCLIENT><USERID>anonymous00000000000000000000000</USERID><USERPASS>anonymous00000000000000000000000</USERPASS><LANGUAGE>ENG</LANGUAGE><FI><ORG>nbofx.fidelity.com</ORG><FID>8288</FID></FI><APPID>QWIN</APPID><APPVER>2700</APPVER></SONRQ></SIGNONMSGSRQV1></OFX>0.8.22
py.warnings [WARNING] - /home/patrick/Projects/Venti/src/downloaders/ofxtools/ofxtools/scripts/ofxget.py:687: SyntaxWarning: No accounts specified; configure at least one of ['checking', 'savings', 'moneymrkt', 'creditline', 'creditcard', 'investment']
warnings.warn(msg, category=SyntaxWarning)
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?OFX OFXHEADER="200" VERSION="220" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="8808ed0b-d6ed-4702-a5b0-f35930010313"?>
<OFX><SIGNONMSGSRQV1><SONRQ><DTCLIENT>20220106235936.834[0:GMT]</DTCLIENT><USERID>anonymous00000000000000000000000</USERID><USERPASS>anonymous00000000000000000000000</USERPASS><LANGUAGE>ENG</LANGUAGE><FI><ORG>nbofx.fidelity.com</ORG><FID>8288</FID></FI><APPID>QWIN</APPID><APPVER>2700</APPVER></SONRQ></SIGNONMSGSRQV1></OFX>Same accept for timezone and guid capitalization.
csingley
commented
2 years ago OK thanks.
Now take the dryrun dump from 0.8.22, manually edit it to include your login credentials, and post it the server URL using curl.
If you don't know the server URL, you can get it via ofxget list netbenefits.
If that succeeds with the 0.8.22 dump, try the same procedure with the HEAD dump and verify that it fails. If it does indeed fail, edit the DTCLIENT timezone to look like the 0.8.22 version, then post that via curl. Success/failure?
If you're not used to working with curl just let me know; I could provide detailed instructions.
Thanks! I don't have logins at Fidelity, or I'd do it myself. With help, we'll lick this yet.
patbakdev
commented
2 years ago Thanks for your patience. Not super familiar with curl, but I think got it right.
❯ curl -X POST https://nbofx.fidelity.com/netbenefits/ofx/download -H "Content-Type: application/xml" -H "Accept: application/xml" -d '<?OFX OFXHEADER="200" VERSION="220" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="8808ed0b-d6ed-4702-a5b0-f35930010313"?> <OFX><SIGNONMSGSRQV1><SONRQ><DTCLIENT>20220106235936.834[0:GMT]</DTCLIENT><USERID>anonymous00000000000000000000000</USERID><USERPASS>anonymous00000000000000000000000</USERPASS><LANGUAGE>ENG</LANGUAGE><FI><ORG>nbofx.fidelity.com</ORG><FID>8288</FID></FI><APPID>QWIN</APPID><APPVER>2700</APPVER></SONRQ></SIGNONMSGSRQV1></OFX>'I got pretty much identical success messages across 0.8.22/HEAD, anonymous/not anonymous, UTC/GMT, 0/+0, and lowercase/uppercase GUID.
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <?OFX OFXHEADER="200" VERSION="220" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="3676130B-356C-40FF-A40F-8A7A066EA988"?><OFX><SIGNONMSGSRSV1> <SONRS> <STATUS><CODE>0</CODE><SEVERITY>INFO</SEVERITY><MESSAGE>SUCCESS</MESSAGE></STATUS><DTSERVER>20220106195422.715[-5:EST]</DTSERVER><LANGUAGE>ENG</LANGUAGE><FI><ORG>nbofx.fidelity.com</ORG><FID>8288</FID></FI></SONRS></SIGNONMSGSRSV1></OFX>% Its getting late, maybe I am missing something. I'll start fresh tomorrow.
csingley
commented
2 years ago I got pretty much identical success messages across 0.8.22/HEAD, anonymous/not anonymous, UTC/GMT, 0/+0, and lowercase/uppercase GUID.
Great, solid result. That means it's not the STMTRQ formatting.
I'll add an option to the OFXClient (and a corresponding CLI arg to ofxget) offering the option to skip the profile lookup and jump straight to making the request itself.
If that doesn't fix the issue, then the issue isn't at the application protocol layer; it's at the transport layer.
Hang tight, I need to do a little work on this.
csingley
commented
2 years ago 2efde5b adds a skip_profile arg to the various OFXClient.request_*() methods, and a corresponding switch in the ofxget script.
Give it a shot, something like ofxget stmt --skipprofile netbenefits, and see if it works. You'll want to have requests installed for this I imagine, since that's what worked for 0.8.22 (and also HEAD for vanilla Fidelity OFX connections).
mbafford
commented
2 years ago @csingley I can't spend a lot of time on this today, unfortunately, but I think some knowledge got forgotten from earlier in the thread:
curl as well, and they worked fineWith 0.8.22 on MacOS System pPython (the one that works), you get a proper error message with a bad username/password:
$ ./ofxget stmt netbenefits -u "err" --write --password "err" --al
ofxtools.scripts.ofxget [ERROR] - SONRS: Request failed, code=15500, severity=ERROR, message='Error occurred logging in'
Traceback (most recent call last):
ValueError: SONRS: Request failed, code=15500, severity=ERROR, message='Error occurred logging in'With HEAD on homebrew Python (which doesn't work for me), you get the HTML error message:
$ ./ofxget stmt netbenefits -u "err" --write --password "err" --all
Using requests <--- debug logging I added to make sure it was following that code path
ofxtools.header.OFXHeaderError: OFX header is malformed:
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
You don't have permission to access "http://nbofx.fidelity.com/netbenefits/ofx/download" on this server.<P>
Reference #18.4d6adc17.1641564952.9af99962
</BODY>
</HTML>Thanks @mbafford. Between year-end crunch and a strenuous regimen of methedrine I had indeed lost the plot a bit.
- latest version of ofxtools works fine if I MITM the requests through Charles or mitmproxy, so it's not the syntax of the request, or really anything - I can just take the requests your library is making and pull them into curl as well, and they worked fine
So it's not the application protocol layer, and it's not some sort of blacklisting (by IP or account).
That makes it transport layer. But not TLS-related.
- you can test this with a bad username/password, the behavior is different between the "working" and not working versions even without a proper u/p
Fantastic, that'll speed things up. I've got some time to look at this.
mbafford
commented
2 years ago I keep intending and then forgetting to make this point. I think it's not even Fidelity doing the front-line blocking, but Akamai:
ofx.fidelity.com. 60 IN CNAME ofx.fidelity.com.edgekey.net.
ofx.fidelity.com.edgekey.net. 300 IN CNAME e92459.x.akamaiedge.net.
e92459.x.akamaiedge.net. 20 IN A 23.73.207.7
e92459.x.akamaiedge.net. 20 IN A 23.73.207.11Which explains why the error isn't an OFX error. It's just a CDN blocking the request for some reason. Which also explains why the username/password doesn't matter for the HTML responses.
csingley
commented
2 years ago It's the god damned User-Agent in the HTTP headers.
If you use some random string (like the default curl version announcement, or something else picked out of a hat), then it gets through just fine. However, if you use "InetClntApp/3.0" (which I'm given to understand is what Quicken sends) then it gets blocked at the HTTP level... presumably checking some other header Quicken sets, and we don't.
We switched to using that user agent string after 0.8.22... some FIs had issues that were alleviated thereby. Two steps forward, one step back.
ofxget stmt --useragent foobar netbenefits -u "err" --password "err" seems to do the trick. You can set this in ofxget.cfg
csingley
commented
2 years ago If htose of you working netbenefits logins can confirm that this fix works for you, then I can include it in the defaults fi.cfg so we can have a global fix, instead of every user needing to patch in this idiocy themselves.
mbafford
commented
2 years ago tl;dr - Yes, editing fi.cfg and adding a user-agent ("ofxtools") for the netbenefits configuration fixes netbenefits for me. Removing that line breaks it again.
So here's my results:
| Requests? | User-Agent? | Provider | Status | |
|---|---|---|---|---|
| Yes | Default | netbenefits | FAIL | |
| Yes | "ofxtools" | netbenefits | SUCCESS | |
| No | Default | netbenefits | FAIL | |
| No | "ofxtools" | netbenefits | SUCCESS | |
| Yes | Default | fidelity | SUCCESS | |
| Yes | "ofxtools" | fidelity | SUCCESS | |
| No | Default | fidelity | FAIL | |
| No | "ofxtools" | fidelity | FAIL |
So on my non-system Python on HEAD on MacOS:
Looks like you can run through permutations on "fidelity" with a bad username/password as well.
The user-agent thing baffles me a little - why does running with MITM fix it if it's a user-agent issue? mitmproxy/charles should have been preserving the user-agent, so it feels like there may be multiple factors being considered. I also know I tested keeping, changing, and removing the user-agent header ofxtools was sending with curl - to no change at all. It just always worked in curl.
However, I have less than no desire to put effort into figuring out the intricacies of the arbitrary blocking algorithms being used by Fidelity/Akamai. I'm just glad we found a workable solution.
mbafford
commented
2 years ago In retrospect, I think I was originally focused on fidelity then we all shifted to netbenefits, which have different behaviors, so both the requests and User-Agent work were needed, just for different reasons/providers.
@csingley Thank you for your help and remote troubleshooting through all of this. This is the kind of BS that shouldn't even exist, but all too-often plagues an open-source project.
csingley
commented
2 years ago
- "netbenefits" works with/without requests, as long as you change the user-agent
- "fidelity" works only with requests, and doesn't seem to care about the user-agent
That's what I was seeing as well.
The user-agent thing baffles me a little - why does running with MITM fix it if it's a user-agent issue? mitmproxy/charles should have been preserving the user-agent, so it feels like there may be multiple factors being considered. I also know I tested keeping, changing, and removing the user-agent header ofxtools was sending with curl - to no change at all. It just always worked in curl.
Yeah I can confirm this. Changing curl's request headers to look like ofxtools, it still succeeds. Curious.
I'm concerned that the root cause eludes us... but for now I'll be satisfied with "not blowing up in our faces".
Following Chris advice, opening a public discussion here.
Starting on 11/19/2021, I am getting “HTTP Error 403: Forbidden” error (full trace and ofxget.cfg are at the bottom). OFXClient call from my own script comes back with the same. I called Fidelity and after going through a chain of reps, talked to someone from OFX support. The person said that no changes have been made in months to OFX service itself, but was not sure if anything security-related was done to endpoint, front-ending OFX.
Quicken and “Fund Manager” (https://www.fundmanagersoftware.com/) still work and successfully retrieve OFX. Both of them seem to use 102 version of OFX request, though am not sure if this fact is relevant. Log snapshots are included further down. Quicken is doing requests to https://ofx.fidelity.com:443/ftgw/OFX/clients/download and https://ofx.fidelity.com/ftgw/OFX/clients/download
Moneydance is having a similar issue as discussed: https://infinitekind.tenderapp.com/discussions/online-banking/20519-usaa-and-fidelity-not-downloading-111921
Ofxhome’s automated OFX test also failed on 11/19/2021: https://www.ofxhome.com/index.php/institution/history/449
Traceback (most recent call last): File "anaconda3/bin/ofxget", line 8, in
sys.exit(main())
File "anaconda3/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 1590, in main
REQUEST_HANDLERSargs["request"]
File "anaconda3/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 710, in request_stmt
with client.request_statements(
File "anaconda3/lib/python3.8/site-packages/ofxtools/Client.py", line 404, in request_statements
return self.download(
File "anaconda3/lib/python3.8/site-packages/ofxtools/Client.py", line 863, in download
response = url_opener(req, *kwargs)
File "anaconda3/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "anaconda3/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "anaconda3/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(args)
File "anaconda3/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "anaconda3/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
---- ofxget.cfg ----- [fidelity] url = https://ofx.fidelity.com:443/ftgw/OFX/clients/download fid = 7776 brokerid = fidelity.com org = fidelity.com version = 220 appid = QWIN appver = 2700 language = ENG
Old url without :443 also does not work.
---- Fund Manager OFX successful OFX request --- OFXHEADER:100 DATA:OFXSGML VERSION:102 SECURITY:NONE ENCODING:USASCII CHARSET:1252 COMPRESSION:NONE OLDFILEUID:NONE NEWFILEUID:NONE
---- from OFXLOG.txt ---- ==== OSU Start (20211119/08:25:45) ====