Fidelity Investments OFX download fails again

[vitaliknet]

Following Chris advice, opening a public discussion here.

Starting on 11/19/2021, I am getting “HTTP Error 403: Forbidden” error (full trace and ofxget.cfg are at the bottom). OFXClient call from my own script comes back with the same. I called Fidelity and after going through a chain of reps, talked to someone from OFX support. The person said that no changes have been made in months to OFX service itself, but was not sure if anything security-related was done to endpoint, front-ending OFX.

Quicken and “Fund Manager” (https://www.fundmanagersoftware.com/) still work and successfully retrieve OFX. Both of them seem to use 102 version of OFX request, though am not sure if this fact is relevant. Log snapshots are included further down. Quicken is doing requests to https://ofx.fidelity.com:443/ftgw/OFX/clients/download and https://ofx.fidelity.com/ftgw/OFX/clients/download

Moneydance is having a similar issue as discussed: https://infinitekind.tenderapp.com/discussions/online-banking/20519-usaa-and-fidelity-not-downloading-111921

Ofxhome’s automated OFX test also failed on 11/19/2021: https://www.ofxhome.com/index.php/institution/history/449

Traceback (most recent call last): File "anaconda3/bin/ofxget", line 8, in sys.exit(main()) File "anaconda3/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 1590, in main REQUEST_HANDLERSargs["request"] File "anaconda3/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 710, in request_stmt with client.request_statements( File "anaconda3/lib/python3.8/site-packages/ofxtools/Client.py", line 404, in request_statements return self.download( File "anaconda3/lib/python3.8/site-packages/ofxtools/Client.py", line 863, in download response = url_opener(req, *kwargs) File "anaconda3/lib/python3.8/urllib/request.py", line 531, in open response = meth(req, response) File "anaconda3/lib/python3.8/urllib/request.py", line 640, in http_response response = self.parent.error( File "anaconda3/lib/python3.8/urllib/request.py", line 569, in error return self._call_chain(args) File "anaconda3/lib/python3.8/urllib/request.py", line 502, in _call_chain result = func(*args) File "anaconda3/lib/python3.8/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden

---- ofxget.cfg ----- [fidelity] url = https://ofx.fidelity.com:443/ftgw/OFX/clients/download fid = 7776 brokerid = fidelity.com org = fidelity.com version = 220 appid = QWIN appver = 2700 language = ENG

Old url without :443 also does not work.

---- Fund Manager OFX successful OFX request --- OFXHEADER:100 DATA:OFXSGML VERSION:102 SECURITY:NONE ENCODING:USASCII CHARSET:1252 COMPRESSION:NONE OLDFILEUID:NONE NEWFILEUID:NONE

20211119152654.649 [myusername] [myuserpass] N ENG fidelity.com 7776 QWIN 2700 29105-FM7-20211119152654.649-649 fidelity.com [myaccount] 20211110 Y N Y Y

---- from OFXLOG.txt ---- ==== OSU Start (20211119/08:25:45) ====

20211119082546.123[-5:EST] [myusername] [myuserpass] ENG fidelity.com 7776 QWIN 2700 [some id] fidelity.com [myaccount] Y N Y Y [truncated]

[patbakdev]

YES!. This works for me as well.

[csingley]

All righty then.

dbf4b93 has the fix to fi.cfg

Thanks to all who participated here. A small victory against the Forces of Wack.

[pacogomez]

thanks for the info in this thread, got my connections to fidelity and netbenefits working now.

[patbakdev]

Just an FYI. Netbenefits (only) stopped working for me.

Interestingly, this fails: ofxget stmt --useragent ofxtools netbenefits -u "err" --password "err" while this works: ofxget stmt --useragent foobar netbenefits -u "err" --password "err"

I changed my calling code to pass my own bullshit useragent, but it seems netbenifts might have decided to half-ass blacklist ofxtools.

[csingley]

Oh FFS. Are we gonna have to start generating random useragent strings each connection?

[patbakdev]

I know that was a rhetorical question :), but fwiw I have been having this issue for a few weeks (using ofxtools ua) and I also tried foobar at least twice in a row without issue. I have now hard coded a different value very specific to me. So for now I think we need to wait and see how long before I get blacklisted. Its possible that ofxtools is too large an audience, but an individual can fly under the radar. In that case I think for netbenefits at least the recommendation should be to set your own unique au, like you do with clientid.

[csingley]

My own recommendation here actually involves a different interface for access by the eagle-eyed team at NetBenefits

[vitaliknet]

@patbakdev My suggestion for anyone who has option of moving to a different broker is to do so.

@csingley maintains another repo here on GitHub with tools to access another broker (or should I say brokers :) with officially supported non-ofx interface. Besides programmatic access to statements, "those brokers" also have trading api and other tools with huge active community. It is just entirely different level of available tech.

[csingley]

@vitaliknet unfortunately, NetBenefits is actually a custodian for company benefit plans. It's the plan trustee/administrator, not the plan beneficiaries, who choose custody for plan assets.

[vitaliknet]

@csingley I know - used to be in that NetBenefits boat too. Sometimes people leave their 401k there though, instead of moving to rollover IRA. But if it is an active plan, then yes - there is no way around. I am in a similar situation with another custodian - want to move out but can't.

[csingley]

What's especially galling is that I just switched to a job where the plan's at NetBenefits, so they're earning money off my assets, and meanwhile apparently making special effort to dis my users.

I mean, I don't especially care that much personally; the OFX download client is not the part of this library I depend on. But I mean come on, GTFO with this "shoes off through the metal detector" bullshit

[mbafford]

My previous working environment of ofxtools 0.8.22 with a slightly hacked ofxget to allow password on the command line - looks to be working fine still.

[aclindsa]

Thanks for this!

For me, both netbenefits and non-netbenefits Fidelity downloads stopped working after February 19. I had assumed they just locked me out for good and hadn't had the time to look into it. I tried again just now, and a user agent of 'InetClntApp/3.0' got me in business with non-netbenefits, but netbenefits required a more creative user-agent to get working (my first choice was a 6-letter lower-case word describing the state of Fidelity OFX).

[dustinfarris]

fwiw i'm using uuidgen to:

ofxget stmt netbenefits --useragent $(uuidgen)

[dustinfarris]

Since May, netbenefits does not return any transactions, it is an empty statement with balances.

[patbakdev]

FYI. I just downloaded today and I am not having this issue. However, I was having an issue some time ago and I looked at my code and I see this comment:

# TODO Only Netbenefits seems to fail if end date equals today. Need to figure out what is going wrong here
#  and remove the offset hack

Hope that helps.

[dustinfarris]

FYI. I just downloaded today and I am not having this issue. However, I was having an issue some time ago and I looked at my code and I see this comment:

# TODO Only Netbenefits seems to fail if end date equals today. Need to figure out what is going wrong here
#  and remove the offset hack

Hope that helps.

Thanks. I'm still getting the same result even if I tweak the end date. I get a valid OFX response, but there are no transactions, just balances. Doesn't seem to matter what date range I give it.

Would you mind sharing your netbenefits config and/or usage so I can compare my own?

[patbakdev]

Sure, it will be a bit messy. I don't store the config in a normal file. I store it as metadata of the account in my beancount.bc file. I also have my own cli with a download command that calls ofxtools. I have copied the relevant code from that module and added comments as to values it is getting from the beancount.bc file for my netbenefits account.

CLIENTUID = "<32 Character String of Upper Case Letters and Numbers"

institution = entry.meta["ofx_institution"] # 'fidelity'
fid = entry.meta["ofx_fid"]                 # '7776'
userid = entry.meta["ofx_userid"]           # Self Explanatory
accountid = entry.meta["account_id"]        # <5 Number Account Id>

useragent = entry.meta.get("ofx_user_agent") or randomstring(10)  # 'ofx_user_agent' is missing, so its randomstring(10)

client = OFXClient(
        url=entry.meta["ofx_url"],  # 'https://nbofx.fidelity.com/netbenefits/ofx/download'
        userid=userid, 
        clientuid=entry.meta.get("ofx_client_id") or CLIENTUID, # 'ofx_client_id' is missing so its CLIENTID
        org=entry.meta.get("ofx_org"), # 'nbofx.fidelity.com'
        fid=entry.meta.get("ofx_fid"), # '8288' 
        version=int(entry.meta.get("ofx_version")) if entry.meta.get("ofx_version") else None, # '220'
        appid=entry.meta.get("ofx_appid"), # Not set so none ?
        appver=entry.meta.get("ofx_appver"), # Not set so none ?
        language=entry.meta.get("ofx_language"), # Not set so none ?
        bankid=entry.meta.get("ofx_bankid"), # 'fidelity' 
        brokerid=entry.meta.get("ofx_broker_id"), # 'nbofx.fidelity.com'
        useragent=useragent
)

# TODO Only Netbenefits seems to fail end date equals today. Need to figure out what is going wrong here
#      and remove the offset hack
datetime.now()
dtend = datetime.combine(date.today(), datetime.min.time()).astimezone(tz=ofxtools.utils.UTC)
offset = entry.meta.get("ofx_dtend_offset") # 1
if offset:
        dtend = datetime.combine(date.today(), datetime.min.time()).astimezone(tz=ofxtools.utils.UTC) - timedelta(days=int(offset))
dtstart = dtend - timedelta(days=download_days)

statement = None
statement_type = entry.meta["ofx_statement_type"] # 'investment'
if "bank" == statement_type:
        statement = StmtRq(
        acctid=accountid,
        accttype=entry.meta["ofx_account_type"].upper(),
        dtstart=dtstart,
        dtend=dtend,
        )
elif "credit" == statement_type:
        statement = CcStmtRq(
        acctid=accountid,
        dtstart=dtstart,
        dtend=dtend,
        )
elif "investment" == statement_type:
        statement = InvStmtRq(
        acctid=accountid,
        dtstart=dtstart,
        dtend=dtend,
        )

filepath = f"Downloads/{institution}-{accountid}.ofx"

response = client.request_statements(get_passwd(f"{fid}-{userid}"), statement)

with open(filepath, "wb") as ofx_file:
    ofx_file.write(response.getbuffer())

[patbakdev]

You got me to take a closer look at things. :) Apparently I don't have any transactions either...but that is because I don't actually have any transactions. Its an old account and I thought there might have been some dividends over time, but apparently not. I am, however, getting balance, holdings, price, vest info, etc. And that is more than I was getting when it broke on my earlier in the year. I think at that time I was just getting an empty file that managed to get pass the xml validation. In any case, hopefully, something I have provided will point you in the right direction.

[patbakdev]

Just another FYI for posterity. I ran into another issue today with netbenefits that looked like a legit error:

ValueError: SONRQ must contain either <USERID> and <USERPASS> or <USERKEY>, but not both

But, nope. I just changed my VPN server and it went through fine. This happens occasionally and the errors coming back are sometimes different.

So now when I run into errors with netbenefits the first thing I do (if safe) is disable my VPN and that usually works.