FI Returns 400 Error on Profile Request

[aclindsa]

I think I've found my first FI which doesn't properly behave on profile requests (and therefore breaks with the new flow for ofxget). It was returning 400 errors when I was testing out your new release, and they seem to have now temporarily blocked my IP for too many attempts while I was trying to test it out more.

Assuming I'm able to confirm later that the issue is that their profile endpoint is misbehaving while statement requests work normally, what's the best way to handle this?

For the record, all the rest of my accounts worked just fine with the new method (including Vanguard!)...

[csingley]

Assuming I'm able to confirm later that the issue is that their profile endpoint is misbehaving while statement requests work normally, what's the best way to handle this?

The best way is to figure out how to resolve the HTTP error on the PROFRQ... it does work for Quicken, right? All these problems should be soluble by better emulation.

The easiest way is to turn that behavior off.

[aclindsa]

I don't have Quicken to check if the profile request works for it. I do have Moneydance, and I don't seem to be able to get it to work with this FI (though it is ostensibly supported).

[aclindsa]

I almost hesitate to ask, but are you willing for another argument to be added to ofxget allow the specified URL to be used verbatim as the one to request the statement(s) from?

[csingley]

Yeah that's fine, but... which FI are we talking about here? I'd like to try wringing a PROFRQ out of them before patching in yet another workaround.

[aclindsa]

NC State Employees Credit Union: https://onlineaccess.ncsecu.org/secuofx/secu.ofx

Okay, you made me do more work! After a little finagling, I am actually able to download a profile request using ofxgo, so we can make this work. In attempting to study the differences, I tried this, which didn't work:

$ ofxget prof -n --url https://onlineaccess.ncsecu.org/secuofx/secu.ofx
Traceback (most recent call last):
  File "/home/aclindsa/.local/bin/ofxget", line 33, in <module>
    sys.exit(load_entry_point('ofxtools==0.9', 'console_scripts', 'ofxget')())
  File "/home/aclindsa/.local/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 1568, in main
    REQUEST_HANDLERS[args["request"]](args)
  File "/home/aclindsa/.local/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 562, in request_profile
    with client.request_profile(
  File "/home/aclindsa/.local/lib/python3.8/site-packages/ofxtools/Client.py", line 500, in request_profile
    proftrnrs = ofx.profmsgsrsv1[0]
TypeError: 'NoneType' object is not subscriptable

I think the traceback is due to the failing profile request, but I would have expected it to print the text of the request when supplying '-n', even when the request fails.

Sigh, and now I've triggered their firewall against me again, I'll have to wait a while before trying again...

[csingley]

TypeError: 'NoneType' object is not subscriptable

That's a bug I introduced with the recent "read the PROFRS" commits. I'll fix it, thanks for reporting.

[csingley]

This regression should be fixed in a0fd41c, so you can at least dump the requests generated by ofxget.

I'll have a look at NCSECU while you're locked out.

[aclindsa]

One thing I just noticed is that you're not setting FI>ORG or FID on your profile requests (or even allowing them to be set by ofxget prof). I was. For NCSECU, fid=1001, org=SECU. In briefly looking at the spec, I don't immediately see anywhere mentioning you can omit those fields for profile requests.

[aclindsa]

The missing FID/ORG do appear to be the issue. They let me back in, and I was able to get a profile download by doing:

diff --git a/ofxtools/scripts/ofxget.py b/ofxtools/scripts/ofxget.py
index c07aafb..ca7a7c1 100644
--- a/ofxtools/scripts/ofxget.py
+++ b/ofxtools/scripts/ofxget.py
@@ -138,7 +138,7 @@ def make_argparser() -> argparse.ArgumentParser:
     subparsers["prof"] = add_subparser(
         subparsers_,
         "prof",
-        format=True,
+        signon=True,
         help=("Download OFX service profile for server"),
     )
     subparsers["acctinfo"] = add_subparser(

and

$ ofxget prof --useragent "ofxtools" --nonewfileuid --url https://onlineaccess.ncsecu.org/secuofx/secu.ofx --fid 1001 --org SECU

I have noticed what appears to be some intermittent failures while doing the profile downloads from them, but I suppose it's possible those are some form of rate limiting that I'm hitting.

[csingley]

That's a bug, too. I didn't have information for NCSECU in fi.cfg, so it's supposed to pull that down from ofxhome. Apparently it's not doing that. Grrr.

I've updated fi.cfg & pushed it. Now what's most needed is to make useragent and nonewfileuid configurable in ofxtools.cfg and fi.cfg.... and fix the bug related to pulling down configs from ofxhome. Plus what looks like dodgy subparser construction in ofxget, as you note....

[aclindsa]

So I've pulled in your latest changes and have two more wrinkles to throw at you:

1. I realized that the original 400 error occurred when I was specifying FID and ORG on the command-line using something like ofxget --stmt --user USERNAME --org SECU --fid 1001 --bankid BANKID --savings ACCTID --nonewfileuid --useragent "ofxtools" --url https://onlineaccess.ncsecu.org/secuofx/secu.ofx. I was going to try to investigate that when I found....
2. When I add -n to the above command, I get the following:

Traceback (most recent call last):
  File "/mnt/data/documents/beancount/external/ofxtools/ofxtools/scripts/ofxget.py", line 1581, in <module>
    main()
  File "/mnt/data/documents/beancount/external/ofxtools/ofxtools/scripts/ofxget.py", line 1577, in main
    REQUEST_HANDLERS[args["request"]](args)
  File "/mnt/data/documents/beancount/external/ofxtools/ofxtools/scripts/ofxget.py", line 707, in request_stmt
    with client.request_statements(
  File "/mnt/data/documents/beancount/external/ofxtools/ofxtools/Client.py", line 344, in request_statements
    RqCls2url = self._get_service_urls()
  File "/mnt/data/documents/beancount/external/ofxtools/ofxtools/Client.py", line 416, in _get_service_urls
    profile = self.request_profile()
  File "/mnt/data/documents/beancount/external/ofxtools/ofxtools/Client.py", line 485, in request_profile
    response = self._request_profile(
  File "/mnt/data/documents/beancount/external/ofxtools/ofxtools/Client.py", line 560, in _request_profile
    return self.download(
  File "/mnt/data/documents/beancount/external/ofxtools/ofxtools/Client.py", line 823, in download
    response = url_opener(req, **kwargs)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

Note that I think this failure is ultimately because I've angered their firewall again, but I wonder if we should be skipping actually attempting the profile request for -n, and instead printing both it and the subsequent statement request out?

[csingley]

I wonder if we should be skipping actually attempting the profile request for -n, and instead printing both it and the subsequent statement request out?

Yeah this is another bug introduced by the recent hasty implementation of "read the PROFRS" logic. The problem is OFXClient.request_statements() call to OFXClient._get_service_urls(), which of course sends out a PROFRQ.

I think it should just print the STMTRQ; it's easy enough to get ofxget to dump the PROFRQ if you want that. Either way is simple to implement though; I'll just check it in.

That first one though... it's looking like there's some problems with the unpleasant logic for merging in configs from user config file, master config file, CLI, and ofxhome. It looks like you added the new options useragent and nonewfileuid into ofxget.configurable_srvr, so my first guess isn't it, and it'll require more digging...

[csingley]

The good news is that ofxget does in fact read useragent and nonewfileuid from the config files. I'll go ahead and update fi.cfg accordingly, instead of making everybody stick it in their ofxget.cg.

With this change, you should now just be able to do run ofxget prof ncsecu and ofxget stmt ncsecu in the normal manner, without all the extra CLI flags.

More to debug though...

[csingley]

diff --git a/ofxtools/scripts/ofxget.py b/ofxtools/scripts/ofxget.py
index c07aafb..ca7a7c1 100644
--- a/ofxtools/scripts/ofxget.py
+++ b/ofxtools/scripts/ofxget.py
@@ -138,7 +138,7 @@ def make_argparser() -> argparse.ArgumentParser:
     subparsers["prof"] = add_subparser(
         subparsers_,
         "prof",
-        format=True,
+        signon=True,
         help=("Download OFX service profile for server"),
     )
     subparsers["acctinfo"] = add_subparser(

Nice catch! Yes, that's exactly right. I'll commit that fix too.

[csingley]

That's a bug, too. I didn't have information for NCSECU in fi.cfg, so it's supposed to pull that down from ofxhome. Apparently it's not doing that. Grrr.

That at least is not a bug. It does in fact pull down the configs from ofxhome; you just weren't using the ncsecu server name but were instead specifying all the args from the command line. Your actual bug is fixed by your patch allowing signon args for the profile subparser.

[csingley]

The more fundamental issue raised by all of these bugs is why test_client.OFXV1ClientTestCase.testRequestStatementsDryrun() wasn't failing for any of them!

[rkhwaja]

How do you find the profrq urls? I see from a different issue that Vanguard, for instance, has a different url than the normal one that you send statement requests to and yet ofxhome only has one url listed.

[aclindsa]

Interesting - Though I am able to successfully get profile requests from the command-line, I am still getting urllib.error.HTTPError: HTTP Error 400: Bad Request for the profile requests made as a preface to the statement request. I'll debug a little more later to confirm, but I'm wondering if this is because they are not expecting you to send an actual username/password with the profile request.

[aclindsa]

@rkhwaja In almost all cases other than Vanguard, the profile and statement request URLs are identical.

[aclindsa]

I'm wondering if this is because they are not expecting you to send an actual username/password with the profile request.

Scratch that. It's because _get_service_urls() doesn't pass through gen_newfileuid.

[aclindsa]

@csingley I believe https://github.com/csingley/ofxtools/pull/108 contains the final change needed to complete this week's chapter of "Why can't my FI just obey the OFX spec?".

[vRootEbal]

onlineaccess.ncsecu.org do now working??? please give me working post ofx request

[csingley]

What I want is a pony. Please give me a pony.

[vRootEbal]

What I want is a pony. Please give me a pony.

i have pony, and i give you pony, if you give me ofx for ncsecu))))

[csingley]

All talk, no pony! Pls give pony now

[aclindsa]

What I want is a pony. Please give me a pony.

i have pony, and i give you pony, if you give me ofx for ncsecu))))

If anyone wants a pony, they're going to have to complain to NCSECU directly (please do!). It has stopped working again - they have started more intentionally blocking requests which do not come from Quicken. My understanding from bugging them about it is that their blocking is more akin to a firewall (based on IP's of Intuit's servers) and not likely something we can work around with something like changing the HTTP headers we send. I do not have a Quicken install to be able to test whether Quicken is still able to ping NCSECU's severs directly or whether they are proxying requests indirectly through their servers.