search

egh / ledger-autosync

Synchronize your ledger-cli files with your bank.
GNU General Public License v3.0
273 stars 42 forks source link

"ValueError: need more than 1 value to unpack" #24

Closed paxperscientiam closed 7 years ago

paxperscientiam commented 7 years ago

Hello @egh

ledger-autosync is unfortunately not working for me.

Executing ledger-autosync transactions.ofx outputs:

|13:32:56|cdr35@Prometheus:[VFCU]> ledger-autosync transactions.ofx
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/ledger-autosync", line 11, in <module>
    sys.exit(run())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ledgerautosync/cli.py", line 182, in run
    ledger_file = find_ledger_file()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ledgerautosync/cli.py", line 50, in find_ledger_file
    re.match(r".*--file\s+([^\s]+).*", ledgerrc).group(1)))
AttributeError: 'NoneType' object has no attribute 'group'
|13:33:08|cdr35@Prometheus:[VFCU]>

Executing ledger-autosync transactions.ofx -l ./journal.dat outputs:

|13:30:22|cdr35@Prometheus:[VFCU]> ledger-autosync transactions.ofx -l ./journal.dat
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/ledger-autosync", line 11, in <module>
    sys.exit(run())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ledgerautosync/cli.py", line 222, in run
    import_ofx(ledger, args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ledgerautosync/cli.py", line 107, in import_ofx
    (ofx, txns) = sync.parse_file(args.PATH)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ledgerautosync/sync.py", line 35, in parse_file
    ofx = OfxParser.parse(file(path))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ofxparse/ofxparse.py", line 386, in parse
    ofx_file = OfxPreprocessedFile(file_handle)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ofxparse/ofxparse.py", line 150, in __init__
    super(OfxPreprocessedFile, self).__init__(fh)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ofxparse/ofxparse.py", line 80, in __init__
    self.read_headers()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ofxparse/ofxparse.py", line 93, in read_headers
    header, value = line.split(six.b(":"))
ValueError: need more than 1 value to unpack
|13:31:53|cdr35@Prometheus:[VFCU]>

It also seems that ledger-autosync is not properly parsing LEDGER_FILE, though I am unsure.

Thanks for your attention.

egh commented 7 years ago

Thanks for the reports! I hope this is also fixed in 2d5f457. (fingers crossed). it seems to be an undocumented ledger feature to only parse arguments and not the ledgerrc file, so I hope that works for you.

paxperscientiam commented 7 years ago

You're most welcome. Thanks for the fast reply.

This changed hasn't been "pushed" to pypi, has it? It doesn't seem so.

I'm actually a Python noob. How would I test this without the setup script attempting to install in the default python module directory?

egh commented 7 years ago

No, sorry, it has just been pushed to the git repo.

If you have virtualenv installed:

$ git clone git@github.com:egh/ledger-autosync.git
$ cd ledger-autosync
$ virtualenv venv
$ source venv/bin/activate
$ python setup.py install

That should do the trick.

If this fixes your issue I'll push a new pypi release - it seems like a bug other people are probably seeing.

paxperscientiam commented 7 years ago

Thanks for the clear and concise instructions.

I got everything setup without a hitch. I did have to install python-Levenshtein to remove a complaint, but that's besides the point.

Oh, I also made sure that the HEAD pointed to 2d5f457.

Still getting an error, I'm afraid:

(venv) |13:05:44|cdr35@Prometheus:[ledger-autosync]> ledger-autosync transactions.ofx
Traceback (most recent call last):
  File "/private/tmp/ledger-autosync/venv/bin/ledger-autosync", line 11, in <module>
    load_entry_point('ledger-autosync==0.3.4', 'console_scripts', 'ledger-autosync')()
  File "build/bdist.macosx-10.12-x86_64/egg/ledgerautosync/cli.py", line 194, in run
  File "build/bdist.macosx-10.12-x86_64/egg/ledgerautosync/cli.py", line 51, in find_ledger_file
AttributeError: 'NoneType' object has no attribute 'group'
egh commented 7 years ago

Ah, interesting. I forgot I was parsing the .ledgerrc file. Can you post yours? I think the regex match is failing

paxperscientiam commented 7 years ago

Hmm, it's actually completely commented out! I had done this after figuring that the "pedantic" setting was causing an error.

Results of nosetests -a generic:

..........................
----------------------------------------------------------------------
Ran 26 tests in 91.531s

OK

Results of nosetests -a ledger:

(venv) |18:05:05|cdr35@Prometheus:[ledger-autosync]> nosetests -a ledger
................
----------------------------------------------------------------------
Ran 16 tests in 48.883s

OK

Any other things I could try? Despite it not being required...maybe the python bindings are required?

EDIT Tried a few more things, still using the virtual environment.

I tried one of your test files -- checking.ofx. This worked, though it required that i specify my journal file with the -l switch.

Doing so gave the following out:

(venv) |22:08:56|cdr35@Prometheus:[ledger-autosync]> ledger-autosync -l /Users/cdr35/The\ Cloud/Dropbox/FINANCE/journal.dat checking.ofx
2011/03/31 DIVIDEND EARNED FOR PERIOD OF 03/01/2011 THROUGH 03/31/2011 ANNUAL PERCENTAGE YIELD EARNED IS 0.05%
    ; ofxid: 1101.1452687~7.0000486
    FAKE:1452687~7                                          $0.01
    Expenses:Misc                                          -$0.01

2011/04/05 AUTOMATIC WITHDRAWAL, ELECTRIC BILL WEB(S )
    ; ofxid: 1101.1452687~7.0000487
    FAKE:1452687~7                                        -$34.51
    Expenses:Misc                                          $34.51

2011/04/07 RETURNED CHECK FEE, CHECK # 319 FOR $45.33 ON 04/07/11
    ; ofxid: 1101.1452687~7.0000488
    FAKE:1452687~7                                        -$25.00
    Expenses:Misc                                          $25.00

That looks pretty darn good to me.

My understanding is that ledger-autosync should be able to parse OFX even without "knowing" about any specific journal files. That said, I do recall that it looks for the LEDGER_FILE variable. Mine is defined, though seemingly ignored(?).

ledger-autosync checking.ofx yields:

Traceback (most recent call last):
  File "/private/tmp/ledger-autosync/venv/bin/ledger-autosync", line 11, in <module>
    load_entry_point('ledger-autosync==0.3.4', 'console_scripts', 'ledger-autosync')()
  File "build/bdist.macosx-10.12-x86_64/egg/ledgerautosync/cli.py", line 194, in run
  File "build/bdist.macosx-10.12-x86_64/egg/ledgerautosync/cli.py", line 51, in find_ledger_file
AttributeError: 'NoneType' object has no attribute 'group'

OK, one more thing... My bank doesn't include a block like this:

<DTSERVER>20130525225731.258
  19 │                         <LANGUAGE>ENG
  20 │                         <DTPROFUP>20050531060000.000
  21 │                         <FI>
  22 │                                 <ORG>FAKE
  23 │                                 <FID>1101
  24 │                         </FI>
  25 │                         <INTU.BID>51123
  26 │                         <INTU.USERID>9774652

Adding in that dummy data "fixed" it aside from what is summarized below.

EDIT OK, let me try to summarize all this:

I'm gonna hang back and just answer your questions after spewing all that!

egh commented 7 years ago

Thanks for the very detailed reports.

  1. ledger-autosync should prefer the LEDGER_FILE variable if available. It seems to work with ledger files with spaces in my tests. At least, running LEDGER_FILE=~/test\ file.lgr ledger-autosync in bash on linux works for me
  2. .ledgerrc parsing is a little funky. I can't figure out how to even get ledger to handle files with spaces here. Ideally we could use ledgers handling of the --file ... option here, but then we would also be using any other default settings in .ledgerrc. That has been how ledger-autosync worked until yesterday, when I added support for --args-only. And I like this solution, generally: ledger-autosync relies on the output from ledger being as expected, and if ledgerrc specifies something different, it could be a problem.
  3. It does look like you are also seeing a problem with your ofx files. ledger-autosync uses https://github.com/jseutter/ofxparse to handle the ofx parsing. I can't imagine why your OFX files would be missing those bits. Especially the <FI><ORG> stuff seems pretty important. :)
egh commented 7 years ago

Regarding "shouldn't ledger-autosync parse an OFX file regardless?": I think that ledger-autosync could do better here, but if there is no ledger file, that seems like an error. ledger won't work without a ledger file either. I guess we could pass in -f /dev/null or something, but we'd probably want to warn the user that the program was not actually syncing any transactions, just converting.

paxperscientiam commented 7 years ago

but we'd probably want to warn the user that the program was not actually syncing any transactions, just converting

I think that is a good idea. If we're just printing to stdout, then this is an effective way to communicate/preview what converted content looks like. It also indicates that there were no parsing errors.

Hm..this might be a good opportunity to learn some Python.

egh commented 7 years ago

I have addressed this issue in 80c3eced14379b0f251c5e6efda148f6cc0cc34d

paxperscientiam commented 7 years ago

@egh Hey, so I checked out 80c3ece, but I'm unfortunately still seeing errors. I'm actually still seeing the error regarding missing journal. I used interpreter version 2.7.

Am I missing something?

(venv) |21:22:02|cdr35@Prometheus:[ledger-autosync]> ledger-autosync fixtures/apostrophe.ofx
Error: No journal file was specified (please use -f)
Traceback (most recent call last):
  File "/private/tmp/ofx/ledger-autosync/venv/bin/ledger-autosync", line 11, in <module>
    load_entry_point('ledger-autosync==0.3.4', 'console_scripts', 'ledger-autosync')()
  File "build/bdist.macosx-10.12-x86_64/egg/ledgerautosync/cli.py", line 253, in run
  File "build/bdist.macosx-10.12-x86_64/egg/ledgerautosync/cli.py", line 111, in import_ofx
  File "build/bdist.macosx-10.12-x86_64/egg/ledgerautosync/sync.py", line 36, in parse_file
  File "build/bdist.macosx-10.12-x86_64/egg/ledgerautosync/sync.py", line 82, in filter
  File "build/bdist.macosx-10.12-x86_64/egg/ledgerautosync/sync.py", line 45, in is_txn_synced
  File "build/bdist.macosx-10.12-x86_64/egg/ledgerautosync/ledgerwrap.py", line 166, in check_transaction_by_id
  File "build/bdist.macosx-10.12-x86_64/egg/ledgerautosync/ledgerwrap.py", line 160, in run
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 219, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['ledger', '--args-only', 'csv', '-E', 'meta', u'ofxid=1452687~7.201409206']' returned non-zero exit status 1
egh commented 7 years ago

@paxperscientiam Sorry - I added a new option to prevent loading a ledger file, but I didn't fix the behavior when no ledger file could be found. Should be fixed in a3e4274

paxperscientiam commented 7 years ago

@egh nicely done! Do you this version will hit pypi? Any other testing I could help with?

(venv) |13:27:46|cdr35@Prometheus:[ledger-autosync]> ledger-autosync fixtures/apostrophe.ofx
/Users/cdr35/ofx/ledger-autosync/venv/lib/python2.7/site-packages/fuzzywuzzy-0.15.0-py2.7.egg/fuzzywuzzy/fuzz.py:35: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
LEDGER_FILE environment variable not set, and no .ledgerrc file found, and -l argument was not supplied: running with deduplication disabled. All transactions will be printed!2014/09/20 TRADER JOE'S #541 QPS
    ; ofxid: 1101.1452687~7.201409206
    FAKE:1452687~7                                        -$58.73
    Expenses:Misc                                          $58.73
egh commented 7 years ago

@paxperscientiam I just released 0.3.5.

egh commented 7 years ago

@paxperscientiam Thanks for testing! I appreciate it.