Allow importing via CSV

[scottrobertson]

Being able to import via a CSV would be incredible. Even something as simple as the following:

date	payee	amount	memo	account
2015-01-01	Amazon	100	Something something	Mondo

[rienafairefr]

I've implemented a CSV importer script, can you test it with your data?

[scottrobertson]

Awesome! Shall do. How are you thinking of dealing with duplicates?

[scottrobertson]

Traceback (most recent call last):
  File "scripts/csvimport.py", line 6, in <module>
    from jsontableschema.model import SchemaModel
  File "/usr/local/lib/python2.7/site-packages/jsontableschema-0.5.1-py2.7.egg/jsontableschema/__init__.py", line 7, in <module>
    from . import exceptions
  File "/usr/local/lib/python2.7/site-packages/jsontableschema-0.5.1-py2.7.egg/jsontableschema/exceptions.py", line 7, in <module>
    import jsonschema.exceptions
  File "/usr/local/lib/python2.7/site-packages/jsonschema-2.5.1-py2.7.egg/jsonschema/__init__.py", line 12, in <module>
    from jsonschema.exceptions import (
  File "/usr/local/lib/python2.7/site-packages/jsonschema-2.5.1-py2.7.egg/jsonschema/exceptions.py", line 6, in <module>
    from jsonschema import _utils
  File "/usr/local/lib/python2.7/site-packages/jsonschema-2.5.1-py2.7.egg/jsonschema/_utils.py", line 6, in <module>
    from jsonschema.compat import str_types, MutableMapping, urlsplit
  File "/usr/local/lib/python2.7/site-packages/jsonschema-2.5.1-py2.7.egg/jsonschema/compat.py", line 39, in <module>
    from functools32 import lru_cache
ImportError: No module named functools32

[rienafairefr]

For now, duplicates are imported as is, no effort to remove them. I will try to not import a transaction if a matching one already exists in nYNAB (same date, amount, category, memo etc).

I think the error you get is due to an old version of pip or setuptools, during jsontableschema install the functools32 dependency is not installed, just install it manually (pip install functools32)

[scottrobertson]

For now, duplicates are imported as is, no effort to remove them. I will try to not import a transaction if a matching one already exists in nYNAB (same date, amount, category, memo etc).

That would be perfect. Just very basic checking is enough :)

I think the error you get is due to an old version of pip or setuptools, during jsontableschema install the functools32 dependency is not installed, just install it manually (pip install functools32)

Just installed it via brew. Shall try and install it manually and see. Man, there must be a better way for python to handle this stuff. Like a Gemfile :P

[rienafairefr]

duplicates should be handled now, see commit

[rienafairefr]

python setup.py install is the way python handle this stuff, that should work ^^

[scottrobertson]

Thank you. Getting pynYNAB.Client.BudgetNotFound now. Tried the name, ID etc

[scottrobertson]

Ok, sorted that. They are all "My Budget" for some reason.

Traceback (most recent call last):
  File "scripts/csvimport.py", line 39, in <module>
    get_logger().ERROR('This schema doesn''t exist in csv_schemas')
  File "build/bdist.macosx-10.11-x86_64/egg/pynYNAB/config.py", line 20, in get_logger
AttributeError: 'NoneType' object has no attribute 'upper'

Is my next issue. I am using --schema=example

[rienafairefr]

Hi Scott, I've sorted a bit the scripts interface, see scripts/README.rst for the documentation. while in the scripts folder I think now you can do:

python csvmport.py data.csv example

"My Budget" default and logger level being None was a bug.

[scottrobertson]

Traceback (most recent call last):
  File "scripts/csvimport.py", line 10, in <module>
    from pynYNAB.Client import clientfromargs
ImportError: cannot import name clientfromargs

[rienafairefr]

did you get and install the new source? If you want to be able to git pull for the package and get it system wide easily, you can use do python setup.py develop

[scottrobertson]

I did a git pull, but could not run python setup.py develop. I posted in the other ticket about that, as it throws an error.

[rienafairefr]

Indeed, there was a bug in setup.py (I had changed the README name markup), it's fixed now. Sorry about that

[scottrobertson]

Seems to run ok now, but it's not inserting anything into YNAB

[scottrobertson]

Looking at the JSON response in YNAB, all of my budgets are named "my budget" for some reason... very odd. Perhaps it's because i created them way back in the beta.

[rienafairefr]

I know at some point pynYNAB defaulted to 'My Budget' for the budget name. If you want to insert in another budget specify it in ynab.conf or add --budgetname YourBudgetNamee to the call to csvimport script. Can you paste here a snippet of the csv data, the full command line for the call to csvimport.py, and the ynab.conf (put ****\ where your ID/password is) if applicable

[scottrobertson]

Well, the actual budget name is "Scott's Budget" in YNAB's interface, but not in the JSON oddly.

python pynYNAB/scripts/csvimport.py ~/Downloads/ynab.csv example --email=**** --password=**** --budget='My Budget'

2016-03-08, Testing, 10, Something, Mondo

[rienafairefr]

I've documented the usage in the documentation,

usage: csvimport.py [-h] [--email Email] [--password Password]
                    [--level LoggingLevel] [--budgetname BudgetName]
                    CSVpath schemaName [AccountName]

I think the correct call in your case would be:

python pynYNAB/scripts/csvimport.py ~/Downloads/ynab.csv example --email * --password ** --budgetname "Scott's Budget"

[scottrobertson]

It seems to find the budget ok with My Budget. If i use Scott's Budget it breaks.

No budget by this name found in nYNAB

[scottrobertson]

Perhaps print each transaction out that will be added to be_transactions

[scottrobertson]

error: python-dateutil 2.5.0 is installed but python-dateutil<=2.4.2 is required by set(['dateparser'])

Getting that now when running setup. What is the best way to deal with that?

[rienafairefr]

I used dateparser dependency before I think, not anymore, so I removed it. I guess something else is installed on your system that needed python-dateutil 2.5.0 . You could use a virtualenv to install pynYNAB in isolation but I think that's overkill.

[scottrobertson]

Ah ok. So is it safe to just ignore that?

Anyway, how do i enable logging? I have tried --level=debug but the transactions are not being output at all

[rienafairefr]

Thanks for doing the testing. pull the new source from the repo, then restart the setup, that should work now.

config key value pairs are specified --key value, not --key=value, so debug would be --level debug but it's not really verbose for now. I'll add more

[scottrobertson]

Thanks.

I am getting no output at all

[scottrobertson]

python pynYNAB/scripts/csvimport.py ~/Downloads/ynab.csv example --level debug --email *** --password *** --budgetname "My Budget"

[rienafairefr]

can you get new source, commit a6bf0812e6a5d1b07c684bb8ceeb2a46760575ba and see if you still get nothing ? This works for me:

D:\Documents\nYNABapi>C:\Python27\python.exe pynYNAB\scripts\csvimport.py tests\data\test.csv example --level debug
pynYNAB CSV import
DEBUG:pynYNAB:selected schema example
DEBUG:pynYNAB:schema headers [u'date', u'payee', u'amount', u'memo', u'account']

DEBUG:pynYNAB:OK starting the import from D:\Documents\nYNABapi\tests\data\test.csv
DEBUG:pynYNAB:read line 2016-02-01,Super Pants Inc.,-20,Buying pants,Cash

DEBUG:pynYNAB:searching for account Cash
ERROR:pynYNAB:Couldnt find this account: Cash

In this case, email,password,budgetname are setup in ynab.conf

[scottrobertson]

That worked :) it now cannot find the Account though

[scottrobertson]

Think i may know why

[scottrobertson]

woohoo, that worked. I had a space at the start of the account name.

Duplicate checking does not seem to be working.

[rienafairefr]

have you tested with the latest code? Seems to be working now