db branch beta test

[abesto]

First impressions follow. I got around to this pretty late today, so I didn't follow up on things I could've. Will do more testing in a few days. All this is using the CSV import script.

• With loglevel debug, CSV import of 50 entries generates a metric shitton of logs. This slows down execution by several seconds compared to loglevel info. Probably not an actual issue, more an interesting note.
• Many objects miss a friendly __repr__ method, leading to output lines like [03/29/2016 21:36:03] DEBUG:pynYNAB:track_append <pynYNAB.schema.budget.Payee object at 0x103d35a20> that are not as useful as they could be.
• I was initially a bit unclear on where I should be looking for the DB file, then realized it's an in-memory DB. So this is more an idea than a problem: being able to configure the DB connection string may be useful, in my case I'd set it to some file on my disk, and save some network traffic.
• Transactions didn't actually show up in the budget. I didn't yet look into what exactly is going wrong, will add more info.

[rienafairefr]

Thanks for taking the time!

• logging being ultra verbose is probably partly amplified by echo= True in sqlalchemy, basically printing all the movements into and out of the db, which is a lot obviously :-) I'll remove the echo 'in production', and maybe tone down the debug statements :+1:
• I remember having the repr from the object's dictionary, I'll reinstate that.
• I think how I will do this is basically having multiple client subclasses, some persisting the db (into a file or into another db specified by the connection string), some not (I'd keep the default in-memory not persisting I think)
• Tests were passing AFAIK, testing addition and deletion of transactions. I'll investigate, thanks !

Cheers,

[abesto]

Glad to hear some of this is useful :)

Started tracking down why transactions were not added with the CSV import, got as far as verifying that nheadears works in the schema, then started getting this exception on all runs. Only on the db branch, not on master:

Traceback (most recent call last):
  File "/Users/abesto/playground/otp2ynab-db-test/virtualenv/src/pynynab/pynYNAB/scripts/csvimport.py", line 170, in <module>
    csvimport_main()
  File "/Users/abesto/playground/otp2ynab-db-test/virtualenv/src/pynynab/pynYNAB/scripts/csvimport.py", line 46, in csvimport_main
    do_csvimport(args)
  File "/Users/abesto/playground/otp2ynab-db-test/virtualenv/src/pynynab/pynYNAB/scripts/csvimport.py", line 166, in do_csvimport
    client.add_transactions(transaction_list(args,client))
  File "/Users/abesto/playground/otp2ynab-db-test/virtualenv/src/pynynab/pynYNAB/scripts/csvimport.py", line 130, in transaction_list
    entities_subcategory_id = get_subcategory(client,master,sub).id
  File "/Users/abesto/playground/otp2ynab-db-test/virtualenv/src/pynynab/pynYNAB/scripts/common.py", line 37, in get_subcategory
    s for s in client.budget.be_subcategories
  File "/Users/abesto/playground/otp2ynab-db-test/virtualenv/src/pynynab/pynYNAB/scripts/common.py", line 38, in <genexpr>
    if s.master_category.name == master_category_name and s.name == subcategory_name)
AttributeError: 'NoneType' object has no attribute 'name'

My best guess is something changed on the server side that breaks the relationships between the master and the subcategories, but I lack a bunch of domain knowledge to verify that.

[rienafairefr]

Thanks for the feedback! I guess it's a matter of the database being all populated on the client side, basically this part of the code is searching for a subcategory in the client database, and probably some Subcategory objects (maybe all), don't yet have the entities_master_category_id field populated or linked, so the s.master_category (which loads the "parent" MasterCategory) returns None. I've just pushed a change that might prevent the system crashing when searching a subcategory, give it a try :-)

[abesto]

Hm, now I'm getting this:

ERROR:pynYNAB:Couldnt find this master:subcategory True Expenses:Medical

In debug mode there's this line (way before the error), I guess this shows that at one point or another we knew about the category:

DEBUG:pynYNAB:track_append {'entities_account_id': None, 'name': 'Medical', 'note': None, 'type': 'DFT', 'goal_creation_month': None, 'entities_master_category_id': '6c1839a9-eb24-4319-b6fa-4398f6d3ad9b', 'monthly_funding': 0, 'target_balance_month': None, 'internal_name': None, 'id': '7bce7602-b1b0-43b5-bed5-39dfc0fd0ef8', 'sortable_index': 190000, 'target_balance': 0.0, 'is_hidden': None, 'goal_type': None, 'is_tombstone': False}

Is it possible there's something strange with this budget? (This would be so much easier if there was a test account to reproduce issues. At some point I might subscribe for one if it makes sense)

[rienafairefr]

Well, if this budget doesn't contain a category named True Expenses with a subcategory Medical, then the message makes sense. Does 6c1839a9-eb24-4319-b6fa-4398f6d3ad9b correspond to a master_category named "True Expenses" ?

Do you think the csv import should create the category if it does not exist ? Or maybe erroring out like this is better ?

[abesto]

The master category is indeed "True Expenses", so to the best of my knowledge the category exists.

I think not creating categories on import is a sensible default, it's what I expected when first starting to use the importer. Maybe an option down the line makes sense.

[rienafairefr]

It's really weird, AFAIK the category import is working, look at the test test_import_categoryschema or recently added test_import_categoryschema_bad_category, they both pass.

I've just committed some debug statements to look at the be_subcategories and be_master_categories collections when searching for a subcategory, maybe it'll help see the problem

[abesto]

Alright, gonna assume my test budget is messed up. Will create a new one and see what's what. (Expect a few days delay). I'm out of comments about the branch by the way :)

Thanks!

[rienafairefr]

Thanks for your feedback :-) I'll probably take the plunge and merge the branch very soon, I'm closing this issue for now, don't hesitate to open another one if problems arise. Cheers