Store varmetric values in a table

[timstaley]

Rebased version of https://github.com/transientskp/tkp/pull/469 (Actually the rebase was non-trivial, which is why I didn't force-push over the old branch. Think I got it, though.)

Hey @gijzelaerr , took a look at this, looks good (I think! It's taken me all day just to get back up to speed, so I've not dived deep into the queries).

Two concrete suggestions:

• If we're switching to sqlalchemy in general, then probably we should rename tkp.db.sql.alchemy to something more specific now - tkp.db.sql.varmetric? Similarly for 'tests/test_database/test_alchemy'
• Would be good to have an entry in the docs for the varmetric table, explaining what the column values represent, alternatively put that in the relevant docstring and figure out how to pull it into the docs. (Actually, now the db-models are in Python we could probably use docstrings throughout, but that's a bigger job.)

And some comments:

• How much work will it be to fix-up Banana to match - I guess the varmetric table just behaves as the old augmented runcat view, so not much?
• In the long run, I'm not sure it's worth having a wrapper function (tkp.tkp.steps.varmetric.execute_store_varmetric). Probably clearer to just have session management out in the open in tkp.main? Though I guess it's a bit awkward right now while we're still mixing old code. Leave it for now I guess, but we should have a think about whether we're going to have session management visible at the top-level, or replace the home-grown Database singleton with a sqlalchemy sessionmaker.
• From a quick glance, it looks like there's a lot of overlapping functionality in tkp.testutil.alchemy and the older tkp.testutil.db_subs. I don't blame you for avoiding that code - it was written in a hurry and could probably use some love - but it does quite a lot of work behind the scenes in setting up mock data. Probably we want to merge and improve this functionality, though it's a job in itself to do so.
• I'm a bit worried that we don't actually test the varmetrics very carefully, though that's been the case since we first implemented the 'augmented running catalog view' while crunching for release 2.0. Ideally we should test them against a known dataset, but actually testing properly needs quite complex mock-data -> goes back to previous point about cleaning up mock-data functions.

Probably some of this should go in an issue... tomorrow.

[gijzelaerr]

1> we discussed this in oxford, and agreed that for now the name 'sqlalchemy' would be best, and we split it up later when the code gets more specialised.

2> yes, the only thing missing was the doc. Indeed I wanted to switch to docstrings, but that is just too much work for now.

3> https://github.com/transientskp/banana/pull/101

4> Yes, the whole wrapper stuff is a bit clunky. Wanted to replace that one day when we would refactor the whole main loop.

5>I couldn't really understand what was happening, started to replicate the behaviour in SQLAlchemy to get myself familiar with the technology and it grew from there. Probably yours is more advanced.

6> I'm worried also we don't test it enough, although this is already more tested than the augmented runningcatalog view. I wasn't really sure what would be a good set of tests, and there was nobody around to discuss this with :'(

[gijzelaerr]

test this please

[gijzelaerr]

@timstaley what do you think is the status of this now?