test dcpy and add PostgresClient

[damonmcc]

related to #192 and #17 resolves #137

refactors and tests some dcpy utils in preparation for building products in a shared persistent postgres DB and expanding the Template DB end-to-end testing

motivation

• to standardize how we build products in our persistent postgres DB, we'd like to use DB schemas to organize and isolate build tables (a DB schema is a group of tables. is this pretty much namespaces?)
• in order to avoid having to specify the schema in all SQL queries (would be a huge pain) we can set the "search path" of a DB connection so that all queries use the specified schema
• currently, our postgres utils use a single environment variable to create a sqlalchemy Engine at the top of the module
  • this forces us to construct the search path in bash in a github action (example) rather than in python
  • our postgres uitls that use the psql CLI need a string rather than an Engine
  • it seems we'd prefer to create and reuse a single sqlalchemy Engine during the lifetime of a build (also seems to be the recommended approach)

changes

• move and call dcpy tests
• fix TDB use of recipes
• add PostgresClient class to manage DB connection and most postgres utils
• add git.run_name git.event_name
• refactor recipes to use PostgresClient
• update the QA tests and page that used deleted postgres utils

[alexrichey]

Apologies, Still reading. Will give this a better look tomorrow morning!

[alexrichey]

@damonmcc What's the rationale for moving dcpy_tests into dcpy? I can think of a few reasons against, the primary one being that the tests shouldn't be part of the API for dcpy, and we wouldn't want our pure module code to have a dependencies upon dev dependencies, ie test frameworks. Also, now any module in dcpy can freely import dcpy.test.* which, while I don't think anybody would do that, it's conventional to make that a little more difficult. Also, you typically will want to easily distribute your package sans tests, (ie in our case, just copying the dcpy folder)

So to pick an outside example, boto3 has a strict separation between the two.

[damonmcc]

@alexrichey appreciate the module vs tests details! do you like the old name of dcpy_test or shortening it to tests?

TLDR: no compelling reason, will move em back!

I felt inclined to put it in the dcpy directory because this repo isn't only dcpy and the only tests in that directory were for dcpy. in the boto3 example, everything in that repo is related to that single python package, so isn't it kinda of boto3/boto3 and boto3/tests?

from a quick look at a random local virtual environment I have, pandas doesn't have a strict separation. I see the tests in my file system. same for packages like jsonschema, scipy, sklearn, matplotlib, numpy, and others

[alexrichey]

@damonmcc I suppose this is all pretty build-system specific, but I think you're touching on something correct though - it would be nice to have the tests grouped with the package, not in the project root. And maybe the stuff @fvankrieken is doing with setuptools would allow us to exclude the dcpy.test folder in builds.

[fvankrieken]

@damonmcc I suppose this is all pretty build-system specific, but I think you're touching on something correct though - it would be nice to have the tests grouped with the package, not in the project root. And maybe the stuff @fvankrieken is doing with setuptools would allow us to exclude the dcpy.test folder in builds.

We could have src and tests within the dcpy folder

[alexrichey]

There's also this approach, which has a somewhat telescoping folder structure, but is nicely structured to group functionality while also preserving namespaces. It would make me a little bit sad to introduce src into our module names, e.g. dcpy.src.utils.*.

But it's neat that we can do this stuff now, via setuptools

[damonmcc]

@alexrichey @fvankrieken

just a ping. hope the new commits cover everything, doesn't seem worth doing a dcpy.src folder change in this PR

probably best to do a squash & merge with this one?