This PR is a follow on from #113 to do all database-touching unit tests in Postgres as well as sqlite.
Because this removes the sqlite testing database, it closes #34. It does not fill the entire wishlist of #34, in that it doesn't support having lots of different database structures. Long-term, I think the best approach is to implement #7 and then have all tests run against the latest database structure, except for unit tests of the migration tools (which will need other means of building the test databases.)
It also converts the unit tests (and thus CI) to use the version of dbprocessing in the build directory...this lets you test updates without installing them.
The biggest commit is the one that converts the RBSP_mageis database to a JSON file instead of sqlite database. This did a huge amount of work in one commit, because otherwise there would be multiple checked-in versions of the binary test database file, making the repository much larger (this is what we're trying to avoid with this PR). So this commit:
Cuts down the size of the magEIS test database
Changes tests based on the new database contents
Updates tests to load data from a JSON file, allowing postgresql testing.
I found this approach makes the sqlite tests slightly slower (about a minute instead of 30s on my machine) and the Postgresql tests are quite a bit slower yet (about 5 minutes for the whole suite, on my machine). Reducing the size of the test database further didn't affect this much; it seems to be primarily in the table setup and teardown. But that can potentially be looked at later...it's not a ridiculous length of time for the test suite.
Fortunately there were only a couple of Postgresql-related bugs found and fixed by doing these tests.
The use of Postgresql databases is added to the documentation.
PR Checklist
[X] Pull request has descriptive title
[X] Pull request gives overview of changes
[X] New code has inline comments where necessary
[X] (N/A) Any new modules, functions or classes have docstrings consistent with dbprocessing style
[X] Major new functionality has appropriate Sphinx documentation
[X] (N/A) Added an entry to release notes if fixing a major bug or providing a major new feature
[X] New features and bug fixes should have unit tests
[X] Relevant issues are linked in the description (use Closes # if this PR closes the issue, or some other reference, such as See # if it is related in some other way)
This PR is a follow on from #113 to do all database-touching unit tests in Postgres as well as sqlite.
Because this removes the sqlite testing database, it closes #34. It does not fill the entire wishlist of #34, in that it doesn't support having lots of different database structures. Long-term, I think the best approach is to implement #7 and then have all tests run against the latest database structure, except for unit tests of the migration tools (which will need other means of building the test databases.)
It also converts the unit tests (and thus CI) to use the version of dbprocessing in the
builddirectory...this lets you test updates without installing them.The biggest commit is the one that converts the RBSP_mageis database to a JSON file instead of sqlite database. This did a huge amount of work in one commit, because otherwise there would be multiple checked-in versions of the binary test database file, making the repository much larger (this is what we're trying to avoid with this PR). So this commit:
I found this approach makes the sqlite tests slightly slower (about a minute instead of 30s on my machine) and the Postgresql tests are quite a bit slower yet (about 5 minutes for the whole suite, on my machine). Reducing the size of the test database further didn't affect this much; it seems to be primarily in the table setup and teardown. But that can potentially be looked at later...it's not a ridiculous length of time for the test suite.
Fortunately there were only a couple of Postgresql-related bugs found and fixed by doing these tests.
The use of Postgresql databases is added to the documentation.
PR Checklist
Closes #if this PR closes the issue, or some other reference, such asSee #if it is related in some other way)