Currently, in order to write an integration test against one of the engines, it needs to be tacked on to test_integration.py.
This file is starting to get a bit unwieldy:
It's ~2700 lines and growing
Every test using TestContext automatically gets 3 versions: df, query and pyspark. Often you only want one of these, typically query (and almost never pyspark) leading to a bunch of skipping logic within tests if the test_type is not query
Every new test has to go into this file, even if its only for a specific database, which means you have to add skipping logic to ensure its being invoked for the correct database
It sits at the same level in the hierarchy as the unit tests for each engine adapter, making it slightly confusing as to where new tests should go
This PR makes the following changes:
Moves everything related to the engine adapter integration tests into an integration subfolder
Moves the engine adapter / TestContext setup logic into a common place
Demonstrates adding an integration test file scoped to a specific engine (to be extended on in future PR's)
The goal here is to set the foundation for breaking these up further in future. Tests affecting all adapters can stay in test_integration.py and then tests for a specific database can be put in test_integration_<database>.py.
The immediate motivation was needing somewhere to put tests for the upcoming partition management logic in the Athena and Postgres adapters but these changes also give the option of moving eg the pyspark tests out of the main file and into a Spark-specific area
Currently, in order to write an integration test against one of the engines, it needs to be tacked on to
test_integration.py.This file is starting to get a bit unwieldy:
TestContextautomatically gets 3 versions:df,queryandpyspark. Often you only want one of these, typicallyquery(and almost neverpyspark) leading to a bunch of skipping logic within tests if the test_type is notqueryThis PR makes the following changes:
integrationsubfolderThe goal here is to set the foundation for breaking these up further in future. Tests affecting all adapters can stay in
test_integration.pyand then tests for a specific database can be put intest_integration_<database>.py.The immediate motivation was needing somewhere to put tests for the upcoming partition management logic in the Athena and Postgres adapters but these changes also give the option of moving eg the
pysparktests out of the main file and into a Spark-specific area