This PR modifies the Spark extractor to allow extracting from the iasWorld test environment. Previously, we only extracted from the production environment.
Data pulled from the test environment is put into its own dedicated database in Athena, iasworld_test. Note that the test environment doesn't replicate the production environment; not all tables are available in test, and the most recent year of data is not present.
I've already successfully run this pipeline, so you can find the test data in Athena under iasworld_test.
This PR modifies the Spark extractor to allow extracting from the iasWorld test environment. Previously, we only extracted from the production environment.
This setup is necessary to facilitate https://github.com/ccao-data/enterprise-intelligence/issues/131, which requires pulling data that is only loaded into the test environment.
Data pulled from the test environment is put into its own dedicated database in Athena,
iasworld_test. Note that the test environment doesn't replicate the production environment; not all tables are available in test, and the most recent year of data is not present.I've already successfully run this pipeline, so you can find the test data in Athena under
iasworld_test.