GoogleCloudPlatform / professional-services-data-validator

Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match
Apache License 2.0
407 stars 119 forks source link

Views should not be listed in output of find-tables #1342

Open nj1973 opened 6 hours ago

nj1973 commented 6 hours ago

Test case

Oracle:

create view pso_data_validator.dvt_core_types_vw as
select id,col_int8 from pso_data_validator.dvt_core_types;

Notice dvt_core_types_vw is included in the output:

$ data-validation find-tables --source-conn ora --target-conn ora --allowed-schemas pso_data_validator
[{"schema_name": "pso_data_validator", "table_name": "DVT-IDENTIFIER$_#", "target_schema_name": "pso_data_validator", "target_table_name": "DVT-IDENTIFIER$_#"},
...
{"schema_name": "pso_data_validator", "table_name": "dvt_core_types_vw", "target_schema_name": "pso_data_validator", "target_table_name": "dvt_core_types_vw"}]

PostgreSQL:

create view pso_data_validator.dvt_core_types_vw as
select id,col_int8 from pso_data_validator.dvt_core_types;

Notice dvt_core_types_vw is included in the output:

$ data-validation find-tables --source-conn pg_local --target-conn pg_local --allowed-schemas pso_data_validator
[{"schema_name": "pso_data_validator", "table_name": "dvt_bool", "target_schema_name": "pso_data_validator", "target_table_name": "dvt_bool"},
...
{"schema_name": "pso_data_validator", "table_name": "dvt_core_types_vw", "target_schema_name": "pso_data_validator", "target_table_name": "dvt_core_types_vw"}]

This is not good because most views tend to only perform well when predicates are used on them. For a data validation like DVT executes this could perform very badly and be preblematic for source/target systems.