fix: Fixing multiple function call, to get schema in custom-query validation for Hive

GoogleCloudPlatform / professional-services-data-validator

Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match

Apache License 2.0

399 stars 114 forks source link

Closed piyushsarraf closed 1 month ago

piyushsarraf commented 3 months ago

This PR fixes following:

Multiple function call to _get_schema_using_query function for custom-query validation.
Use custom-query as is with an addition of LIMIT function, instead of having custom-query as a subquery, earlier it was little cost-intensive to customer, if custom-query returns huge amount of rows.
Changing fetchall() to fetchone(), just to improve the performance, as we only require the column headers.

piyushsarraf commented 3 months ago

/gcbrun

piyushsarraf commented 3 months ago

/gcbrun

piyushsarraf commented 1 month ago

/gcbrun

piyushsarraf commented 1 month ago

/gcbrun

helensilva14 commented 1 month ago

Hi all! Just to confirm: there's really no opened issues to be fixed by this PR?

piyushsarraf commented 1 month ago

Hi all! Just to confirm: there's really no opened issues to be fixed by this PR?

@helensilva14, changes are made to solve another branch of this issue.

To decrease multiple function call to _get_schema_using_query
Avoiding nested query to get schema of a custom-query, which can improve cost implied to user, if the custom-query table is large.

piyushsarraf commented 1 month ago

/gcbrun

helensilva14 commented 1 month ago

/gcbrun