search

GoogleCloudPlatform / bigquery-utils

Useful scripts, udfs, views, and other utilities for migration and data warehouse operations in BigQuery.
https://cloud.google.com/bigquery/
Apache License 2.0
1.07k stars 269 forks source link

making anti pattern opitmization script generic for any input table #399

Closed franklinWhaite closed 3 months ago

franklinWhaite commented 3 months ago
  1. The queries_grouped_by_hash.sql references the query column which is no longer available. I removed that column and renamed the file queries_grouped_by_hash_org.sql
  2. It is common for users that lack org level access to edit the queries_grouped_by_hash.sql to reference INFORMATION_SCHEMA.JOBS instead of INFORMATION_SCHEMA.JOBS_BY_PROJECT. I created queries_grouped_by_hash_project.sql which runs at project level instead of org level.
  3. Some users use the output of queries_grouped_by_hash_project.sql and other prefer to work with viewable_queries_grouped_by_hash.sql. Since these generate tables with different schemas I made the run_anti_pattern_tool.sh script generic so it can run against any table given that necessary arguments are provided
afleisc commented 3 months ago

The only thing I would do is potentially move the scripts/optimization/run_anti_pattern_tool.sh and the antipattern portion of the README under the anti-pattern-recognittion-tool-scripts folder, just so its clear that the run_all_scripts won't execute the antipattern tool as well

franklinWhaite commented 3 months ago

I have moved the run_anti_pattern_tool.sh script to the anti-pattern-recognittion-tool-scripts folder.

Nevertheless if we move antipattern portion of the README under the anti-pattern-recognittion-tool-scripts folder the users might miss the existence of the tool. I would rather have our documentation on how to use the scripts centralized than spread across several READMEs.

I understand this may be confusing, so i slightly changed the README to make it more clear that that the run_all_scripts won't execute the antipattern tool as well.

afleisc commented 3 months ago

LGTM