sodadata / soda-sql

Soda SQL and Soda Spark have been deprecated and replaced by Soda Core. docs.soda.io/soda-core/overview.html
https://docs.soda.io/
Apache License 2.0
59 stars 16 forks source link

Make schema/dataset property part of table scan specification #180

Open sebastianvillarroel opened 2 years ago

sebastianvillarroel commented 2 years ago

Is your feature request related to a problem? Please describe. Use case: When having multiple datasets or schemas, the mono-configuration of the warehouse.yml expects that all tables are available in one schema/dataset. The idea is that the property should be part of each table YML specification. The below is my attempt to change it to a placeholder schema.

(soda-sql)seba% soda scan warehouse.yml ./tables/code_to_division_mapping.yml

google.api_core.exceptions.NotFound: 404 Not found: Table umg-de:sodasql.code_to_division_mapping was not found in location US

(job ID: xxxxxxxxx)

     -----Query Job SQL Follows-----       

    |    .    |    .    |    .    |
   1:SELECT 
   2:  COUNT(*) 
   3:FROM `sodasql.code_to_division_mapping`
    |    .    |    .    |    .    |
  | Test error for "row_count > 0": name 'row_count' is not defined
  | Executed 1 queries in 0:00:01.690635
  | Scan summary ------
  | 1 measurements computed
  | 1 tests executed
  | 1 of 1 tests failed:
  |   Test test(row_count > 0) failed with measurements null
  | Errors occurred!
  |   [error] Exception during aggregation query
  |   [test_execution_error] Test "row_count > 0" failed
  | Exiting with code 1
  | Starting new HTTPS connection (1): collect.soda.io:443
  | https://collect.soda.io:443 "POST /v1/traces HTTP/1.1" 200 0

Describe the solution you'd like property schema/dataset should be at table.yml specification and not at warehouse.yml.

Soda SQL Version: 2.1.2

sebastianvillarroel commented 2 years ago

@vijaykiran are you Ok if I propose a PR?

vijaykiran commented 2 years ago

@sebastianvillarroel Sorry for the delay .. can you please write a short summary how it would change the yml ? I'd say this should be "optional" config .. not to cause any issues with existing ymls.