search

sodadata / soda-sql

Soda SQL and Soda Spark have been deprecated and replaced by Soda Core. docs.soda.io/soda-core/overview.html
https://docs.soda.io/
Apache License 2.0
59 stars 16 forks source link

`dbt_cloud_api_token` in warehouse yaml breaks `soda scan` #192

Closed bastienboutonnet closed 2 years ago

bastienboutonnet commented 2 years ago

Describe the bug when the dbt_cloud_api_token key is added into the warehouse.yaml file, soda scan fails because the parsing of the warehouse yaml encounters a key that is not valid.

We probably want to make sure that this key is ok with soda scan when the dbt package is installed or just fine full stop

version: latest main

  [warning] (dbt_cloud_api_token) Invalid key in telemetry/warehouse.yml : dbt_cloud_api_token
  | If you think this is a bug in Soda SQL, please open an issue at: https://github.com/sodadata/soda-sql/issues/new/choose
  | Exiting with code 1
  | Starting new HTTPS connection (1): collect.soda.io:443
  | https://collect.soda.io:443 "POST /v1/traces HTTP/1.1" 200 0

1   | 2.1.2
  | Scanning /Users/bastienboutonnet/repos/intelligence_analytics/soda_checks/telemetry/tables/soda_sql_datasource_sessions.yml ...
  | Injecting ssl_wrap_socket_with_ocsp
  | cache directory: /Users/bastienboutonnet/Library/Caches/Snowflake
  | Failed to import pyarrow. Cannot use pandas fetch API
  | Failed to import ArrowResult. No Apache Arrow result set format can be used. ImportError: No module named 'snowflake.connector.arrow_result'
  | Invalid key in telemetry/warehouse.yml : dbt_cloud_api_token
  | Scan failed: telemetry/warehouse.yml configuration errors:
  [warning] (dbt_cloud_api_token) Invalid key in telemetry/warehouse.yml : dbt_cloud_api_token
Traceback (most recent call last):
  File "/Users/bastienboutonnet/repos/soda-sql/core/sodasql/cli/cli.py", line 460, in scan
    scan = scan_builder.build(offline=offline)
  File "/Users/bastienboutonnet/repos/soda-sql/core/sodasql/scan/scan_builder.py", line 165, in build
    parser.assert_no_warnings_or_errors()
  File "/Users/bastienboutonnet/repos/soda-sql/core/sodasql/scan/parser.py", line 118, in assert_no_warnings_or_errors
    raise AssertionError(f'{self.description} configuration errors: \n  '
AssertionError: telemetry/warehouse.yml configuration errors:
  [warning] (dbt_cloud_api_token) Invalid key in telemetry/warehouse.yml : dbt_cloud_api_token
vijaykiran commented 2 years ago

For now, I'm changing it not to raise an exception so Scan can continue when an unknown key is specified. Checking which keys are needed at the parsing level with packages installed might need bigger changes to parsing logic, I'll defer it to later.