sodadata / soda-sql

Soda SQL and Soda Spark have been deprecated and replaced by Soda Core. docs.soda.io/soda-core/overview.html
https://docs.soda.io/
Apache License 2.0
60 stars 15 forks source link

Getting error after executing soda scan comamnd for soda-mysql #209

Open danilyef opened 2 years ago

danilyef commented 2 years ago

I got the following error after executing soda scan command:

soda_sql soda scan warehouse.yml tables/table_name.yml
  | 2.1.3
  | Scanning tables/some_table.yml ...
  | There is no value specified for valid_values for column source
  | There is no value specified for valid_min for column source
  | There is no value specified for valid_max for column source
  | # _do_auth(): user: root
  | # _do_auth(): self._auth_plugin: 
  | new_auth_plugin: mysql_native_password
  | No Soda Cloud account configured
  | Executing SQL query: 
SELECT column_name, data_type, is_nullable 
FROM information_schema.columns 
WHERE lower(table_name) = 'some_table' 
  AND table_schema = 'some_schema'
  | SQL took 0:00:00.011701
  |   id (int) not null
  |   type (varchar) not null
  |   userId (int) not null
  |   pushToken (varchar) not null
  |   source (varchar) not null
  |   createDate (int) not null
  |   modifyDate (int) not null
  | 7 columns:
  | Query measurement: schema = [{'name': 'id', 'type': 'int', 'dataType': 'int', 'nullable': False, 'logicalType': 'number', 'semanticType': 'number'}, {'name': 'type', 'type': 'varchar', 'dataType': 'varchar', 'nullable': False, 'logicalType': 'text', 'semanticType': 'text'}, {'name': 'userId', 'type': 'int', 'dataType': 'int', 'nullable': False, 'logicalType': 'number', 'semanticType': 'number'}, {'name': 'pushToken', 'type': 'varchar', 'dataType': 'varchar', 'nullable': False, 'logicalType': 'text', 'semanticType': 'text'}, {'name': 'source', 'type': 'varchar', 'dataType': 'varchar', 'nullable': False, 'logicalType': 'number', 'semanticType': 'number'}, {'name': 'createDate', 'type': 'int', 'dataType': 'int', 'nullable': False, 'logicalType': 'number', 'semanticType': 'number'}, {'name': 'modifyDate', 'type': 'int', 'dataType': 'int', 'nullable': False, 'logicalType': 'number', 'semanticType': 'number'}]
  | Exception during aggregation query
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.8/site-packages/sodasql/scan/scan.py", line 254, in _query_aggregations
    fields.append(dialect.sql_expr_count_conditional(scan_column.non_missing_condition, column_name, ))
TypeError: sql_expr_count_conditional() takes 2 positional arguments but 3 were given
  | Test error for "row_count > 0": name 'row_count' is not defined
  | Test error for "invalid_percentage == 0": name 'invalid_percentage' is not defined
  | Executed 1 queries in 0:00:00.041383
  | Scan summary ------
  | 1 measurements computed
  | 2 tests executed
  | 2 of 2 tests failed:
  |   Test test(row_count > 0) failed with measurements null
  |   Test column(source) test(invalid_percentage == 0) failed with measurements null
  | Errors occurred!
  |   [error] Exception during aggregation query
  |   [test_execution_error] Test "row_count > 0" failed
  |   [test_execution_error] Test "invalid_percentage == 0" failed
  | Exiting with code 1
  | Starting new HTTPS connection (1): collect.soda.io:443
  | https://collect.soda.io:443 "POST /v1/traces HTTP/1.1" 200 0

table_name.yml

table_name: table_name
metrics:
  - row_count
  - missing_count
  - missing_percentage
  - values_count
  - values_percentage
  - invalid_count
  - invalid_percentage
  - valid_count
  - valid_percentage
  - avg_length
  - max_length
  - min_length
  - avg
  - sum
  - max
  - min
  - stddev
  - variance
tests:
  - row_count > 0
columns:
  source:
    valid_format: number_whole
    tests:
      - invalid_percentage == 0

warehouse.yml

name: soda_sql
connection:
  type: mysql
  host: my_host
  port: '3306'
  username: env_var(MYSQL_USERNAME)
  password: env_var(MYSQL_PASSWORD)
  database: my_database

OS: macOs Python Version: Python 3.8.12 pip Version: 22.0.3 Soda CLI version:2.1.3 Warehouse Type: mysql Mysql version: 8.0.26