sodadata / soda-sql

Soda SQL and Soda Spark have been deprecated and replaced by Soda Core. docs.soda.io/soda-core/overview.html
https://docs.soda.io/
Apache License 2.0
61 stars 16 forks source link

A bug when scanning mysql data #199

Open sanazmather opened 2 years ago

sanazmather commented 2 years ago

Describe the bug A clear and concise description of what the bug is and what you expected instead. I installed mysql data source and I want to scan a table. the yml file for the table is correct but it gives error. This is the error I get: | executing {'type': 'sodaSqlScanEnd', 'scanReference': '8252a679-f9d8-448c-ba62-6738b26f4e47', 'errors': [{'type': 'error', 'message': 'Exception during aggregation query', 'exception': 'sql_expr_count_conditional() takes 2 positional arguments but 3 were given'}, {'type': 'test_execution_error', 'message': 'Test "avg" failed', 'exception': "name 'avg' is not defined"}]}

To Reproduce Steps to reproduce the behavior:

  1. Create a new test in scan.yml
    table_name: standard_module_daily
    metrics:
    - row_count
    - missing_count
    - missing_percentage
    - values_count
    - values_percentage
    - invalid_count
    - invalid_percentage
    - valid_count
    - valid_percentage
    - avg_length
    - max_length
    - min_length
    - avg
    - sum
    - max
    - min
    - stddev
    - variance
    columns:
    visits:
    tests:
      - max
      - min
      - avg
  2. Run soda scan warehouse.yml tables/standard_module_daily.yml 3 Error
    
    Traceback (most recent call last):
    File "/usr/local/lib/python3.9/site-packages/sodasql/scan/scan.py", line 254, in _query_aggregations
    fields.append(dialect.sql_expr_count_conditional(scan_column.non_missing_condition, column_name, ))
    TypeError: sql_expr_count_conditional() takes 2 positional arguments but 3 were given
    | Fetching custom metrics with scanReference: 8252a679-f9d8-448c-ba62-6738b26f4e47
    | Starting new HTTPS connection (1): cloud.soda.io:443
    | https://cloud.soda.io:443 "POST /api/query HTTP/1.1" 200 2
    | Test column(visits) test(max) passed with measurements {}
    | Test column(visits) test(min) passed with measurements {}
    | Test error for "avg": name 'avg' is not defined
    | Soda Cloud scan send test results
    | executing {'type': 'sodaSqlScanTestResults', 'scanReference': '8252a679-f9d8-448c-ba62-6738b26f4e47', 'testResults': [{'id': '{"column":"visits","express) test(max)', 'expression': 'max', 'columnName': 'visits', 'source': 'soda-sql', 'passed': True, 'skipped': False, 'values': {}}, {'id': '{"column":"visits",n(visits) test(min)', 'expression': 'min', 'columnName': 'visits', 'source': 'soda-sql', 'passed': True, 'skipped': False, 'values': {}}, {'id': '{"column":": 'column(visits) test(avg)', 'expression': 'avg', 'columnName': 'visits', 'source': 'soda-sql', 'error': "name 'avg' is not defined"}]}
    | Starting new HTTPS connection (1): cloud.soda.io:443
    | https://cloud.soda.io:443 "POST /api/command HTTP/1.1" 200 16
    | Executed 1 queries in 0:00:02.449222
    | Soda Cloud scan end with errors
    | executing {'type': 'sodaSqlScanEnd', 'scanReference': '8252a679-f9d8-448c-ba62-6738b26f4e47', 'errors': [{'type': 'error', 'message': 'Exception during a arguments but 3 were given'}, {'type': 'test_execution_error', 'message': 'Test "avg" failed', 'exception': "name 'avg' is not defined"}]}
    | Starting new HTTPS connection (1): cloud.soda.io:443
    | https://cloud.soda.io:443 "POST /api/command HTTP/1.1" 200 2
    | Scan summary ------
    | 1 measurements computed
    | 3 tests executed
    | 1 of 3 tests failed:
    |   Test column(visits) test(avg) failed with measurements null
    | Errors occurred!
    |   [error] Exception during aggregation query
    |   [test_execution_error] Test "avg" failed
    | Exiting with code 1


**Context**
__Include your scan.yml or warehouse.yml when relevant__
warehouse.yml
name: mysql
connection:
  type: mysql
  host: 
  port: '3306'
  username: 
  password: 
  database: 

**OS**: 
**Python Version**: 3.9
**Soda SQL Version**: 2.1.3
**Warehouse Type**: mysql