sodadata / soda-sql

Soda SQL and Soda Spark have been deprecated and replaced by Soda Core. docs.soda.io/soda-core/overview.html
https://docs.soda.io/
Apache License 2.0
61 stars 17 forks source link

Cannot filter without variable #196

Closed thiagodeschamps closed 2 years ago

thiagodeschamps commented 2 years ago

Describe the bug: Soda scan fails when filter is used and no variables are passed, even thought the filter itself does not require any variable. To Reproduce Steps to reproduce the behavior:

  1. Create a scan.yml with a filter clause, like date(process_time) = '2022-01-20'
  2. Run soda scan warehouse.yml scan.yml
  3. Obtain the following error:
thiago@BRSPOBITANL1756:~/Documents/soda/bq$ soda scan warehouse.yml tables/demo.yml 
  | 2.1.2
  | Scanning tables/demo.yml ...
  | No Soda Cloud account configured
  | Scan failed: No variables provided while filter "DATE(process_time) = '2020-09-02'" specified
Traceback (most recent call last):
  File "/home/thiago/.local/lib/python3.8/site-packages/sodasql/cli/cli.py", line 459, in scan
    scan = scan_builder.build(offline=offline)
  File "/home/thiago/.local/lib/python3.8/site-packages/sodasql/scan/scan_builder.py", line 175, in build
    return Scan(warehouse=warehouse,
  File "/home/thiago/.local/lib/python3.8/site-packages/sodasql/scan/scan.py", line 82, in __init__
    raise RuntimeError(f'No variables provided while filter "{str(scan_yml.filter)}" specified')
RuntimeError: No variables provided while filter "DATE(process_time) = '2020-09-02'" specified
  | If you think this is a bug in Soda SQL, please open an issue at: https://github.com/sodadata/soda-sql/issues/new/choose
  | Exiting with code 1
  | Starting new HTTPS connection (1): collect.soda.io:443
  | https://collect.soda.io:443 "POST /v1/traces HTTP/1.1" 200 0

Context

table_name: bob2_pedido_de_venda_cabecalho 
filter: DATE(process_time) = '2022-01-20' 
metrics: 
- row_count 
columns: 
  csn: 
    tests: 
      - row_count > 0 
  - pedc_dt_envio_rastreamento: 
    - tests: 
      - row_count > 0

OS: Ubuntu 20.04.3 LTS Python Version: 3.8.10 Soda SQL Version: soda-sql-bigquery (2.1.2) Warehouse Type: BigQuery

lucas-houles commented 2 years ago

Hi @thiagodeschamps, I've just checked the doc and the variable is mandatory when you apply a filter: https://docs.soda.io/soda-sql/filtering.html I'm wondering if adding a feature to handle static filter (without variable) could be a good thing, what do you think @vijaykiran ?

vijaykiran commented 2 years ago

Yes, that makes sense!