TITLE :Incompatible with Airflow due to MarkupSafe conflict IN SODA SQL
Describe the bug
soda-core:2.2.2 is incompatible with Airflow:2.4.1 due to MarkupSafe version conflict
... soda-sql-bigquery (>=2.1.6,<3.0.0) requires markupsafe (2.0.1).
And because apache-airflow (2.4.1) requires MarkupSafe (>=2.1.1) (2), apache-airflow (2.4.1) is incompatible with soda-sql-bigquery (>=2.1.6,<3.0.0)
To Reproduce
Steps to reproduce the behavior:
create pyproject.toml with dependencies to
airflow = "2.4.1"
soda-sql-bigquery = "^2.1.6"
any other soda-sql-xxx with same version
run poetry update sod-sql-bigquery ...
observe conflict
OS: MacOs
Python Version: 3.9
Soda SQL Version: 2.2.2
TITLE : Applying a filter to a row_count 'same as' comparison aggregation IN SODA SQL
Describe the bug
Applying a filter to a row_count check only applies to the metric, not the external column we're evaluating against.
I can see why this is expected behaviour, but I would like a way to optionally apply the filter to the comparison aggregation.
To Reproduce
Steps to reproduce the behavior:
Create a new test in scan.yml
filter foo [hourly]:
where: from_iso8601_timestamp(time) between DATE_ADD('hour', -1, NOW()) AND NOW()
checks for foo [hourly]:
row_count same as bar
Run soda scan with -V verbose flag
Observe the "otherRowcount" SQL doesn't include the filter and so counts all rows
Soda Core 3.0.4
Reading configuration file "configuration.yml"
Reading SodaCL file "checks_foo.yml"
Scan execution starts
Query datalake_dev.foo[hourly].aggregation[0]:
SELECT
COUNT()
FROM datalake.foo
WHERE from_iso8601_timestamp(time) between DATE_ADD('hour', -1, NOW()) AND NOW()
Query datalake_dev.bar.aggregation[0]:
SELECT
COUNT()
FROM datalake.bar
Scan summary:
2/2 queries OK
datalake_dev.foo[hourly].aggregation[0] [OK] 0:00:01.819557
datalake_dev.bar.aggregation[0] [OK] 0:00:01.262278
1/1 check FAILED:
foo [hourly] in datalake_dev
row_count same as bar [FAILED]
value: -12047
rowCount: 9
otherRowCount: 12056
Oops! 1 failures. 0 warnings. 0 errors. 0 pass.
Context
As above
OS: MAC
Python Version: 3.8.9
Soda SQL Version: 3.0.4
Warehouse Type: AWS Athena
TITLE :Incompatible with Airflow due to MarkupSafe conflict IN SODA SQL
Describe the bug soda-core:2.2.2 is incompatible with Airflow:2.4.1 due to MarkupSafe version conflict
... soda-sql-bigquery (>=2.1.6,<3.0.0) requires markupsafe (2.0.1). And because apache-airflow (2.4.1) requires MarkupSafe (>=2.1.1) (2), apache-airflow (2.4.1) is incompatible with soda-sql-bigquery (>=2.1.6,<3.0.0) To Reproduce Steps to reproduce the behavior:
create pyproject.toml with dependencies to airflow = "2.4.1" soda-sql-bigquery = "^2.1.6" any other soda-sql-xxx with same version run poetry update sod-sql-bigquery ... observe conflict OS: MacOs Python Version: 3.9 Soda SQL Version: 2.2.2
TITLE : Applying a filter to a row_count 'same as' comparison aggregation IN SODA SQL
Describe the bug
Applying a filter to a row_count check only applies to the metric, not the external column we're evaluating against.
I can see why this is expected behaviour, but I would like a way to optionally apply the filter to the comparison aggregation.
To Reproduce
Steps to reproduce the behavior:
Create a new test in scan.yml filter foo [hourly]: where: from_iso8601_timestamp(time) between DATE_ADD('hour', -1, NOW()) AND NOW()
checks for foo [hourly]:
Observe the "otherRowcount" SQL doesn't include the filter and so counts all rows
Soda Core 3.0.4 Reading configuration file "configuration.yml" Reading SodaCL file "checks_foo.yml" Scan execution starts Query datalake_dev.foo[hourly].aggregation[0]: SELECT COUNT() FROM datalake.foo WHERE from_iso8601_timestamp(time) between DATE_ADD('hour', -1, NOW()) AND NOW() Query datalake_dev.bar.aggregation[0]: SELECT COUNT() FROM datalake.bar Scan summary: 2/2 queries OK datalake_dev.foo[hourly].aggregation[0] [OK] 0:00:01.819557 datalake_dev.bar.aggregation[0] [OK] 0:00:01.262278 1/1 check FAILED: foo [hourly] in datalake_dev row_count same as bar [FAILED] value: -12047 rowCount: 9 otherRowCount: 12056 Oops! 1 failures. 0 warnings. 0 errors. 0 pass.
Context As above
OS: MAC Python Version: 3.8.9 Soda SQL Version: 3.0.4 Warehouse Type: AWS Athena