data-mie / dbt-profiler

Macros for generating dbt model data profiles
Apache License 2.0
81 stars 33 forks source link

Add a median measure #75

Closed toohsk closed 1 year ago

toohsk commented 1 year ago

Description & motivation

Hi @stumelius,

I would like to add a median measurement in profiler because it would be a good indicator when you want to know the skewness simply.
Feel free to discard the PR if you think it not so useful.

Checklist

stumelius commented 1 year ago

@toohsk The code looks good, great work! There are errors when running the models in BigQuery:

12:30:32  Database Error in model profile (models/profile.sql)
12:30:32    SELECT list expression references column id which is neither grouped nor aggregated at [36:27]

To me it looks like the issue is with using the percentile_dict window function in a group by select statement. What do you think?

toohsk commented 1 year ago

@stumelius Thanks for checking this PR 😄 ! I hadn't tested BigQuery, so I didn't realize this error, but it seems you're right. I fixed the query for bigquery so could you run it on my behalf?

stumelius commented 1 year ago

@toohsk All the automated tests are now passing. Well done and thanks for your contribution @toohsk :) Merging this now

stumelius commented 1 year ago

@toohsk Here's the release https://github.com/data-mie/dbt-profiler/releases/tag/0.7.0

toohsk commented 1 year ago

Thank you so much @stumelius ! I'm happy to have contributed to this project.