search

elementary-data / elementary

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
https://www.elementary-data.com/
Apache License 2.0
1.94k stars 165 forks source link

current_length value in split_list_to_chunks! #1736

Closed Smitspeed29 closed 2 weeks ago

Smitspeed29 commented 2 weeks ago

Describe the bug Looks like the value of current_length not getting set properly in split_list_to_chunks.sql

Reason for flagging, we are unable to split the query properly wrt query_max_size

https://github.com/elementary-data/dbt-data-reliability/blob/master/macros/utils/list_utils/split_list_to_chunks.sql

Also I am new to Jinja, so If I said something stupid please ignore. Thanks

MikaKerman commented 2 weeks ago

Hi @Smitspeed29! To better understand what's going on, could you let me know where you noticed that current_length isn’t being set correctly? Any specific part of the code execution or output where this issue shows up would be really helpful.

Niteeshkanungo commented 2 weeks ago

@MikaKerman I think we found the bug, the chunk length is not set properly with iterations, wanted to see if this PR can be reviewed and tested? Link - https://github.com/elementary-data/dbt-data-reliability/pull/763

MikaKerman commented 2 weeks ago

Closing since https://github.com/elementary-data/dbt-data-reliability/pull/765 was merged.