microsoft / dbt-fabric

MIT License
79 stars 28 forks source link

`calc_batch_size` exceeding allowed parameters #101

Closed colin-rogers-dbt closed 11 months ago

colin-rogers-dbt commented 11 months ago

When running dbt seed I'm unable to load a 7 column 2000 row CSV file as it exceeds # of parameters.

Current workaround is to override calc_batch_size macro and add a -1 so it's:

{% macro calc_batch_size(num_columns) %}
    {#
        SQL Server allows for a max of 2100 parameters in a single statement.
        Check if the max_batch_size fits with the number of columns, otherwise
        reduce the batch size so it fits.
    #}
    {% set max_batch_size = get_batch_size() %}
    {% set calculated_batch = (2100 / num_columns)-1 | int %}
    {% set batch_size = [max_batch_size, calculated_batch] | min %}
    {{ return(batch_size) }}
{%  endmacro %}
prdpsvs commented 11 months ago

Addressed this release 1.7.2

tkiehn commented 4 months ago

The | int filter seems to only apply to the -1 and not the whole result which caused errors in my project when loading seeds.

I opened a PR that addresses the issue: #207