[FEATURE] Use of additional filters for limiting the data in stage macros

[yedu1985]

Is your feature request related to a problem? Please describe.

Pulling the incremental data from the PSA area by comparing the Last_record_load_date/Specific date by applying the filter in the stage macro

Describe the solution you'd like A clear and concise description of what you want to happen.

where true {# minimize data usage for sandbox/cicd #} {% if not target.name in ['dev', 'uat', 'prod'] or env_var( 'DBT_CICD_RUN', 'false' ) == 'true' %} and date(updated_on) >= date_sub(current_date(), interval 1 month) {% endif %}

{% if is_incremental() %}
    {# Note:

THE updated_on that used for incremental loading ),

    and date(updated_on) >= coalesce(
        (
            select date(max(updated_on))
            from {{ this }}
            where rsrc not in ('ERROR', 'SYSTEM')
        ),
        '1900-01-01'
    )
{% endif %}

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

AB#5358

[DVAlexHiggs]

Hi @yedu1985 Thanks for this. We're actually currently working on new loading features, this being one of them. v0.10.x releases are bringing Quality of life improvements for loading, the first of which was solving the long-standing limitation on Satellite loading.

In the meantime I recommend you look at incremental predicates, a dbt feature that already exists to do just this!