search

fal-ai / dbt-fal

do more with dbt. dbt-fal helps you run Python alongside dbt, so you can send Slack alerts, detect anomalies and build machine learning models.
https://fal.ai/dbt-fal
Apache License 2.0
851 stars 71 forks source link

Support for incremental models #836

Open gerardpc opened 1 year ago

gerardpc commented 1 year ago

Context We are using fal with a python model in SQL Server, and are trying to update the model to an incremental one:

def model(dbt, session):
    """."""
    dbt.config(materialized="incremental")
    ...

However, I'm getting this error when doing dbt run:

09:18:35  Running with dbt=1.3.3
09:18:35  Found 16 models, 7 tests, 0 snapshots, 0 analyses, 438 macros, 0 operations, 0 seed files, 19 sources, 0 exposures, 0 metrics
09:18:35  
09:18:39  Concurrency: 1 threads (target='dev')
09:18:39  
09:18:39  1 of 1 START python incremental model dbo.eq_con_width_ratio ................... [RUN]
09:18:39  1 of 1 ERROR creating python incremental model dbo.eq_con_width_ratio .......... [ERROR in 0.05s]
09:18:39  
09:18:39  Finished running 1 incremental model in 0 hours 0 minutes and 3.84 seconds (3.84s).
09:18:39  
09:18:39  Completed with 1 error and 0 warnings:
09:18:39  
09:18:39  Runtime Error in model eq_con_width_ratio (models/eq_con_width_ratio.py)
09:18:39    Materialization "materialization_incremental_default" only supports languages ['sql']; got "python"
09:18:39  
09:18:39  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

After some research it appears that incremental models are not yet supported by fal.

Describe the solution you'd like The possibility to use incremental python models instead of just full table recreations.

Is there an existing feature request for this? Apparently not, I've been suggested to open one from the slack channel, https://app.slack.com/client/T0VLPD22H/C02V8QW3Q4Q/thread/C02V8QW3Q4Q-1682416894.173549