Incremental processing within a Python model

[CraigWilson-ZOE]

Context We are trying to change a dbt Python model to be incremental. At the moment it runs in 30 minutes processing all data for all time. We need to move this to be incremental for processing to ensure that this model runs in a more reasonable time. We cannot do this currently due to the lack of support in FAL for incremental Python models.

Is your feature request related to a problem? Please describe. When we attempt to make a Python model incremental, we are presented with an error stating that we need to install PySpark and Java.

Describe the solution you'd like Be able to use the in-built logic to make a dbt model incremental (see here) and this just work correctly like a "normal" dbt python model.

Describe alternatives you've considered The only alternative we have here is to switch away from using FAL and to use the prescribed way from dbt to use DataProc clusters to do the Python processing.

Additional context No additional context

Is there an existing feature request for this?

• [x] I have searched the existing issues -> Similar issue here but I talked to Burkay and he asked for an issue to be raised.

[dnascimento]

We have the same challenge, our dbt process is OOM and fails. Could someone give some pointers on how to implement this?

[CraigWilson-ZOE]

Is there any update to this? It has been open for sometime and I have also discussed this with the team directly many times.