search

fal-ai / dbt-fal

do more with dbt. dbt-fal helps you run Python alongside dbt, so you can send Slack alerts, detect anomalies and build machine learning models.
https://fal.ai/dbt-fal
Apache License 2.0
851 stars 71 forks source link

Koldstart pins dill==0.3.5.11 which has compatibility issues with Python 3.11 #767

Open kinghuang opened 1 year ago

kinghuang commented 1 year ago

Describe the bug

I'm not sure where to report issues with koldstart.

I'm using dbt-fal==1.4.1 on Python 3.11. koldstart==0.6.19 brings in dill==0.3.5.11, which lacks a fix for uqfoundation/dill#514. This causes dill to fail when used on Python 3.11.

AttributeError: 'code' object has no attribute 'co_endlinetable'.

The error is encountered when using a Python model with pandarallel, which uses dill to serialize functions.

Your environment

Core:
  - installed: 1.4.1
  - latest:    1.4.1 - Up to date!

Plugins:
  - fal:              1.4.1 - Up to date!
  - redshift:         1.4.0 - Up to date!
  - postgres:         1.4.1 - Up to date!
  - fal_experimental: 1.4.1 - Could not determine latest version

How to reproduce

Use pandarallel to parallelize a dataframe operation in a Python model.

from pandarallel import pandarallel

def model(dbt, fal):
    pandarallel.initialize(verbose=2)
    input_df = dbt.ref("foundations_wells_names_raw_minimized")
    output_df = input_df.parallel_apply(lambda x: x)
    return output_df

Expected behavior

The dataframe is processed and returned as the model output.

Actual behavior

An AttributeError occurs when pandarallel attempts to serialize the function.

AttributeError: 'code' object has no attribute 'co_endlinetable'.

Screenshots

Additional context

Forcing dill==0.3.6 works.

chamini2 commented 1 year ago

Hey, @kinghuang it seems dill did not add official Python 3.11 support until after 0.3.6, so a version with Python 3.11 support has not been released.

That said, I will see if we can get koldstart to use 0.3.6 at least. Thanks for bringing this up!