[BUG] Databricks integration is not working as documented

[richiesgr]

Mage version

0.9.72

Describe the bug

After reading the documentation the pipeline type in file metadata.yaml cannot be set to databricks it's always return back to python.

In result there is no support to convert spark dataframe to pandas dataframe:

df_spark = spark.sql("select * table") return a spark dataframe generate a pickle errror

to make it work you need to explicitly convert to pandas df_spark = spark.sql("select * table") .toPandas() works

To reproduce

• Create a pipeline
• Edit in metadata.yaml type from python to databricks
• Save
• Run data loader (generic)

  df_spark = spark.sql("select * from dev.test.student")
  return df_spark

• Get Error Can't pickle <function SparkSession.reduce ....

Expected behavior

Support databricks pipeline as documented Handle spark dataframe as expected return

Screenshots

Operating system

Docker on macos

Additional context

No response

[wangxiaoyou1993]

Databricks has updates on their library. So the guide is outdated. We'll update it when we get time.