search

recommenders-team / recommenders

Best Practices on Recommendation Systems
https://recommenders-team.github.io/recommenders/intro.html
MIT License
18.71k stars 3.06k forks source link

[BUG] Azure Databricks movielens.load_spark_df errors #2034

Closed mlonazure closed 9 months ago

mlonazure commented 9 months ago

Description

df = movielens.load_spark_df(spark = spark, size="100k", schema = schema,dbutils=dbutils, local_cache_path = '/dbfs/tmp')

fails with error, com.databricks.sql.io.FileReadException: Error while reading file file:/dbfs/tmp/u.data.

In which platform does it happen?

How do we replicate the issue?

df = movielens.load_spark_df(spark = spark, size="100k", schema = schema,dbutils=dbutils, local_cache_path = '/dbfs/tmp')

Expected behavior (i.e. solution)

Load the data into the dataframe.

Other Comments

I've configured the azure databricks library like the setup file.

mlonazure commented 9 months ago

I finally resolved this by switching to databricks runtime 11.2 as opposed to 12.2. Please note that the Readme/setup says that this repo has been tested on both.

mlonazure commented 9 months ago

I finally resolved this by switching to databricks runtime 11.2 as opposed to 12.2. Please note that the Readme/setup says that this repo has been tested on both.