kedro ipython immediately fails for starter=spaceflights-pyspark-viz

[gpierard]

Description

installed the requirements.txt, getting this immediately after kedro ipython kedro.io.core.DatasetError: Class 'spark.SparkDataset' not found, is this a typo?

is there any additional config needed for starter=spaceflights-pyspark-viz?

I tried this on Windows with 0.19.1 and 0.19.2 both yielding the same issue. kedro-datasets is 1.5.1

[datajoely]

Answered on stackoverflow as well, but would be easiest to track here

if you explicitly install pip install kedro-datasets[spark.SparkDataset] does it work? This should be part of the requirements.txt

[merelcht]

Hi @gpierard, thanks for flagging this issue. Could you provide some more info so it's easier for the team to find out what's going on?

• Which kedro version are you using?
• If you have kedro-datasets installed, which version are you using?
• Which python version are you using?
• Which OS (Unix/Windows)?

Thanks!

[astrojuanlu]

IIUC from https://github.com/kedro-org/kedro/issues/3545#issue-2095153118, Kedro 0.17.6 and Python 3.8.5, am I right @gpierard?

Any chance you can upgrade to a newer version? 0.17.6 is very old.

[gpierard]

thanks for your answers, sorry I should have mentioned that I tried this on Windows with 0.19.1 and 0.19.2 both yielding the same issue. kedro-datasets is 1.5.1 and kedro-datasets[spark.SparkDataset] is installed as well. This issue is fully separate from the others mentioned.

[merelcht]

@gpierard The datasets were only renamed in kedro-datasets 1.7.0 so before that it would be looking at spark.SparkDataSet. Can you try using the latest version 2.0.0?

[gpierard]

I confirm that kedro-datasets 2.0.0 solves the issue and that the spark session is correctly configured. thanks.