search

Kotlin / kotlin-spark-api

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
Apache License 2.0
452 stars 34 forks source link

Cannot read file from Jupyter in Datalore #164

Closed Jolanrensen closed 2 years ago

Jolanrensen commented 2 years ago 
No FileSystem for scheme "file"
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "file"

As reported in Slack: https://kotlinlang.slack.com/archives/C4W52CFEZ/p1654119134612899

Related issue: https://stackoverflow.com/questions/17265002/hadoop-no-filesystem-for-scheme-file/27532248#27532248

Fix: Add

props = mapOf(
            "fs.hdfs.impl" to org.apache.hadoop.hdfs.DistributedFileSystem::class.java.name,
            "fs.file.impl" to org.apache.hadoop.fs.LocalFileSystem::class.java.name,
        )

to withSpark call

Jolanrensen commented 2 years ago

strangely it works fine in withSparkStreaming { withSpark(ssc) { ... } }