Spark 3

[derrickoswald]

With the availability of Spark 3.0, there is a porting effort required to produce a version that runs with Spark 3, e.g. 2.12-3.0.0-5.0.0. Of note:

• it can use Hadoop version 3.2.1
• it uses Scala 2.12
• there are some changes to the handling of files in Hadoop due to the upgraded Hadoop version

  
  Spark context Web UI available at http://192.168.10.221:4041
  Spark context available as 'sc' (master = local[*], app id = local-1573335188812).
  Spark session available as 'spark'.
  Welcome to
    ____              __
   / __/__  ___ _____/ /__
  _\ \/ _ \/ _ `/ __/  '_/
  /___/ .__/\_,_/_/ /_/\_\   version 3.0.0-preview
    /_/

Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_222) Type in expressions to have them evaluated. Type :help for more information.

[derrickoswald]

Opened branch spark-3.0.0.

• CIMReader seems to function just fine
• CIMExport relies on spark-cassandra-connector which has no available version for Scala 2.12 or Spark 3.0.0

Pending inquiry over spark-cassandra-connector support going forward.

[derrickoswald]

The spark-cassandra-connector version 2.4.2 supports Spark 3.0.0. This seems to work just fine.

[derrickoswald]

With the release of 2.12-3.0.0-5.0.2 support for Spark 3.0.0 is complete.