Open lishengzhe opened 4 years ago
spark-tensorflow-connector seems not working with Spark 2.0.X (The prerequisite in readme is Spark 2.0 or later)
spark-tensorflow-connector
I got the following error when compiling with Spark 2.0.2
$ mvn install [INFO] Scanning for projects... [INFO] [INFO] -----------< org.tensorflow:spark-tensorflow-connector_2.11 >----------- [INFO] Building spark-tensorflow-connector 1.10.0 [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- scala-maven-plugin:3.2.2:add-source (compile) @ spark-tensorflow-connector_2.11 --- [INFO] Add Source directory: /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala [INFO] Add Test Source directory: /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/test/scala [INFO] [INFO] --- scala-maven-plugin:3.2.2:add-source (test) @ spark-tensorflow-connector_2.11 --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-tensorflow-connector_2.11 --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] [INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ spark-tensorflow-connector_2.11 --- [INFO] Changes detected - recompiling the module! [INFO] [INFO] --- scala-maven-plugin:3.2.2:compile (compile) @ spark-tensorflow-connector_2.11 --- [INFO] Using incremental compilation [INFO] Compiling 10 Scala sources to /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/target/classes... [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:121: not found: value ArrayData [ERROR] Int64ListFeatureEncoder.encode(ArrayData.toArrayData(row.get(index)).toIntArray().map(_.toLong)) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:123: not found: value ArrayData [ERROR] Int64ListFeatureEncoder.encode(ArrayData.toArrayData(row.get(index)).toLongArray()) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:125: not found: value ArrayData [ERROR] FloatListFeatureEncoder.encode(ArrayData.toArrayData(row.get(index)).toFloatArray()) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:127: not found: value ArrayData [ERROR] FloatListFeatureEncoder.encode(ArrayData.toArrayData(row.get(index)).toDoubleArray().map(_.toFloat)) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:129: not found: value ArrayData [ERROR] val decimalArray = ArrayData.toArrayData(row.get(index)).toArray[Decimal](DataTypes.createDecimalType()) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:132: not found: value ArrayData [ERROR] BytesListFeatureEncoder.encode(ArrayData.toArrayData(row.get(index)) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:133: object ObjectType in package types cannot be accessed in package org.apache.spark.sql.types [ERROR] .toArray[String](ObjectType(classOf[String])).map(_.getBytes)) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:135: not found: value ArrayData [ERROR] BytesListFeatureEncoder.encode(ArrayData.toArrayData(row.get(index)).toArray[Array[Byte]](BinaryType)) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:153: not found: value ArrayData [ERROR] val longArrays = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:154: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toIntArray().map(_.toLong).toSeq [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:159: not found: value ArrayData [ERROR] val longArrays = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:160: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toLongArray().toSeq [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:165: not found: value ArrayData [ERROR] val floatArrays = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:166: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toFloatArray().toSeq [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:171: not found: value ArrayData [ERROR] val floatArrays = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:172: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toDoubleArray().map(_.toFloat).toSeq [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:177: not found: value ArrayData [ERROR] val floatArrays = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:178: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toArray[Decimal](DataTypes.createDecimalType()).map(_.toFloat).toSeq [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:183: not found: value ArrayData [ERROR] val arrayData = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:184: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toArray[String](ObjectType(classOf[String])).toSeq.map(_.getBytes) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:184: not found: value ObjectType [ERROR] ArrayData.toArrayData(arr).toArray[String](ObjectType(classOf[String])).toSeq.map(_.getBytes) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:189: not found: value ArrayData [ERROR] val arrayData = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:190: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toArray[Array[Byte]](BinaryType).toSeq [ERROR] ^ [ERROR] 23 errors found [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 10.272 s [INFO] Finished at: 2020-04-28T17:10:02+09:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (compile) on project spark-tensorflow-connector_2.11: Execution compile of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.: CompileFailed -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
The problem solves when I change Spark version to 2.1.X (or above)
spark-tensorflow-connectorseems not working with Spark 2.0.X (The prerequisite in readme is Spark 2.0 or later)I got the following error when compiling with Spark 2.0.2
The problem solves when I change Spark version to 2.1.X (or above)