search

scylladb / scylla-migrator

Migrate data extract using Spark to Scylla, normally from Cassandra/parquet files. Alt. from DynamoDB to Scylla Alternator.
https://migrator.docs.scylladb.com/stable/
Apache License 2.0
55 stars 34 forks source link

Validator uses obsolete param #27

Closed tarzanek closed 3 years ago

tarzanek commented 3 years ago 
Exception in thread "main" java.lang.IllegalArgumentException: spark.cassandra.dev.customFromDriver is deprecated (Analytics Connector 1.0) and is no longer in use. The ability to load new driver type converters at runtime has been removed

line 125 should be removed in https://github.com/scylladb/scylla-migrator/blob/master/src/main/scala/com/scylladb/migrator/Validator.scala#L125

tarzanek commented 3 years ago

after removing above we still get:

Exception in thread "main" java.lang.IllegalArgumentException: spark.cassandra.dev.customFromDriver is deprecated (Analytics Connector 1.0) and is no longer in use. The ability to load new driver type converters at runtime has been removed
        at com.datastax.spark.connector.util.DeprecatedConfigParameter.maybeReplace(ConfigParameter.scala:48)
        at com.datastax.spark.connector.util.ConfigCheck$$anonfun$checkConfig$1.apply(ConfigCheck.scala:48)
        at com.datastax.spark.connector.util.ConfigCheck$$anonfun$checkConfig$1.apply(ConfigCheck.scala:47)
        at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
        at com.datastax.spark.connector.util.ConfigCheck$.checkConfig(ConfigCheck.scala:47)
        at com.datastax.spark.connector.cql.CassandraConnectorConf$.apply(CassandraConnectorConf.scala:347)
        at com.scylladb.migrator.Connectors$.sourceConnector(Connectors.scala:18)
        at com.scylladb.migrator.Validator$.runValidation(Validator.scala:34)
        at com.scylladb.migrator.Validator$.main(Validator.scala:140)
        at com.scylladb.migrator.Validator.main(Validator.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
[root@centos7 scylla-migrator]# vi spark-cassandra-connector/connector/src/main/scala/com/datastax/spark/connector/types/ColumnTypeConf.scala
[root@centos7 scylla-migrator]# ls -lart^C
[root@centos7 scylla-migrator]# grep -Ri customFromDriver *
spark-cassandra-connector/connector/src/main/scala/com/datastax/spark/connector/types/ColumnTypeConf.scala:case class ColumnTypeConf(customFromDriver: Option[String])
spark-cassandra-connector/connector/src/main/scala/com/datastax/spark/connector/types/ColumnTypeConf.scala:    "spark.cassandra.dev.customFromDriver",
phenriqueabr commented 3 years ago

[root@centos7 bin]# ./spark-submit --version Welcome to

 / __/__  ___ _____/ /__
_\ \/ _ \/ _ `/ __/  '_/

// ./_,// //_\ version 2.4.4 /_/

Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_262 Branch Compiled by user on 2019-08-27T21:21:38Z Revision Url

tarzanek commented 3 years ago

rebuilt the validator without that line and it works, so ignore my previous paste, it was an unclean build will open a PR