search

zero-one-group / geni

A Clojure dataframe library that runs on Spark
Apache License 2.0
283 stars 28 forks source link

Can't run example #296

Closed andres-moreno closed 3 years ago

andres-moreno commented 3 years ago

Info

Info Value
Operating System Pop_Os 20.10
Geni Version 0.035
JDK OpenJDK Runtime Environment (build 11.0.9.1+1-Ubuntu-0ubuntu1.20.10)
Spark Version 3.0.1

Problem / Steps to reproduce

Incompatible library:

Scala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0

Steps:

lein new geni geni-cookbook cd geni-cookbook && lein run

The log file:

{:clojure.main/message "Syntax error (JsonMappingException) compiling at (/tmp/form-init3765768439809039245.clj:1:73).\nScala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0\n", :clojure.main/triage {:clojure.error/phase :compile-syntax-check, :clojure.error/line 1, :clojure.error/column 73, :clojure.error/source "form-init3765768439809039245.clj", :clojure.error/path "/tmp/form-init3765768439809039245.clj", :clojure.error/class com.fasterxml.jackson.databind.JsonMappingException, :clojure.error/cause "Scala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0"}, :clojure.main/trace {:via [{:type clojure.lang.Compiler$CompilerException, :message "Syntax error compiling at (/tmp/form-init3765768439809039245.clj:1:73).", :data {:clojure.error/phase :compile-syntax-check, :clojure.error/line 1, :clojure.error/column 73, :clojure.error/source "/tmp/form-init3765768439809039245.clj"}, :at [clojure.lang.Compiler load "Compiler.java" 7648]} {:type java.lang.ExceptionInInitializerError, :at [org.apache.spark.sql.execution.SparkPlan executeQuery "SparkPlan.scala" 210]} {:type com.fasterxml.jackson.databind.JsonMappingException, :message "Scala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0", :at [com.fasterxml.jackson.module.scala.JacksonModule setupModule "JacksonModule.scala" 61]}], :trace [[com.fasterxml.jackson.module.scala.JacksonModule setupModule "JacksonModule.scala" 61] [com.fasterxml.jackson.module.scala.JacksonModule setupModule$ "JacksonModule.scala" 46] [com.fasterxml.jackson.module.scala.DefaultScalaModule setupModule "DefaultScalaModule.scala" 17] [com.fasterxml.jackson.databind.ObjectMapper registerModule "ObjectMapper.java" 816] [org.apache.spark.rdd.RDDOperationScope$

"RDDOperationScope.scala" 82] [org.apache.spark.rdd.RDDOperationScope$ "RDDOperationScope.scala" -1] [org.apache.spark.sql.execution.SparkPlan executeQuery "SparkPlan.scala" 210] [org.apache.spark.sql.execution.SparkPlan execute "SparkPlan.scala" 171] [org.apache.spark.sql.execution.QueryExecution toRdd$lzycompute "QueryExecution.scala" 122] [org.apache.spark.sql.execution.QueryExecution toRdd "QueryExecution.scala" 121] [org.apache.spark.sql.Dataset rdd$lzycompute "Dataset.scala" 3200] [org.apache.spark.sql.Dataset rdd "Dataset.scala" 3198] [org.apache.spark.ml.PredictorParams extractInstances "Predictor.scala" 80] [org.apache.spark.ml.PredictorParams extractInstances$ "Predictor.scala" 70] [org.apache.spark.ml.Predictor extractInstances "Predictor.scala" 114] [org.apache.spark.ml.classification.LogisticRegression $anonfun$train$1 "LogisticRegression.scala" 488] [org.apache.spark.ml.util.Instrumentation$ $anonfun$instrumented$1 "Instrumentation.scala" 191] [scala.util.Try$ apply "Try.scala" 213] [org.apache.spark.ml.util.Instrumentation$ instrumented "Instrumentation.scala" 191] [org.apache.spark.ml.classification.LogisticRegression train "LogisticRegression.scala" 487] [org.apache.spark.ml.classification.LogisticRegression train "LogisticRegression.scala" 482] [org.apache.spark.ml.classification.LogisticRegression train "LogisticRegression.scala" 281] [org.apache.spark.ml.Predictor fit "Predictor.scala" 150] [org.apache.spark.ml.Predictor fit "Predictor.scala" 114] [org.apache.spark.ml.Pipeline $anonfun$fit$5 "Pipeline.scala" 151] [org.apache.spark.ml.MLEvents withFitEvent "events.scala" 132] [org.apache.spark.ml.MLEvents withFitEvent$ "events.scala" 125] [org.apache.spark.ml.util.Instrumentation withFitEvent "Instrumentation.scala" 42] [org.apache.spark.ml.Pipeline $anonfun$fit$4 "Pipeline.scala" 151] [scala.collection.Iterator foreach "Iterator.scala" 941] [scala.collection.Iterator foreach$ "Iterator.scala" 941] [scala.collection.AbstractIterator foreach "Iterator.scala" 1429] [scala.collection.IterableViewLike$Transformed foreach "IterableViewLike.scala" 47] [scala.collection.IterableViewLike$Transformed foreach$ "IterableViewLike.scala" 47] [scala.collection.SeqViewLike$AbstractTransformed foreach "SeqViewLike.scala" 40] [org.apache.spark.ml.Pipeline $anonfun$fit$2 "Pipeline.scala" 147] [org.apache.spark.ml.MLEvents withFitEvent "events.scala" 132] [org.apache.spark.ml.MLEvents withFitEvent$ "events.scala" 125] [org.apache.spark.ml.util.Instrumentation withFitEvent "Instrumentation.scala" 42] [org.apache.spark.ml.Pipeline $anonfun$fit$1 "Pipeline.scala" 133] [org.apache.spark.ml.util.Instrumentation$ $anonfun$instrumented$1 "Instrumentation.scala" 191] [scala.util.Try$ apply "Try.scala" 213] [org.apache.spark.ml.util.Instrumentation$ instrumented "Instrumentation.scala" 191] [org.apache.spark.ml.Pipeline fit "Pipeline.scala" 133] [jdk.internal.reflect.NativeMethodAccessorImpl invoke0 "NativeMethodAccessorImpl.java" -2] [jdk.internal.reflect.NativeMethodAccessorImpl invoke "NativeMethodAccessorImpl.java" 62] [jdk.internal.reflect.DelegatingMethodAccessorImpl invoke "DelegatingMethodAccessorImpl.java" 43] [java.lang.reflect.Method invoke "Method.java" 566] [clojure.lang.Reflector invokeMatchingMethod "Reflector.java" 167] [clojure.lang.Reflector invokeInstanceMethod "Reflector.java" 102] [zero_one.geni.ml$fit invokeStatic "ml.clj" 164] [zero_one.geni.ml$fit invoke "ml.clj" 163] [spark_app.core$_main invokeStatic "core.clj" 46] [spark_app.core$_main doInvoke "core.clj" 44] [clojure.lang.RestFn invoke "RestFn.java" 397] [clojure.lang.Var invoke "Var.java" 380] [user$eval140 invokeStatic "form-init3765768439809039245.clj" 1] [user$eval140 invoke "form-init3765768439809039245.clj" 1] [clojure.lang.Compiler eval "Compiler.java" 7177] [clojure.lang.Compiler eval "Compiler.java" 7167] [clojure.lang.Compiler load "Compiler.java" 7636] [clojure.lang.Compiler loadFile "Compiler.java" 7574] [clojure.main$load_script invokeStatic "main.clj" 475] [clojure.main$init_opt invokeStatic "main.clj" 477] [clojure.main$init_opt invoke "main.clj" 477] [clojure.main$initialize invokeStatic "main.clj" 508] [clojure.main$null_opt invokeStatic "main.clj" 542] [clojure.main$null_opt invoke "main.clj" 539] [clojure.main$main invokeStatic "main.clj" 664] [clojure.main$main doInvoke "main.clj" 616] [clojure.lang.RestFn applyTo "RestFn.java" 137] [clojure.lang.Var applyTo "Var.java" 705] [clojure.main main "main.java" 40]], :cause "Scala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0", :phase :compile-syntax-check}}
andres-moreno commented 3 years ago

Added an exclusion of jackson-databind tometosin/jsonista and got the example to work.

 :dependencies [[org.clojure/clojure "1.10.1"]
                 [zero.one/geni "0.0.35"]
                 [metosin/jsonista "0.2.7"
                  :exclusions [commons-codec com.fasterxml.jackson.core/jackson-databind]]
                 [expound "0.8.6"]
                 ;; Spark
                 [org.apache.spark/spark-core_2.12 "3.0.1"]
                 [org.apache.spark/spark-hive_2.12 "3.0.1"]
                 [org.apache.spark/spark-mllib_2.12 "3.0.1"]
                 [org.apache.spark/spark-sql_2.12 "3.0.1"]
                 [org.apache.spark/spark-streaming_2.12 "3.0.1"]
                 [org.apache.spark/spark-yarn_2.12 "3.0.1"]
                 [com.github.fommil.netlib/all "1.1.2" :extension "pom"]
                 ; Arrow
                 [org.apache.arrow/arrow-memory-netty "2.0.0"]
                 [org.apache.arrow/arrow-memory-core "2.0.0"]
                 [org.apache.arrow/arrow-vector "2.0.0"
                  :exclusions [commons-codec com.fasterxml.jackson.core/jackson-databind]]
                 ;; Databases
                 [mysql/mysql-connector-java "8.0.22"]
                 [org.postgresql/postgresql "42.2.18"]
                 [org.xerial/sqlite-jdbc "3.32.3.2"]]