search

suaaa7 / spark-ml-lr

Logistic regression with Spark ML
Apache License 2.0
0 stars 0 forks source link

Add test for transformer #3

Open suaaa7 opened 4 years ago

suaaa7 commented 4 years ago

Referenced repository

https://github.com/apache/spark/blob/master/mllib/src/test/scala/org/apache/spark/ml/feature/OneHotEncoderSuite.scala

suaaa7 commented 4 years ago

suffixの「Suite」とは？

テストケースの集合のこと

suaaa7 commented 4 years ago

Unit Testing spark dataframes transformation chaining https://stackoverflow.com/questions/54403226/unit-testing-spark-dataframes-transformation-chaining

https://github.com/holdenk/spark-testing-base

suaaa7 commented 4 years ago 
20/01/29 12:22:22 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect.
[info] ColRenameTransformerSuite:
[info] spark.ml.transformer.ColRenameTransformerSuite *** ABORTED ***
[info]   java.lang.ExceptionInInitializerError:
[info]   at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
[info]   at org.apache.spark.SparkContext.parallelize(SparkContext.scala:716)
[info]   at spark.ml.transformer.ColRenameTransformerSuite$$anonfun$1.apply$mcV$sp(ColRenameTransformerSuite.scala:43)
[info]   at spark.ml.transformer.ColRenameTransformerSuite$$anonfun$1.apply(ColRenameTransformerSuite.scala:22)
[info]   at spark.ml.transformer.ColRenameTransformerSuite$$anonfun$1.apply(ColRenameTransformerSuite.scala:22)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.9
suaaa7 commented 4 years ago

https://engineering.mobalab.net/2018/07/23/testing-spark-dataframe/

suaaa7 commented 4 years ago

https://stackoverflow.com/questions/45869367/spark-unit-test-fails-with-exceptionininitializererror-with-latest-version-of

suaaa7 commented 4 years ago

https://stackoverflow.com/questions/51515010/incompatible-jackson-version-spark-structured-streaming

suaaa7 commented 4 years ago

https://stackoverflow.com/questions/47951910/spark2-2-1-incompatible-jackson-version-2-8-8

suaaa7 commented 4 years ago 
Exception in thread "delete Spark temp dirs" java.lang.ExceptionInInitializerError
        at com.holdenkarau.spark.testing.Utils$$anonfun$shutDownCleanUp$1.apply(Utils.scala:44)
        at com.holdenkarau.spark.testing.Utils$$anonfun$shutDownCleanUp$1.apply(Utils.scala:44)
        at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
        at com.holdenkarau.spark.testing.Utils$.shutDownCleanUp(Utils.scala:44)
        at com.holdenkarau.spark.testing.Utils$$anon$1.run(Utils.scala:39)
Caused by: java.lang.IllegalStateException: Shutdown in progress
        at java.base/java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66)
        at java.base/java.lang.Runtime.addShutdownHook(Runtime.java:215)
        at com.holdenkarau.spark.testing.Utils$.<init>(Utils.scala:37)
        at com.holdenkarau.spark.testing.Utils$.<clinit>(Utils.scala)
        ... 5 more
Exception in thread "Thread-4" java.lang.IllegalAccessError: failed to access class org.apache.hadoop.util.ShutdownHookManager$2 from class org.apache.hadoop.util.ShutdownHookManager (org.apache.hadoop.util.ShutdownHookManager$2 is in unnamed module of loader sbt.internal.ManagedClassLoader$ZombieClassLoader @64f35deb; org.apache.hadoop.util.ShutdownHookManager is in unnamed module of loader sbt.internal.LayeredClassLoader @3f9ab0b6)
        at org.apache.hadoop.util.ShutdownHookManager.getShutdownHooksInOrder(ShutdownHookManager.java:183)
        at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:64)