search

titicaca / spark-iforest

Isolation Forest on Spark
Apache License 2.0
227 stars 89 forks source link

pyspark-iforest不支持spark2.2.0版本吗 #12

Closed wwb306 closed 5 years ago

wwb306 commented 5 years ago

我的spark版本是：2.2.1，使用了v2.2.0的releases编译了jar包后，安装了Python目录下的pyspark-iforest，但在使用过程中报错：

Traceback (most recent call last):
File "/appcom/apps/hduser0011/wangwenbin520/test_iforest.py", line 19, in <module>
 model = iforest.fit(df) File "/appcom/spark-2.2.1/python/lib/pyspark.zip/pyspark/ml/base.py", line 64, in fit 
File "/appcom/spark-2.2.1/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 265, in _fit 
File "/appcom/spark-2.2.1/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 261, in _fit_java 
File "/appcom/spark-2.2.1/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 124, in _transfer_params_to_java 
File "/appcom/spark-2.2.1/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 113, in _make_java_param_pair 
File "/appcom/spark-2.2.1/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
File "/appcom/spark-2.2.1/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco 
File "/appcom/spark-2.2.1/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319,
 in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o80.getParam. : java.util.NoSuchElementException: Param approxQuantileRelativeError does not exist. 
at org.apache.spark.ml.param.Params$$anonfun$getParam$2.apply(params.scala:601) at
 org.apache.spark.ml.param.Params$$anonfun$getParam$2.apply(params.scala:601) at
 scala.Option.getOrElse(Option.scala:121) at 
org.apache.spark.ml.param.Params$class.getParam(params.scala:600) at 
org.apache.spark.ml.PipelineStage.getParam(Pipeline.scala:42) at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at
 java.lang.reflect.Method.invoke(Method.java:498) at 
py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at 
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at 
py4j.Gateway.invoke(Gateway.java:280) at 
py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at 
py4j.commands.CallCommand.execute(CallCommand.java:79) at 
py4j.GatewayConnection.run(GatewayConnection.java:214) at 
java.lang.Thread.run(Thread.java:745)

所以，当前的Python是不支持spark2.2.1的版本对吗？

titicaca commented 5 years ago

是的 当前pyspark-iforest 仅支持master 分支中的 spark-v2.4.0的版本