search

intel-analytics / analytics-zoo

Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray
https://analytics-zoo.readthedocs.io/
Apache License 2.0
18 stars 4 forks source link

BigDL2.0 "JavaPackage is not callable" on init_spark_on_local if import bigdl.orca #102

Closed shanyu-sys closed 3 years ago

shanyu-sys commented 3 years ago

init_spark_on_local doesn't work with import bigdl.orca. It works if I import init_spark_on_local only.

>>> from bigdl.dllib.utils.nncontext import init_spark_on_local
>>> import bigdl.orca
>>> sc = init_spark_on_local()
Current pyspark location is : /opt/spark-2.4.3-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/__init__.py
Start to getOrCreate SparkContext
pyspark_submit_args is:  --driver-class-path /home/shan/sources/analytics-zoo/dist/lib/bigdl-dllib-0.14.0-SNAPSHOT-jar-with-dependencies.jar pyspark-shell
2021-09-14 08:27:00 WARN  Utils:66 - Your hostname, shan-dev resolves to a loopback address: 127.0.1.1; using 10.239.158.164 instead (on interface enp0s31f6)
2021-09-14 08:27:00 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2021-09-14 08:27:00 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/shan/sources/analytics-zoo/dist/lib/bigdl-orca-0.14.0-SNAPSHOT-python-api.zip/bigdl/dllib/utils/nncontext.py", line 53, in init_spark_on_local
  File "/home/shan/sources/analytics-zoo/dist/lib/bigdl-orca-0.14.0-SNAPSHOT-python-api.zip/bigdl/dllib/utils/spark.py", line 57, in init_spark_on_local
  File "/home/shan/sources/analytics-zoo/dist/lib/bigdl-orca-0.14.0-SNAPSHOT-python-api.zip/bigdl/dllib/utils/nncontext.py", line 404, in init_nncontext
  File "/home/shan/sources/analytics-zoo/dist/lib/bigdl-orca-0.14.0-SNAPSHOT-python-api.zip/bigdl/dllib/utils/common.py", line 454, in redire_spark_logs
  File "/home/shan/sources/analytics-zoo/dist/lib/bigdl-orca-0.14.0-SNAPSHOT-python-api.zip/bigdl/dllib/utils/common.py", line 597, in callBigDlFunc
  File "/home/shan/sources/analytics-zoo/dist/lib/bigdl-orca-0.14.0-SNAPSHOT-python-api.zip/bigdl/dllib/utils/common.py", line 56, in instance
  File "/home/shan/sources/analytics-zoo/dist/lib/bigdl-orca-0.14.0-SNAPSHOT-python-api.zip/bigdl/dllib/utils/common.py", line 104, in __init__
TypeError: 'JavaPackage' object is not callable

That might be caused by orca __init__. It works before this update.

shanyu-sys commented 3 years ago

@dding3 Ding, could you have a look at this issue?