search

tensorflow / ecosystem

Integration of TensorFlow with other open-source frameworks
Apache License 2.0
1.37k stars 392 forks source link

'JavaPackage' object is not callable #173

Closed wenmin-wu closed 3 years ago

wenmin-wu commented 3 years ago

When I follow the document to test the following codes, I came across this Error: Codes:

sc = pyspark.SparkContext
sc._jvm.org.tensorflow.spark.datasources.tfrecords.udf.DataFrameTfrConverter.getRowToTFRecordExampleUdf()

Error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-d5aa4aefa440> in <module>
      1 sc = pyspark.SparkContext
----> 2 sc._jvm.org.tensorflow.spark.datasources.tfrecords.udf.DataFrameTfrConverter.getRowToTFRecordExampleUdf()

TypeError: 'JavaPackage' object is not callable

spark version: 2.4.3

Conf:

[INFO 2020-10-11 14:51:14.676 spark]: conf:
spark.app.name=pythia-mfp-infer
spark.kubernetes.namespace=econs-workspace
spark.driver.maxResultSize=6g
spark.driver.memory=8g
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2
spark.hadoop.mapreduce.output.fileoutputformat.compress=true
spark.hadoop.mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec
spark.hadoop.fs.s3.maxRetries=10
spark.sql.windowExec.buffer.spill.threshold=2097152
spark.sql.windowExec.buffer.in.memory.threshold=2097152
spark.jars.packages=org.apache.spark:spark-avro_2.11:2.4.4,org.tensorflow:spark-tensorflow-connector_2.11:1.14.0
spark.kubernetes.executor.annotation.iam.amazonaws.com/role=datalake-dev-stg
spark.executorEnv.AWS_METADATA_SERVICE_TIMEOUT=10
spark.executorEnv.AWS_METADATA_SERVICE_NUM_ATTEMPTS=10
spark.executorEnv.AWS_IAM_ROLE=datalake-dev-stg
spark.executor.instances=5
wenmin-wu commented 3 years ago

Fixed by upgrading the jar version to org.tensorflow:spark-tensorflow-connector_2.11:1.15.0.

If you follow the example, you may also encounter AttributeError, this is because the spark uses pickle which cannot pickle the EstimatorWrapper. To solve this problem, I put the EstimatorWrapper in a package and upload it to pypi. So we can install the package on both the driver and executors with pip install estimator-wrapper.