[BUG] TypeError 'javapackage' object is not callable

live2awesome commented 2 years ago

SynapseML version

0.10.1

System information

Language version (e.g. python 3.8, scala 2.12):
Spark Version (e.g. 3.2.2):
Spark Platform (e.g. Synapse, Databricks): python : 3.7 Pyspark version : 3.1.2

Describe the problem

TypeError Traceback (most recent call last) ~\AppData\Local\Temp\ipykernel_33260\4209124749.py in ----> 1 lime_results=lime_explain(rf_model,validation_df,string_columns, numerical_columns, double_columns, boolean_columns, mapping).transform(0) 2 # lime_results=lime_explain(rf_model,validation_df,string_columns, numerical_columns, double_columns, boolean_columns, mapping)~\AppData\Local\Temp\ipykernel_33260\4049160313.py in transform(self, idx) 47 # print(type(lime_results)) 48---> 49 lime_results =self.lime.transform(explain_instances)\ 50 .select(["weights", "probability", "label", "predictedLabel"] + self.categorical_features + self.numeric_features + self.boolean_features + self.double_features)\ 51 .withColumn("weights", vec2array(col("weights").getItem(1)))\~\AppData\Local\conda\conda\envs\xai_env\lib\site-packages\pyspark\ml\base.py in transform(self, dataset, params) 215 return self.copy(params)._transform(dataset) 216 else:--> 217 return self._transform(dataset) 218 else: 219 raise ValueError("Params must be a param map but got %s." % type(params))~\AppData\Local\conda\conda\envs\xai_env\lib\site-packages\pyspark\ml\wrapper.py in _transform(self, dataset) 347 348 def _transform(self, dataset):--> 349 self._transfer_params_to_java() 350 return DataFrame(self._java_obj.transform(dataset._jdf), dataset.sql_ctx) 351~\AppData\Local\conda\conda\envs\xai_env\lib\site-packages\synapse\ml\core\schema\Utils.py in _transfer_params_to_java(self)TypeError: 'JavaPackage' object is not callable

Code to reproduce issue

https://www.kaggle.com/code/reyasadhu/lime-pyspark/notebook

Other info / logs

TypeError Traceback (most recent call last) ~\AppData\Local\Temp\ipykernel_33260\4209124749.py in ----> 1 lime_results=lime_explain(rf_model,validation_df,string_columns, numerical_columns, double_columns, boolean_columns, mapping).transform(0) 2 # lime_results=lime_explain(rf_model,validation_df,string_columns, numerical_columns, double_columns, boolean_columns, mapping)~\AppData\Local\Temp\ipykernel_33260\4049160313.py in transform(self, idx) 47 # print(type(lime_results)) 48---> 49 lime_results =self.lime.transform(explain_instances)\ 50 .select(["weights", "probability", "label", "predictedLabel"] + self.categorical_features + self.numeric_features + self.boolean_features + self.double_features)\ 51 .withColumn("weights", vec2array(col("weights").getItem(1)))\~\AppData\Local\conda\conda\envs\xai_env\lib\site-packages\pyspark\ml\base.py in transform(self, dataset, params) 215 return self.copy(params)._transform(dataset) 216 else:--> 217 return self._transform(dataset) 218 else: 219 raise ValueError("Params must be a param map but got %s." % type(params))~\AppData\Local\conda\conda\envs\xai_env\lib\site-packages\pyspark\ml\wrapper.py in _transform(self, dataset) 347 348 def _transform(self, dataset):--> 349 self._transfer_params_to_java() 350 return DataFrame(self._java_obj.transform(dataset._jdf), dataset.sql_ctx) 351~\AppData\Local\conda\conda\envs\xai_env\lib\site-packages\synapse\ml\core\schema\Utils.py in _transfer_params_to_java(self)TypeError: 'JavaPackage' object is not callable

What component(s) does this bug affect?

[X] area/cognitive: Cognitive project
[X] area/core: Core project
[ ] area/deep-learning: DeepLearning project
[ ] area/lightgbm: Lightgbm project
[ ] area/opencv: Opencv project
[ ] area/vw: VW project
[ ] area/website: Website
[ ] area/build: Project build system
[X] area/notebooks: Samples under notebooks folder
[ ] area/docker: Docker usage
[ ] area/models: models related issue

What language(s) does this bug affect?

[ ] language/scala: Scala source code
[X] language/python: Pyspark APIs
[ ] language/r: R APIs
[ ] language/csharp: .NET APIs
[ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

[ ] integrations/synapse: Azure Synapse integrations
[ ] integrations/azureml: Azure ML integrations
[ ] integrations/databricks: Databricks integrations

github-actions[bot] commented 2 years ago

Hey @live2awesome :wave:! Thank you so much for reporting the issue/feature request :rotating_light:. Someone from SynapseML Team will be looking to triage this issue soon. We appreciate your patience.

serena-ruan commented 1 year ago

Hi @live2awesome sorry for late response. From your code snippet I see this error

It's AttributeError: 'NoneType' object has no attribute 'toArray' instead of your pasted error, could you elaborate more? It would be nice if you could have a mini repro so I could help take a look :)

microsoft / SynapseML