predict() got an unexpected keyword argument 'meta'

[yantriks-edi-bice]

/kind bug

What steps did you take and what happened:

Deployed Kubeflow 1.0 on GCP and worked through xgboost-high-level-apis.ipynb. All went well until the endpoint.predict_nparray(test_X) step which generated following error

---

JSONDecodeError Traceback (most recent call last)

in 1 #(train_X, train_y), (test_X, test_y) = read_input("ames_dataset/train.csv") ----> 2 endpoint.predict_nparray(test_X) 3 test_X[1,] ~/.local/lib/python3.6/site-packages/kubeflow/fairing/ml_tasks/tasks.py in predict_nparray(self, data, feature_names) 124 serialized_data = json.dumps(pdata) 125 r = requests.post(self.url, data={'json': serialized_data}) --> 126 return json.loads(r.text) 127 128 def delete(self): /usr/lib/python3.6/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 352 parse_int is None and parse_float is None and 353 parse_constant is None and object_pairs_hook is None and not kw): --> 354 return _default_decoder.decode(s) 355 if cls is None: 356 cls = JSONDecoder /usr/lib/python3.6/json/decoder.py in decode(self, s, _w) 337 338 """ --> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 340 end = _w(s, end).end() 341 if end != len(s): /usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx) 355 obj, end = self.scan_once(s, idx) 356 except StopIteration as err: --> 357 raise JSONDecodeError("Expecting value", s, err.value) from None 358 return obj, end JSONDecodeError: Expecting value: line 1 column 1 (char 0) And on the GCP project error side another error which coincides in time but have not been able to { insertId: "u0az0ar5xu86c13t7" labels: { k8s-pod/fairing-deployer: "serving" k8s-pod/fairing-id: "83b1d074-5d82-11ea-a6ca-3a811b82dfb5" k8s-pod/pod-template-hash: "7946c4cd6c" } logName: "projects/saas-ml-dev/logs/stderr" receiveTimestamp: "2020-03-03T21:34:05.657934723Z" resource: { labels: { cluster_name: "kf-tst" container_name: "model" location: "us-central1-a" namespace_name: "kubeflow-edi-bice" pod_name: "fairing-deployer-5gzht-7946c4cd6c-vzvmj" project_id: "saas-ml-dev" } type: "k8s_container" } severity: "ERROR" textPayload: "Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/seldon_core/user_model.py", line 199, in client_predict return user_model.predict(features, feature_names, **kwargs) TypeError: predict() got an unexpected keyword argument 'meta' " timestamp: "2020-03-03T21:34:04.298606981Z" } **What did you expect to happen:** Prediction result printed to notebook output **Anything else you would like to add:** **Environment:** - Fairing version: (use `python -c "import kubeflow.fairing; print(kubeflow.fairing.__version__)"`): Version: 0.7.1 - Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): 1.0 - Minikube version: - Kubernetes version: (use `kubectl version`): 1.14.10-gke.22 - OS (e.g. from `/etc/os-release`): GCP Container OS **NOTE**: If you are using fair from master, please provide us the git commit hash.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/bug	0.99

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

[chantikummari]

facing same issue with azure also:

JSONDecodeError Traceback (most recent call last)

in 1 (train_X, train_y), (test_X, test_y) = read_input("ames_dataset/train.csv") ----> 2 endpoint.predict_nparray(test_X) 3 #test_X[1,] 4 #endpoint.predict(test_X) ~/.local/lib/python3.6/site-packages/kubeflow/fairing/ml_tasks/tasks.py in predict_nparray(self, data, feature_names) 124 serialized_data = json.dumps(pdata) 125 r = requests.post(self.url, data={'json': serialized_data}) --> 126 return json.loads(r.text) 127 128 def delete(self): /usr/lib/python3.6/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 352 parse_int is None and parse_float is None and 353 parse_constant is None and object_pairs_hook is None and not kw): --> 354 return _default_decoder.decode(s) 355 if cls is None: 356 cls = JSONDecoder /usr/lib/python3.6/json/decoder.py in decode(self, s, _w) 337 338 """ --> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 340 end = _w(s, end).end() 341 if end != len(s): /usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx) 355 obj, end = self.scan_once(s, idx) 356 except StopIteration as err: --> 357 raise JSONDecodeError("Expecting value", s, err.value) from None 358 return obj, end JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[chaitanyak52]

I see the same issue on azure

---

JSONDecodeError Traceback (most recent call last)

in 1 (train_X, train_y), (test_X, test_y) = read_input("ames_dataset/train.csv") ----> 2 endpoint.predict_nparray(test_X) ~/.local/lib/python3.6/site-packages/kubeflow/fairing/ml_tasks/tasks.py in predict_nparray(self, data, feature_names) 124 serialized_data = json.dumps(pdata) 125 r = requests.post(self.url, data={'json': serialized_data}) --> 126 return json.loads(r.text) 127 128 def delete(self): /usr/lib/python3.6/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 352 parse_int is None and parse_float is None and 353 parse_constant is None and object_pairs_hook is None and not kw): --> 354 return _default_decoder.decode(s) 355 if cls is None: 356 cls = JSONDecoder /usr/lib/python3.6/json/decoder.py in decode(self, s, _w) 337 338 """ --> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 340 end = _w(s, end).end() 341 if end != len(s): /usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx) 355 obj, end = self.scan_once(s, idx) 356 except StopIteration as err: --> 357 raise JSONDecodeError("Expecting value", s, err.value) from None 358 return obj, end JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[jinchihe]

Anyone interested in solving this problem? Thanks

[chantikummari]

Anyone interested in solving this problem? Thanks

Yes

[jinchihe]

@chantikummari Did you tried this with new kubeflow-fairing? the example is executed and passed in CI testing. Thanks.

[chantikummari]

@chantikummari Did you tried this with new kubeflow-fairing? the example is executed and passed in CI testing. Thanks.

I didn't try yet. We skipped that fairing feature from our project. I will definitely check it if any requirement comes related to it.

[jinchihe]

So I suggest use latest kubeflow-fairing, and use new the example, the example is updated, such as in the PR: https://github.com/kubeflow/fairing/pull/335

[achal-wwt]

@chantikummari, @jinchihe I was facing the same issue while trying Fairing on kubeflow cluster on AWS. I was trying to deploy my Fashion MNIST example using Fairing. My error got resolved when I put in feature_names=None in my predict function. My function call was - def predict(self, X, feature_names=None): # code to predict

[Juggernaut1997]

@achal-wwt Hey can you share how did you deploy it. Because I am facing the same issue and not getting how to resolve it

[achal-wwt]

@Juggernaut1997 I currently don't have the code available to share but if you are facing issue while using the trained model i.e. while getting prediction . Just make sure that you pass in feature_names as an argument in the predict function even though you are not using the argument.

[Juggernaut1997]

@achal-wwt Hi thanks for the reply, sorry i didn't respond early. But i am still facing the same issue.

[Anh-BK]

I realize that when I run seldon-core in the container, I occur the error like above. However, when I run with cpu, the error is gone.