search

JohnSnowLabs / nlu

1 line for thousands of State of The Art NLP models in hundreds of languages The fastest and most accurate way to solve text problems.
Apache License 2.0
854 stars 130 forks source link

spark nlu load error #82

Open itsvignesh opened 2 years ago

itsvignesh commented 2 years ago

Am trying to explore NLU models first and then the NLU Healthcare models. nlu.load('emotion') step is failing. Attached the logs.

OS – Linux RHEL Pyspark – version 3.0.1 Command used for install - python3 -m pip install nlu pyspark==3.0.1 --trusted-host pypi.org --trusted-host files.pythonhosted.org I have created a python venv and install the NLU per above command.

I also tried reinstalling with below command: python3 -m pip install --upgrade nlu streamlit pyspark==3.0.2

Code below: import nlu pp=nlu.load('emotion') classifierdl_use_emotion download started this may take some time. Approximate size to download 21.3 MB [ / ] An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.downloadModel. : java.lang.NoClassDefFoundError: org/tensorflow/Tensor at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.read(TensorflowWrapper.scala:397) at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel.readTensorflowModel(TensorflowSerializeModel.scala:145) at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel.readTensorflowModel$(TensorflowSerializeModel.scala:120) at com.johnsnowlabs.nlp.annotators.classifier.dl.ClassifierDLModel$.readTensorflowModel(ClassifierDLModel.scala:291) at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.readTensorflow(ClassifierDLModel.scala:278) at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.readTensorflow$(ClassifierDLModel.scala:276) at com.johnsnowlabs.nlp.annotators.classifier.dl.ClassifierDLModel$.readTensorflow(ClassifierDLModel.scala:291) at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.$anonfun$$init$$1(ClassifierDLModel.scala:285) at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.$anonfun$$init$$1$adapted(ClassifierDLModel.scala:285) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$onRead$1(ParamsAndFeaturesReadable.scala:47) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$onRead$1$adapted(ParamsAndFeaturesReadable.scala:46) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.onRead(ParamsAndFeaturesReadable.scala:46) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$read$1(ParamsAndFeaturesReadable.scala:57) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$read$1$adapted(ParamsAndFeaturesReadable.scala:57) at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:35) at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:24) at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadModel(ResourceDownloader.scala:333) at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadModel(ResourceDownloader.scala:327) at com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader$.downloadModel(ResourceDownloader.scala:456) at com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.downloadModel(ResourceDownloader.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.tensorflow.Tensor at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 34 more [OK!] EXCEPTION: Could not resolve singular Component for type=emotion and nlp_ref=classifierdl_use_emotion and nlu_ref=emotion and lang =en Traceback (most recent call last): File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/pipe/component_resolution.py", line 852, in construct_component_from_identifier is_licensed=is_licensed) File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/components/classifier.py", line 69, in init else : self.model = ClassifierDl.get_pretrained_model(nlp_ref, language) File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/components/classifiers/classifier_dl/classifier_dl.py", line 11, in get_pretrained_model return ClassifierDLModel.pretrained(name,language,bucket) \ File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/sparknlp/annotator.py", line 8063, in pretrained return ResourceDownloader.downloadModel(ClassifierDLModel, name, lang, remote_loc) File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/sparknlp/pretrained.py", line 62, in downloadModel raise e File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/sparknlp/pretrained.py", line 59, in downloadModel j_obj = _internal._DownloadModel(reader.name, name, language, remote_loc, j_dwn).apply() File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/sparknlp/internal.py", line 214, in init name, language, remote_loc) File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/sparknlp/internal.py", line 165, in init self._java_obj = self.new_java_obj(java_obj, args) File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/sparknlp/internal.py", line 175, in new_java_obj return self._new_java_obj(java_class, args) File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/pyspark/ml/wrapper.py", line 69, in _new_java_obj return java_obj(java_args) File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/py4j/java_gateway.py", line 1305, in call answer, self.gateway_client, self.target_id, self.name) File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/pyspark/sql/utils.py", line 128, in deco return f(a, **kw) File "/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/py4j/protocol.py", line 328, in get_return_value format(target_id, ".", name), value) py4j.protocol.Py4JJavaError: An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.downloadModel. : java.lang.NoClassDefFoundError: org/tensorflow/Tensor at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.read(TensorflowWrapper.scala:397) at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel.readTensorflowModel(TensorflowSerializeModel.scala:145) at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel.readTensorflowModel$(TensorflowSerializeModel.scala:120) at com.johnsnowlabs.nlp.annotators.classifier.dl.ClassifierDLModel$.readTensorflowModel(ClassifierDLModel.scala:291) at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.readTensorflow(ClassifierDLModel.scala:278) at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.readTensorflow$(ClassifierDLModel.scala:276) at com.johnsnowlabs.nlp.annotators.classifier.dl.ClassifierDLModel$.readTensorflow(ClassifierDLModel.scala:291) at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.$anonfun$$init$$1(ClassifierDLModel.scala:285) at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadClassifierDLTensorflowModel.$anonfun$$init$$1$adapted(ClassifierDLModel.scala:285) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$onRead$1(ParamsAndFeaturesReadable.scala:47) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$onRead$1$adapted(ParamsAndFeaturesReadable.scala:46) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.onRead(ParamsAndFeaturesReadable.scala:46) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$read$1(ParamsAndFeaturesReadable.scala:57) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable.$anonfun$read$1$adapted(ParamsAndFeaturesReadable.scala:57) at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:35) at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:24) at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadModel(ResourceDownloader.scala:333) at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadModel(ResourceDownloader.scala:327) at com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader$.downloadModel(ResourceDownloader.scala:456) at com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.downloadModel(ResourceDownloader.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.tensorflow.Tensor at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 34 more

ValueError Traceback (most recent call last) /apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/init.py in load(request, path, verbose, gpu, streamlit_caching) 341 if nlu_ref == '': continue --> 342 nlu_component = nlu_ref_to_component(nlu_ref, authenticated=is_authenticated) 343 # if we get a list of components, then the NLU reference is a pipeline, we do not need to check order

/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/pipe/component_resolution.py in nlu_ref_to_component(nlu_reference, detect_lang, authenticated, is_recursive_call) 322 authenticated=authenticated, --> 323 is_recursive_call=is_recursive_call) 324 if resolved_component is None:

/apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/pipe/component_resolution.py in resolve_component_from_parsed_query_data(lang, component_type, dataset, component_embeddings, nlu_ref, trainable, path, authenticated, is_recursive_call) 467 if constructed_component is None: --> 468 raise ValueError(f'EXCEPTION : Could not create NLU component for nlp_ref={nlp_ref} and nlu_ref={nlu_ref}') 469 else:

ValueError: EXCEPTION : Could not create NLU component for nlp_ref=classifierdl_use_emotion and nlu_ref=emotion

During handling of the above exception, another exception occurred:

Exception Traceback (most recent call last)

in ----> 1 pp=nlu.load('emotion') /apps/sparknlp/spark-nlu/lib64/python3.6/site-packages/nlu/__init__.py in load(request, path, verbose, gpu, streamlit_caching) 360 print(e[1]) 361 raise Exception( --> 362 "Something went wrong during loading and fitting the pipe. Check the other prints for more information and also verbose mode. Did you use a correct model reference?") 363 364 Exception: Something went wrong during loading and fitting the pipe. Check the other prints for more information and also verbose mode. Did you use a correct model reference?
filemon11 commented 2 years ago

I get the same error on Windows 10 despite following instructions step by step.