clemEssien / spark-nlp-phi-annotator

Apache License 2.0
1 stars 0 forks source link

Problems initializing spark session #7

Open clemEssien opened 3 years ago

clemEssien commented 3 years ago
Ivy Default Cache set to: /var/www/.ivy2/cache
phi-annotator    | The jars for the packages stored in: /var/www/.ivy2/jars
phi-annotator    | com.johnsnowlabs.nlp#spark-nlp_2.12 added as a dependency
phi-annotator    | :: resolving dependencies :: org.apache.spark#spark-submit-parent-9c8beda1-f199-4552-b85d-0f095156e99d;1.0
phi-annotator    |      confs: [default]
phi-annotator    | Exception in thread "main" java.io.FileNotFoundException: /var/www/.ivy2/cache/resolved-org.apache.spark-spark-submit-parent-9c8beda1-f199-4552-b85d-0f095156e99d-1.0.xml (No such file or directory)
phi-annotator    |      at java.io.FileOutputStream.open0(Native Method)
phi-annotator    |      at java.io.FileOutputStream.open(FileOutputStream.java:270)
phi-annotator    |      at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
phi-annotator    |      at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
phi-annotator    |      at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:70)
phi-annotator    |      at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:62)
phi-annotator    |      at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.toIvyFile(DefaultModuleDescriptor.java:563)
phi-annotator    |      at org.apache.ivy.core.cache.DefaultResolutionCacheManager.saveResolvedModuleDescriptor(DefaultResolutionCacheManager.java:176)
phi-annotator    |      at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:245)
phi-annotator    |      at org.apache.ivy.Ivy.resolve(Ivy.java:523)
phi-annotator    |      at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1427)
phi-annotator    |      at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
phi-annotator    |      at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
phi-annotator    |      at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
phi-annotator    |      at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
phi-annotator    |      at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
phi-annotator    |      at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
phi-annotator    |      at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
phi-annotator    |      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
phi-annotator    |      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
phi-annotator    | Failed to add operation for POST /api/v1/textContactAnnotations
phi-annotator    | Traceback (most recent call last):
phi-annotator    |   File "openapi_server/__main__.py", line 10, in <module>
phi-annotator    |     app.add_api('openapi.yaml', pythonic_params=True)
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/apps/flask_app.py", line 57, in add_api
phi-annotator    |     api = super(FlaskApp, self).add_api(specification, **kwargs)
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/apps/abstract.py", line 144, in add_api
phi-annotator    |     api = self.api_cls(specification,
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/apis/abstract.py", line 111, in __init__
phi-annotator    |     self.add_paths()
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/apis/abstract.py", line 219, in add_paths
phi-annotator    |     self._handle_add_operation_error(path, method, sys.exc_info())
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/apis/abstract.py", line 231, in _handle_add_operation_error
phi-annotator    |     raise value.with_traceback(traceback)
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/apis/abstract.py", line 209, in add_paths
phi-annotator    |     self.add_operation(path, method)
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/apis/abstract.py", line 162, in add_operation
phi-annotator    |     operation = make_operation(
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/operations/__init__.py", line 8, in make_operation
phi-annotator    |     return spec.operation_cls.from_spec(spec, *args, **kwargs)
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/operations/openapi.py", line 128, in from_spec
phi-annotator    |     return cls(
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/operations/openapi.py", line 75, in __init__
phi-annotator    |     super(OpenAPIOperation, self).__init__(
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/operations/abstract.py", line 96, in __init__
phi-annotator    |     self._resolution = resolver.resolve(self)
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/resolver.py", line 40, in resolve
phi-annotator    |     return Resolution(self.resolve_function_from_operation_id(operation_id), operation_id)
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/resolver.py", line 61, in resolve_function_from_operation_id
phi-annotator    |     return self.function_resolver(operation_id)
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/connexion/utils.py", line 111, in get_function_from_name
phi-annotator    |     module = importlib.import_module(module_name)
phi-annotator    |   File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
phi-annotator    |     return _bootstrap._gcd_import(name[level:], package, level)
phi-annotator    |   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
phi-annotator    |   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
phi-annotator    |   File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
phi-annotator    |   File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
phi-annotator    |   File "<frozen importlib._bootstrap_external>", line 855, in exec_module
phi-annotator    |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
phi-annotator    |   File "/opt/app/./openapi_server/controllers/text_contact_annotation_controller.py", line 4, in <module>
phi-annotator    |     from openapi_server import nlp_config as cf
phi-annotator    |   File "/opt/app/./openapi_server/nlp_config.py", line 20, in <module>
phi-annotator    |     spark = Spark().spark
phi-annotator    |   File "/opt/app/./openapi_server/nlp_config.py", line 17, in __init__
phi-annotator    |     self.spark = sparknlp_jsl.start("3.1.0-1778570a3f59d3059ed1e58375192fd61b114fc9", params=params)
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/sparknlp_jsl/__init__.py", line 80, in start
phi-annotator    |     return builder.getOrCreate()
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/pyspark/sql/session.py", line 228, in getOrCreate
phi-annotator    |     sc = SparkContext.getOrCreate(sparkConf)
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/pyspark/context.py", line 384, in getOrCreate
phi-annotator    |     SparkContext(conf=conf or SparkConf())
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/pyspark/context.py", line 144, in __init__
phi-annotator    |     SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/pyspark/context.py", line 331, in _ensure_initialized
phi-annotator    |     SparkContext._gateway = gateway or launch_gateway(conf)
phi-annotator    |   File "/usr/local/lib/python3.9/site-packages/pyspark/java_gateway.py", line 108, in launch_gateway
phi-annotator    |     raise Exception("Java gateway process exited before sending its port number")
phi-annotator    | Exception: Java gateway process exited before sending its port number
phi-annotator    | unable to load app 0 (mountpoint='') (callable not found or import error)
phi-annotator    | *** no app loaded. going in full dynamic mode ***

Problem description

In an attempt to initialize the spark session, the program attempts to look for spark dependencies at ' /var/www/.ivy2/' which it doesn't have access to and the location does not exist

tschaffter commented 3 years ago

Code blocks are not made with triple simple quotes (''') but ```

Code block example:

counter = 10
clemEssien commented 3 years ago

I have made the correction