Open olivierr42 opened 2 weeks ago
Hi @olivierr42
The support for Apple Silicon is experimental at this point. This is true for all the DL based models/annotators. The Word2Vec is pure written by using machine learning algorithm so it works independent of the operating system.
It seems like the issue is with downloading the model. There seems to be a way to load the models from local storage, but I cannot seem to be able to make it work (it's trying to find a assets
subfolder within the model folder, which does not exist if I download from the provided url).
Do you have any tips to make it work locally?
What is the error when downloading models? You can always test it quickly in Google Colab to be sure whether it's the model or your environment.
Spark NLP works 100% offline, you can follow this instruction that shows how to download any model, extract it, and just use .load()
instead of .pretrained()
: https://sparknlp.org/docs/en/install#offline
PS: Your Spark application must have access to that local path
Is there an existing issue for this?
Who can help?
@maziyarpanahi I saw you answered to similar requests in the past. Thank you in advance.
What are you working on?
I am working with a in-house dataset. This is not an official exemple. I am trying to use this model specifically: https://sparknlp.org/api/python/reference/autosummary/sparknlp/annotator/embeddings/xlm_roberta_embeddings/index.html
I got the same issue when trying to load the SentenceDetectorDL model (mentioned on the Hub for this model)
Current Behavior
When I try to instantiate my pipeline:
I get the following error:
Expected Behavior
I know support for M1 is experimental, but I would expect it not to crash. Especially since I am able to run Word2Vec models without issue.
Steps To Reproduce
Spark NLP version and Apache Spark
sparknlp = '5.3.3' pyspark = '3.5.1'
Type of Spark Application
Python Application
Java Version
java version "1.8.0_411"
Java Home Directory
/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home
Setup and installation
poetry add sparknlp=5.3.3
Operating System and Version
Mac M1 Sonomo 14.5
Link to your project (if available)
No response
Additional Information
I do not have issue with Word2Vec models. I also tried with SParkNLP 5.4.1, to no avail.