Tyler-Rendina
commented
9 months ago @maziyarpanahi to reproduce:
Dockerfile for ECR Image referenced by EMR Serverless Application
FROM public.ecr.aws/emr-serverless/spark/emr-6.14.0:20230928-x86_64
USER root
RUN pip3 install spark-nlp==5.1.4
USER hadoop:hadoopSpark Submit via Console
{
"applicationConfiguration": [
{
"classification": "spark-defaults",
"configurations": [],
"properties": {
"spark.serializer": "org.apache.spark.serializer.KryoSerializer",
"spark.sql.hive.convertMetastoreParquet": "false",
"spark.kryoserializer.buffer.max": "2000M",
"spark.driver.maxResultSize": "0",
"spark.jars.packages": "com.johnsnowlabs.nlp:spark-nlp_2.12:5.1.4",
"spark.jars": "/usr/lib/hudi/hudi-spark-bundle.jar, s3://BUCKET/jars/spark-nlp_2.12-5.1.4.jar",
"spark.jsl.settings.pretrained.cache_folder": "s3://BUCKET/sparknlp/cache",
"spark.jsl.settings.aws.region": "REGION"
}
}
]
}Sentence Detector DL Model is the point of contention here.
class forEachBatchProcessor:
def __init__(self):
document_assembler = (
DocumentAssembler()
.setInputCol("content")
.setOutputCol("document")
.setCleanupMode("shrink_full")
)
cleanUpPatterns = [
"(https?:\/\/)?([\da-z\.-]+)\.([a-z\.]{2,6})([\/\w\.-]*)|^\.\.\."
]
document_normalizer = (
DocumentNormalizer()
.setInputCols("document")
.setOutputCol("normalizedDocument")
.setLowercase(False)
.setAction("clean")
.setPatterns(cleanUpPatterns)
.setReplacement(" ")
.setPolicy("pretty_all")
)
sentence_detector_dl = (
SentenceDetectorDLModel()
.pretrained()
.setInputCols(["normalizedDocument"])
.setOutputCol("sentences")
.setMinLength(5)
.setExplodeSentences(False)
)
self.pipeline = Pipeline(
stages=[document_assembler, document_normalizer, sentence_detector_dl]
)Ultimately it's a file not found error in /home/hadoop/cache_pretrained which has inspired this feature request.
github-actions[bot]
Description
Problem
Is it feasible to set
spark.jsl.settings.pretrained.cache_folderto an s3 bucket and authenticate using the IAM role specified with spark submit?Preferred Solution
Instead of using
spark.jsl.settings.aws.credentials.access_key_idandspark.jsl.settings.aws.credentials.secret_access_keywhen settingspark.jsl.settings.pretrained.cache_folderto an s3 location, can a user instead pass an IAM role?Additional Context
A workaround was attempted, failed, and documented below:
boto3examples/python/quick_start_offline.ipynb