search

aws-samples / emr-serverless-samples

Example code for running Spark and Hive jobs on EMR Serverless.
https://aws.amazon.com/emr/serverless/
MIT No Attribution
155 stars 78 forks source link

how to exec hive sql file with parameters #46

Closed mazhi1024 closed 1 year ago

mazhi1024 commented 1 year ago

I want to exec hsql file with parameters. local shell script add "-hivevar date_param=string:$date_param" ,example like this： """

!/bin/bash

date_param=date "+%Y%m%d"

aws emr-serverless start-job-run \ --application-id XXXXX \ --execution-role-arn arn:aws:iam::xxxxx:role/xxxxxxxx\ --job-driver '{ "hive": { "initQueryFile": "s3://mm-emr/emr-serverless-hive/sql/createTable.sql", "query": "s3://mm-emr/emr-serverless-hive/sql/insert3.sql", "parameters": "--hiveconf hive.exec.scratchdir=s3://mm-emr/emr-serverless-hive/hive/scratch --hiveconf hive.metastore.warehouse.dir=s3://aiways-emr/emr-serverless-hive/hive/warehouse -hivevar date_param=string:$date_param" } }' """ and i load the sql files to s3 path and run the app. Get a Error :An error occurred (ValidationException) when calling the StartJobRun operation: Flag '-hivevar' is not supported

how should i do ?

dacort commented 1 year ago

Based on the docs, it looks like you need a double dash instead of a single dash before hivevar. For example, you have -hivevar but I think it needs to be --hivevar.