awslabs / amazon-emr-cli

A command-line interface for packaging, deploying, and running your EMR Serverless Spark jobs
Apache License 2.0
35 stars 12 forks source link

Add --job-args and --spark-submit-opts for EMR on EC2 #30

Closed kirillklimenko closed 9 months ago

kirillklimenko commented 10 months ago

Add --job-args (variable never used) and --spark-submit-opts (variable not exists) for EMR on EC2. Currently --job-args as well as --spark-submit-opts works only for EMR Serverless:

# application_id indicates EMR Serverless job
    if application_id is not None:
        # We require entry-point and job-role
        if entry_point is None or job_role is None:
            raise click.BadArgumentUsage(
                "--entry-point and --job-role are required if --application-id is used."
            )

        if job_args:
            job_args = job_args.split(",")
        emrs = EMRServerless(application_id, job_role, p)
        emrs.run_job(
            job_name, job_args, spark_submit_opts, wait, show_stdout, s3_logs_uri
        )

    # cluster_id indicates EMR on EC2 job
    if cluster_id is not None:
        if job_args:
            job_args = job_args.split(",")
        emr = EMREC2(cluster_id, p, job_role)
        emr.run_job(job_name, job_args, wait, show_stdout)  # add spark_submit_opts
### Tasks
- [ ] Add `--job-args` to EMREC2.run_job
- [ ] Add `--spark-submit-opts` to EMREC2.run_job
dacort commented 9 months ago

Fixed in #35