hadoop_jar_step requires optional attributes "main_class" and "properties"

atsalolikhin-spokeo commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.14.10 hashicorp/aws v3.51.0

Affected Resource(s)

aws_emr_cluster

hadoop_jar_step in step of aws_emr_cluster

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

See https://stackoverflow.com/q/62236138/10266059 for an example.

I ran into the same issue.

Terraform documentation says the attributes main_class and properties are optional, but when I run Terraform to add an EMR step (hadoop_jar_step), it errors out with:

 Inappropriate value for attribute "step": element 0: attribute "hadoop_jar_step": element 0: attributes "main_class" and "properties" are required.

Debug Output

Panic Output

Expected Behavior

Bring up an EMR cluster and add a Step.

Actual Behavior

Terraform errors out.

Steps to Reproduce

resource "aws_emr_cluster" "example" {
  # ... other configuration ...

  step = [
    {
      action_on_failure = "TERMINATE_CLUSTER"
      name              = "Setup Hadoop Debugging"

      hadoop_jar_step = [
        {
          jar  = "command-runner.jar"
          args = ["state-pusher-script"]
        }
      ]
    }
  ]

  # Optional: ignore outside changes to running cluster steps
  lifecycle {
    ignore_changes = [step]
  }
}

Important Factoids

References

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/emr_cluster

Either the documentation needs to be fixed, or the provider, right? Documentation says the attributes are optional but Terraform bombs out if they are not specified.

atsalolikhin-spokeo commented 3 years ago

The AWS EMR API documentation says MainClass and Properties are not required.

atsalolikhin-spokeo commented 3 years ago

Therefore I believe the provider needs to be fixed.

atsalolikhin-spokeo commented 3 years ago

In the meantime, I found and unzipped command-runner.jar on my EMR master.

META-INF/MANIFEST.MF has this line:

Main-Class: com.amazonaws.emr.command.runner.CommandRunner

So I've added the de facto required main_class and properties, and can now use Terraform to add a Hadoop step:

step = [
    {
      action_on_failure = "TERMINATE_CLUSTER"
      name              = "Setup Hadoop Debugging"

      hadoop_jar_step = [
        {
          jar        = "command-runner.jar"
          args       = ["state-pusher-script"]
          main_class = "com.amazonaws.emr.command.runner.CommandRunner"
          properties = {}
        }
      ]
    }
  ]

atsalolikhin-spokeo commented 3 years ago

While the step was added, the step failed:

[~/Downloads] $ gunzip -dc stderr.gz
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/aws/emr/emrfs/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Cannot run program "com.amazonaws.emr.command.runner.CommandRunner" (in directory "."): error=2, No such file or directory
        at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:140)
        at com.amazonaws.emr.command.runner.CommandRunner.main(CommandRunner.java:23)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: java.io.IOException: Cannot run program "com.amazonaws.emr.command.runner.CommandRunner" (in directory "."): error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
        at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:93)
        ... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
        at java.lang.ProcessImpl.start(ProcessImpl.java:134)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
        ... 8 more
[~/Downloads] $

atsalolikhin-spokeo commented 3 years ago

I tried changing jar to

          jar        = "/usr/share/aws/emr/command-runner/lib/command-runner.jar"

to specify the full path, but I'm still getting the same Java error.

atsalolikhin-spokeo commented 3 years ago

This worked as a workaround:

  step = [
    {
      action_on_failure = "TERMINATE_CLUSTER"
      name              = "Setup Hadoop Debugging"

      hadoop_jar_step = [
        {
          jar = "command-runner.jar"
          args = [
            "state-pusher-script"
          ]
          main_class = ""
          properties = {}
        }
      ]
    }
  ]

github-actions[bot] commented 3 years ago

This functionality has been released in v3.65.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-aws