Closed lakshman-kollipara closed 1 year ago
Hi @lakshman-kollipara thanks for reaching out. The first thing I recommend doing is updating your version of Boto3/Botocore. The latest Boto3 version is 1.28.50 per the CHANGELOG and the latest Botocore version is 1.31.50. (That may not resolve the issue but it is always a good thing to rule out first, especially considering that the versions you are using now are somewhat old and won't have the latest changes.) You also referenced boto 2.49.0 but that is no longer maintained.
I should also note that the start_job_run command corresponds to the EMR Serverless StartRunJob API. Therefore, if there is a bug involving the service API then we would need to escalate this to the EMR Serverless team.
Just to confirm the issue, can you share your debug logs (with any sensitive info redacted) after updating? You can get these by adding boto3.set_stream_logger('')
to your script.
Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.
Describe the bug
When passing
applicationConfiguration
array withspark-hive-site
classification, It's failing with below error:As per AWS spark-hive-site is valid configuration classification: ReleaseGuide
This is how my applicationConfiguration looks like:
Expected Behavior
Expect boto3 to pass configuration for
spark-hive-site
classification to AWS EMRCurrent Behavior
Throwing a
ValidationException
indicating thatUnknown configuration overrides classification: spark-hive-site
Reproduction Steps
call boto3 emr serverless client with below
applicationConfiguration
:Possible Solution
No response
Additional Information/Context
EMR 6.11.0
boto3==1.26.51 boto==2.49.0 botocore==1.29.51
python 3.10
SDK version used
boto3==1.26.51, boto==2.49.0, botocore==1.29.51
Environment details (OS name and version, etc.)
Mac Ventura 13.5.2 (22G91)