Unknown configuration overrides classification: spark-hive-site(short issue description)

lakshman-kollipara commented 1 year ago

Describe the bug

When passing applicationConfiguration array with spark-hive-site classification, It's failing with below error:

botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the StartJobRun operation: Unknown configuration overrides classification: spark-hive-site

As per AWS spark-hive-site is valid configuration classification: ReleaseGuide

This is how my applicationConfiguration looks like:

  "applicationConfiguration": [
        {
            "classification": "hive-site",
            "properties": {
                "aws.glue.catalog.separator": "/",
                "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
                "hive.metastore.glue.catalogid": AWS_ACCOUNT_ID
            }
        },
        {
            "classification": "spark-hive-site",
            "properties": {
                "aws.glue.catalog.separator": "/",
                "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
                "hive.metastore.glue.catalogid": AWS_ACCOUNT_ID
            }
        }
    ]

Expected Behavior

Expect boto3 to pass configuration for spark-hive-site classification to AWS EMR

Current Behavior

Throwing a ValidationException indicating that Unknown configuration overrides classification: spark-hive-site

Reproduction Steps

call boto3 emr serverless client with below applicationConfiguration:

  "applicationConfiguration": [
        {
            "classification": "hive-site",
            "properties": {
                "aws.glue.catalog.separator": "/",
                "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
                "hive.metastore.glue.catalogid": AWS_ACCOUNT_ID
            }
        },
        {
            "classification": "spark-hive-site",
            "properties": {
                "aws.glue.catalog.separator": "/",
                "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
                "hive.metastore.glue.catalogid": AWS_ACCOUNT_ID
            }
        }
    ]

Possible Solution

No response

Additional Information/Context

EMR 6.11.0

boto3==1.26.51 boto==2.49.0 botocore==1.29.51

python 3.10

SDK version used

boto3==1.26.51, boto==2.49.0, botocore==1.29.51

Environment details (OS name and version, etc.)

Mac Ventura 13.5.2 (22G91)

tim-finnigan commented 1 year ago

Hi @lakshman-kollipara thanks for reaching out. The first thing I recommend doing is updating your version of Boto3/Botocore. The latest Boto3 version is 1.28.50 per the CHANGELOG and the latest Botocore version is 1.31.50. (That may not resolve the issue but it is always a good thing to rule out first, especially considering that the versions you are using now are somewhat old and won't have the latest changes.) You also referenced boto 2.49.0 but that is no longer maintained.

I should also note that the start_job_run command corresponds to the EMR Serverless StartRunJob API. Therefore, if there is a bug involving the service API then we would need to escalate this to the EMR Serverless team.

Just to confirm the issue, can you share your debug logs (with any sensitive info redacted) after updating? You can get these by adding boto3.set_stream_logger('') to your script.

github-actions[bot] commented 1 year ago

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

boto / boto3