search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
9.02k stars 1.87k forks source link

create_data_quality_job_definition doesn't accept InferenceAttribute #3794

Closed szafranek closed 1 year ago

szafranek commented 1 year ago

Describe the issue

create_data_quality_job_definition docs list InferenceAttribute as one of the attributes under DataQualityJobInput.EndpointInput. However, passing it results in an error:

An error occurred (ValidationException) when calling the CreateDataQualityJobDefinition operation:
InvalidParameter: 1 validation error(s) found.
- format unsupported with data quality monitoring, , 
CreateDataQualityJobDefinitionInput.DataQualityJobInput.EndpointInput.InferenceAttribute.

EndpointInput attributes EndpointName and LocalPath do work. I didn't test other attributes.

Tested with boto3 version 1.28.2 and Python 3.11.1.

Links

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/client/create_data_quality_job_definition.html

tim-finnigan commented 1 year ago

Hi @szafranek thanks for reaching out. Could you provide a code snippet for reproducing the issue? And could you also share your full debug logs (with sensitive info redacted) by adding boto3.set_stream_logger('') to your script?

1.28.2 is a pretty recent version but there were some SageMaker updates in 1.28.8 per the CHANGELOG so I recommend updating Boto3 and trying again. Depending on the information above we may need to escalate this issue to the SageMaker team for further review as they own the underlying CreateDataQualityJobDefinition API.

github-actions[bot] commented 1 year ago

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

szafranek commented 1 year ago

Hi @tim-finnigan,

Here's a code snippet I used to replicate the issue. I tested it with latest boto3 (1.28.15 as of today) and it's still reproducible.

import boto3

client = boto3.client('sagemaker')

response = client.create_data_quality_job_definition(
    JobDefinitionName='cszafranek-boto3-test-data-quality-job',
    DataQualityBaselineConfig={
        'BaseliningJobName': 'data-quality-monitoring-baseline-280720231638',
        'ConstraintsResource': {
            'S3Uri': 's3://<REDACTED>/data_quality/baseline_output/constraints.json'
        },
        'StatisticsResource': {
            'S3Uri': 's3://<REDACTED>/data_quality/baseline_output/statistics.json'
        }
    },
    DataQualityAppSpecification={
        'ImageUri': '048819808253.dkr.ecr.eu-central-1.amazonaws.com/sagemaker-model-monitor-analyzer',
    },
    DataQualityJobInput={
        'EndpointInput': {
            'EndpointName': 'sagemaker-model-monitoring',
            'LocalPath': '/opt/ml/processing/input/endpoint',
            'S3InputMode': 'File',
            'S3DataDistributionType': 'FullyReplicated',
            'InferenceAttribute': 'churn',
        },
    },
    DataQualityJobOutputConfig={
        'MonitoringOutputs': [
            {
                'S3Output': {
                    'S3Uri': 's3://<REDACTED>/data_quality_reports',
                    'LocalPath': '/opt/ml/processing/output'
                }
            },
        ]
    },
    JobResources={
        'ClusterConfig': {
            'InstanceCount': 1,
            'InstanceType': 'ml.m5.large',
            'VolumeSizeInGB': 1,
        }
    },
    RoleArn='arn:aws:iam::<REDACTED>:role/<REDACTED>'
,
    Tags=[
        {
            'Key': 'creator',
            'Value': 'cszafranek'
        },
    ]
)

I enabled extra debugging and I'm pasting it below the code.

2023-07-31 16:38:00,676 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x103bb9da0>
2023-07-31 16:38:00,678 botocore.hooks [DEBUG] Event creating-client-class.sagemaker: calling handler <function add_generate_presigned_url at 0x103afd800>
2023-07-31 16:38:00,679 botocore.configprovider [DEBUG] Looking for endpoint for sagemaker via: environment_service
2023-07-31 16:38:00,679 botocore.configprovider [DEBUG] Looking for endpoint for sagemaker via: environment_global
2023-07-31 16:38:00,679 botocore.configprovider [DEBUG] Looking for endpoint for sagemaker via: config_service
2023-07-31 16:38:00,679 botocore.configprovider [DEBUG] Looking for endpoint for sagemaker via: config_global
2023-07-31 16:38:00,679 botocore.configprovider [DEBUG] No configured endpoint found.
2023-07-31 16:38:00,682 botocore.endpoint [DEBUG] Setting api.sagemaker timeout as (60, 60)
2023-07-31 16:38:00,683 botocore.client [DEBUG] Registering retry handlers for service: sagemaker
2023-07-31 16:38:00,684 botocore.hooks [DEBUG] Event before-parameter-build.sagemaker.CreateDataQualityJobDefinition: calling handler <function generate_idempotent_uuid at 0x103bbb7e0>
2023-07-31 16:38:00,684 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Region': 'eu-central-1', 'UseDualStack': False, 'UseFIPS': False}
2023-07-31 16:38:00,684 botocore.regions [DEBUG] Endpoint provider result: https://api.sagemaker.eu-central-1.amazonaws.com
2023-07-31 16:38:00,685 botocore.hooks [DEBUG] Event before-call.sagemaker.CreateDataQualityJobDefinition: calling handler <function add_recursion_detection_header at 0x103bbaca0>
2023-07-31 16:38:00,685 botocore.hooks [DEBUG] Event before-call.sagemaker.CreateDataQualityJobDefinition: calling handler <function inject_api_version_header_if_needed at 0x103bd9300>
2023-07-31 16:38:00,685 botocore.endpoint [DEBUG] Making request for OperationModel(name=CreateDataQualityJobDefinition) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'SageMaker.CreateDataQualityJobDefinition', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.28.15 md/Botocore#1.31.15 ua/2.0 os/macos#22.5.0 md/arch#arm64 lang/python#3.11.4 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.15'}, 'body': b'{"JobDefinitionName": "szafranek-boto3-test-data-quality-job", "DataQualityBaselineConfig": {"BaseliningJobName": "data-quality-monitoring-baseline-280720231638", "ConstraintsResource": {"S3Uri": "s3://<REDACTED>/data_quality/baseline_output/constraints.json"}, "StatisticsResource": {"S3Uri": "s3://<REDACTED>/data_quality/baseline_output/statistics.json"}}, "DataQualityAppSpecification": {"ImageUri": "048819808253.dkr.ecr.eu-central-1.amazonaws.com/sagemaker-model-monitor-analyzer"}, "DataQualityJobInput": {"EndpointInput": {"EndpointName": "sagemaker-model-monitoring", "LocalPath": "/opt/ml/processing/input/endpoint", "S3InputMode": "File", "S3DataDistributionType": "FullyReplicated", "InferenceAttribute": "churn"}}, "DataQualityJobOutputConfig": {"MonitoringOutputs": [{"S3Output": {"S3Uri": "s3://<REDACTED>/data_quality_reports", "LocalPath": "/opt/ml/processing/output"}}]}, "JobResources": {"ClusterConfig": {"InstanceCount": 1, "InstanceType": "ml.m5.large", "VolumeSizeInGB": 1}}, "RoleArn": "<REDACTED>", "Tags": [{"Key": "creator", "Value": "szafranek"}]}', 'url': 'https://api.sagemaker.eu-central-1.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x105042590>, 'has_streaming_input': False, 'auth_type': None}}
2023-07-31 16:38:00,685 botocore.hooks [DEBUG] Event request-created.sagemaker.CreateDataQualityJobDefinition: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x105043e50>>
2023-07-31 16:38:00,685 botocore.hooks [DEBUG] Event choose-signer.sagemaker.CreateDataQualityJobDefinition: calling handler <function set_operation_specific_signer at 0x103bbb6a0>
2023-07-31 16:38:00,685 botocore.auth [DEBUG] Calculating signature using v4 auth.
2023-07-31 16:38:00,685 botocore.auth [DEBUG] CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:api.sagemaker.eu-central-1.amazonaws.com
x-amz-date:20230731T143800Z
x-amz-security-token:<REDACTED>
x-amz-target:SageMaker.CreateDataQualityJobDefinition

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
<REDACTED>
2023-07-31 16:38:00,686 botocore.auth [DEBUG] StringToSign:
<REDACTED>
20230731T143800Z
20230731/eu-central-1/sagemaker/aws4_request
<REDACTED>
2023-07-31 16:38:00,686 botocore.auth [DEBUG] Signature:
<REDACTED>
2023-07-31 16:38:00,686 botocore.hooks [DEBUG] Event request-created.sagemaker.CreateDataQualityJobDefinition: calling handler <function add_retry_headers at 0x103bd9a80>
2023-07-31 16:38:00,686 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://api.sagemaker.eu-central-1.amazonaws.com/, headers={'X-Amz-Target': b'SageMaker.CreateDataQualityJobDefinition', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.28.15 md/Botocore#1.31.15 ua/2.0 os/macos#22.5.0 md/arch#arm64 lang/python#3.11.4 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.15', 'X-Amz-Date': b'20230731T143800Z', 'X-Amz-Security-Token': b'<REDACTED>', 'Authorization': b'<REDACTED> Credential=<REDACTED>/20230731/eu-central-1/sagemaker/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=<REDACTED>', 'amz-sdk-invocation-id': b'<REDACTED>', 'amz-sdk-request': b'attempt=1', 'Content-Length': '1225'}>
2023-07-31 16:38:00,686 botocore.httpsession [DEBUG] Certificate path: <REDACTED>venv/lib/python3.11/site-packages/certifi/cacert.pem
2023-07-31 16:38:00,686 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): api.sagemaker.eu-central-1.amazonaws.com:443
2023-07-31 16:38:01,069 urllib3.connectionpool [DEBUG] https://api.sagemaker.eu-central-1.amazonaws.com:443 "POST / HTTP/1.1" 400 237
2023-07-31 16:38:01,071 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': '<REDACTED>', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '237', 'Date': 'Mon, 31 Jul 2023 14:38:00 GMT', 'Connection': 'close'}
2023-07-31 16:38:01,071 botocore.parsers [DEBUG] Response body:
b'{"__type":"ValidationException","message":"InvalidParameter: 1 validation error(s) found.\\n- format unsupported with data quality monitoring, , CreateDataQualityJobDefinitionInput.DataQualityJobInput.EndpointInput.InferenceAttribute.\\n"}'
2023-07-31 16:38:01,082 botocore.hooks [DEBUG] Event needs-retry.sagemaker.CreateDataQualityJobDefinition: calling handler <botocore.retryhandler.RetryHandler object at 0x10489de90>
2023-07-31 16:38:01,082 botocore.retryhandler [DEBUG] No retry needed.
---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
Cell In[10], line 6
      1 import boto3
      4 client = boto3.client('sagemaker')
----> 6 response = client.create_data_quality_job_definition(
      7     JobDefinitionName='szafranek-boto3-test-data-quality-job',
      8     DataQualityBaselineConfig={
      9         'BaseliningJobName': 'data-quality-monitoring-baseline-280720231638',
     10         'ConstraintsResource': {
     11             'S3Uri': 's3://<REDACTED>/constraints.json'
     12         },
     13         'StatisticsResource': {
     14             'S3Uri': 's3://<REDACTED>/statistics.json'
     15         }
     16     },
     17     DataQualityAppSpecification={
     18         'ImageUri': '048819808253.dkr.ecr.eu-central-1.amazonaws.com/sagemaker-model-monitor-analyzer',
     19     },
     20     DataQualityJobInput={
     21         'EndpointInput': {
     22             'EndpointName': 'sagemaker-model-monitoring',
     23             'LocalPath': '/opt/ml/processing/input/endpoint',
     24             'S3InputMode': 'File',
     25             'S3DataDistributionType': 'FullyReplicated',
     26             'InferenceAttribute': 'churn',
     27         },
     28     },
     29     DataQualityJobOutputConfig={
     30         'MonitoringOutputs': [
     31             {
     32                 'S3Output': {
     33                     'S3Uri': 's3://<REDACTED>/data_quality_reports',
     34                     'LocalPath': '/opt/ml/processing/output'
     35                 }
     36             },
     37         ]
     38     },
     39     JobResources={
     40         'ClusterConfig': {
     41             'InstanceCount': 1,
     42             'InstanceType': 'ml.m5.large',
     43             'VolumeSizeInGB': 1,
     44         }
     45     },
     46     RoleArn='<REDACTED>',
     47     Tags=[
     48         {
     49             'Key': 'creator',
     50             'Value': 'szafranek'
     51         },
     52     ]
     53 )

File ~/<REDACTED>/venv/lib/python3.11/site-packages/botocore/client.py:535, in ClientCreator._create_api_method.<locals>._api_call(self, *args, **kwargs)
    531     raise TypeError(
    532         f"{py_operation_name}() only accepts keyword arguments."
    533     )
    534 # The "self" in this scope is referring to the BaseClient.
--> 535 return self._make_api_call(operation_name, kwargs)

File ~/<REDACTED>/venv/lib/python3.11/site-packages/botocore/client.py:980, in BaseClient._make_api_call(self, operation_name, api_params)
    978     error_code = parsed_response.get("Error", {}).get("Code")
    979     error_class = self.exceptions.from_code(error_code)
--> 980     raise error_class(parsed_response, operation_name)
    981 else:
    982     return parsed_response

ClientError: An error occurred (ValidationException) when calling the CreateDataQualityJobDefinition operation: InvalidParameter: 1 validation error(s) found.
- format unsupported with data quality monitoring, , CreateDataQualityJobDefinitionInput.DataQualityJobInput.EndpointInput.InferenceAttribute.

Once I remove InferenceAttribute, this error disappears.

tim-finnigan commented 1 year ago

Hi @szafranek thanks for following up. I'm not able to reproduce this issue using your code snippet and the boto3/botocore versions you provided. Are you using custom botocore models? I recommend reinstalling boto3/botocore and verifying that you don't have custom models overriding the expected behavior.

If I changed the InferenceAttribute param type to a dict then it fails as expected:

botocore.exceptions.ParamValidationError: Parameter validation failed: Invalid type for parameter DataQualityJobInput.EndpointInput.InferenceAttribute, value: {'invalid': 'param'}, type: <class 'dict'>, valid types: <class 'str'>

Based on your logs it looks the issue is with a \\n in your code which I don't see in your snippet. So it could be a hidden pattern getting added that is causing the validation to fail.

github-actions[bot] commented 1 year ago

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.