search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
8.97k stars 1.86k forks source link

Glue - get_dev_endpoint() Does not return "FailureReason" #2167

Closed twnbay78 closed 4 years ago

twnbay78 commented 4 years ago

Upon spinning up dev endpoints, they often fail.

It is specified in the documentation that a "FailureReason" field is returned upon calling "get_dev_endpoint()" call:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get_dev_endpoint

This field is not included in the response, even though the endpoint status is listed as "FAILED":

{ 'DevEndpoint':{ 'EndpointName':'', 'RoleArn':'', 'SecurityGroupIds':[ '' ], 'SubnetId':'', 'YarnEndpointAddress':'', 'PrivateAddress':'', 'ZeppelinRemoteSparkInterpreterPort':, 'Status':'FAILED', 'GlueVersion':'1.0', 'NumberOfNodes':5, 'AvailabilityZone':'us-east-1a', 'VpcId':'', 'CreatedTimestamp':, 'LastModifiedTimestamp':, 'PublicKey':'' } }

It would be helpful to either update the documentation so that it doesn't mislead users, or provide the functionality for users to be able to see why their dev endpoint failed to be provisioned.

swetashre commented 4 years ago

@twnbay78 - Thank you for your post. I would need more information to know what is exactly happening. Can you please provide me full debug log ? You can enable log by adding boto3.set_stream_logger('') to your code.

twnbay78 commented 4 years ago

All I'm doing is performing a create_dev_endpoint() call and after the provisioning of the dev endpoint fails after a few minutes, I perform a get_dev_endpoint() call. The documentation claims that get_dev_endpoint() will return the failure reason but you can see from the HTTP request details that this field is not included. I have no idea why the endpoint is failing to be provisioned. The logs simply show boilerplate boto3/botocore with the raw HTTP headers for both calls, along with their responses.

Log for create_dev_endpoint() :

`019-10-18 14:33:21,282 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2019-10-18 14:33:21,286 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway 2019-10-18 14:33:21,286 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2019-10-18 14:33:21,289 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2019-10-18 14:33:21,289 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2019-10-18 14:33:21,289 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2019-10-18 14:33:21,290 botocore.hooks [DEBUG] Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2019-10-18 14:33:21,293 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2019-10-18 14:33:21,295 botocore.hooks [DEBUG] Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2019-10-18 14:33:21,296 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2019-10-18 14:33:21,296 botocore.hooks [DEBUG] Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2019-10-18 14:33:21,315 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/endpoints.json 2019-10-18 14:33:21,319 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x109175cb0> 2019-10-18 14:33:21,331 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/glue/2017-03-31/service-2.json 2019-10-18 14:33:21,341 botocore.hooks [DEBUG] Event creating-client-class.glue: calling handler <function add_generate_presigned_url at 0x109164830> 2019-10-18 14:33:21,341 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None 2019-10-18 14:33:21,344 botocore.endpoint [DEBUG] Setting glue timeout as (60, 60) 2019-10-18 14:33:21,344 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/_retry.json 2019-10-18 14:33:21,345 botocore.client [DEBUG] Registering retry handlers for service: glue 2019-10-18 14:33:21,506 botocore.hooks [DEBUG] Event before-parameter-build.glue.CreateDevEndpoint: calling handler <function generate_idempotent_uuid at 0x1091954d0> 2019-10-18 14:33:21,507 botocore.hooks [DEBUG] Event before-call.glue.CreateDevEndpoint: calling handler <function inject_api_version_header_if_needed at 0x109197f80> 2019-10-18 14:33:21,507 botocore.endpoint [DEBUG] Making request for OperationModel(name=CreateDevEndpoint) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AWSGlue.CreateDevEndpoint', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.9.234 Python/3.7.4 Darwin/18.7.0 Botocore/1.12.234'}, 'body': b'{"EndpointName": "test4", "RoleArn": "", "SubnetId": "", "SecurityGroupIds": [""], "PublicKey": "", "NumberOfNodes": 5, "GlueVersion": "1.0"}', 'url': 'https://glue.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x1097ed550>, 'has_streaming_input': False, 'auth_type': None}} 2019-10-18 14:33:21,507 botocore.hooks [DEBUG] Event request-created.glue.CreateDevEndpoint: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1097ed510>> 2019-10-18 14:33:21,508 botocore.hooks [DEBUG] Event choose-signer.glue.CreateDevEndpoint: calling handler <function set_operation_specific_signer at 0x1091953b0> 2019-10-18 14:33:21,508 botocore.auth [DEBUG] Calculating signature using v4 auth. 2019-10-18 14:33:21,508 botocore.auth [DEBUG] CanonicalRequest: POST /

content-type:application/x-amz-json-1.1 host:glue.us-east-1.amazonaws.com x-amz-date:20191018T183321Z x-amz-target:AWSGlue.CreateDevEndpoint

content-type;host;x-amz-date;x-amz-security-token;x-amz-target 9d7ac1fdb5c68ac5be12f72a60c4d168458e0b5a50725a079a63ef0e65a52872 2019-10-18 14:33:21,519 botocore.auth [DEBUG] StringToSign: AWS4-HMAC-SHA256 20191018T183321Z 20191018/us-east-1/glue/aws4_request 00f2df4d4bac0c8625807223a6e5d97b88806919a18605b819340dcbd75780cd 2019-10-18 14:33:21,519 botocore.auth [DEBUG] Signature: 2019-10-18 14:33:21,519 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://glue.us-east-1.amazonaws.com/, headers={'X-Amz-Target': b'AWSGlue.CreateDevEndpoint', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.9.234 Python/3.7.4 Darwin/18.7.0 Botocore/1.12.234', 'X-Amz-Date': b'20191018T183321Z', 'X-Amz-Security-Token': b'', 'Authorization': b'', 'Content-Length': '647'}> 2019-10-18 14:33:21,521 urllib3.util.retry [DEBUG] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None) 2019-10-18 14:33:21,521 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): glue.us-east-1.amazonaws.com:443 2019-10-18 14:33:22,543 urllib3.connectionpool [DEBUG] https://glue.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 200 377 2019-10-18 14:33:22,544 botocore.parsers [DEBUG] Response headers: {'Date': 'Fri, 18 Oct 2019 18:33:23 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '377', 'x-amzn-RequestId': 'c3b5cb84-f1d5-11e9-b876-51c234ddb2ab', 'Cache-Control': 'proxy-revalidate', 'Proxy-Connection': 'Keep-Alive', 'Connection': 'Keep-Alive'} 2019-10-18 14:33:22,544 botocore.parsers [DEBUG] Response body: b'{"AvailabilityZone":"us-east-1a","CreatedTimestamp":1.571423602576E9,"EndpointName":"test4","GlueVersion":"1.0","NumberOfNodes":5,"RoleArn":"","SecurityGroupIds":[""],"Status":"PROVISIONING","SubnetId":"","VpcId":"","ZeppelinRemoteSparkInterpreterPort":0}' 2019-10-18 14:33:22,545 botocore.hooks [DEBUG] Event needs-retry.glue.CreateDevEndpoint: calling handler <botocore.retryhandler.RetryHandler object at 0x1097ed9d0> 2019-10-18 14:33:22,545 botocore.retryhandler [DEBUG] No retry needed. `

` 2019-10-18 15:42:34,763 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2019-10-18 15:42:34,766 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway 2019-10-18 15:42:34,767 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2019-10-18 15:42:34,769 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2019-10-18 15:42:34,769 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2019-10-18 15:42:34,770 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2019-10-18 15:42:34,770 botocore.hooks [DEBUG] Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2019-10-18 15:42:34,773 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2019-10-18 15:42:34,774 botocore.hooks [DEBUG] Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2019-10-18 15:42:34,774 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2019-10-18 15:42:34,775 botocore.hooks [DEBUG] Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2019-10-18 15:42:34,802 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/endpoints.json 2019-10-18 15:42:34,806 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x102089c20> 2019-10-18 15:42:34,819 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/glue/2017-03-31/service-2.json 2019-10-18 15:42:34,829 botocore.hooks [DEBUG] Event creating-client-class.glue: calling handler <function add_generate_presigned_url at 0x102078710> 2019-10-18 15:42:34,829 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None 2019-10-18 15:42:34,832 botocore.endpoint [DEBUG] Setting glue timeout as (60, 60) 2019-10-18 15:42:34,833 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/_retry.json 2019-10-18 15:42:34,833 botocore.client [DEBUG] Registering retry handlers for service: glue 2019-10-18 15:42:34,834 botocore.hooks [DEBUG] Event before-parameter-build.glue.GetDevEndpoint: calling handler <function generate_idempotent_uuid at 0x1020a93b0> 2019-10-18 15:42:34,834 botocore.hooks [DEBUG] Event before-call.glue.GetDevEndpoint: calling handler <function inject_api_version_header_if_needed at 0x1020abe60> 2019-10-18 15:42:34,835 botocore.endpoint [DEBUG] Making request for OperationModel(name=GetDevEndpoint) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AWSGlue.GetDevEndpoint', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.9.234 Python/3.7.4 Darwin/18.7.0 Botocore/1.12.234'}, 'body': b'{"EndpointName": "test4"}', 'url': 'https://glue.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x1027004d0>, 'has_streaming_input': False, 'auth_type': None}} 2019-10-18 15:42:34,835 botocore.hooks [DEBUG] Event request-created.glue.GetDevEndpoint: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x102700490>> 2019-10-18 15:42:34,835 botocore.hooks [DEBUG] Event choose-signer.glue.GetDevEndpoint: calling handler <function set_operation_specific_signer at 0x1020a9290> 2019-10-18 15:42:34,835 botocore.auth [DEBUG] Calculating signature using v4 auth. 2019-10-18 15:42:34,835 botocore.auth [DEBUG] CanonicalRequest: POST /

content-type:application/x-amz-json-1.1 host:glue.us-east-1.amazonaws.com x-amz-date:20191018T194234Z x-amz-target:AWSGlue.GetDevEndpoint

content-type;host;x-amz-date;x-amz-security-token;x-amz-target 2019-10-18 15:42:34,836 botocore.auth [DEBUG] StringToSign: AWS4-HMAC-SHA256 20191018/us-east-1/glue/aws4_request 2019-10-18 15:42:34,836 botocore.auth [DEBUG] Signature: 2019-10-18 15:42:34,836 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://glue.us-east-1.amazonaws.com/, headers={'X-Amz-Target': b'AWSGlue.GetDevEndpoint', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.9.234 Python/3.7.4 Darwin/18.7.0 Botocore/1.12.234', 'X-Amz-Date': b'20191018T194234Z', 'X-Amz-Security-Token': b'', 'Content-Length': '25'}> 2019-10-18 15:42:34,837 urllib3.util.retry [DEBUG] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None) 2019-10-18 15:42:34,837 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): glue.us-east-1.amazonaws.com:443 2019-10-18 15:42:35,206 urllib3.connectionpool [DEBUG] https://glue.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 200 955 2019-10-18 15:42:35,207 botocore.parsers [DEBUG] Response headers: {'Date': 'Fri, 18 Oct 2019 19:42:35 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '955', 'x-amzn-RequestId': '6f21d5e1-f1df-11e9-9beb-cd999e37963a', 'Cache-Control': 'proxy-revalidate', 'Proxy-Connection': 'Keep-Alive', 'Connection': 'Keep-Alive'} 2019-10-18 15:42:35,207 botocore.parsers [DEBUG] Response body: b'{"DevEndpoint":{"AvailabilityZone":"us-east-1a","CreatedTimestamp":1.571423602576E9,"EndpointName":"test4","GlueVersion":"1.0","LastModifiedTimestamp":1.57142502365E9,"NumberOfNodes":5,"PrivateAddress":"","PublicKey":"","RoleArn":"","SecurityGroupIds":[""],"Status":"FAILED","SubnetId":"","VpcId":"","YarnEndpointAddress":"","ZeppelinRemoteSparkInterpreterPort":9007}}' 2019-10-18 15:42:35,209 botocore.hooks [DEBUG] Event needs-retry.glue.GetDevEndpoint: calling handler <botocore.retryhandler.RetryHandler object at 0x102700950> 2019-10-18 15:42:35,209 botocore.retryhandler [DEBUG] No retry needed.`

swetashre commented 4 years ago

Thank you for providing me with debug logs. This response is coming from service team. Boto3 only outsource the results to you. I would recommend contacting service team as they would be the appropriate team to know the reason behind the issue. You can contact the team either on their forum or create a ticket to AWS Support.

no-response[bot] commented 4 years ago

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.