search

Azure / azure-rest-api-specs

The source for REST API specifications for Microsoft Azure.
MIT License
2.63k stars 5.06k forks source link

GatewayTimeout for `Microsoft.Sql/databases` with `api-version=2014-04-01` / package version `2017-03-01-preview` #16897

Open aristosvo opened 2 years ago

aristosvo commented 2 years ago

@azureSQLGitHub When trying to switch Zone Redundancy a SQL database with package version 2017-03-01-preview of the Microsoft.Sql API, GatewayTimeouts are showing up:

2021/11/29 10:50:29 [DEBUG] AzureRM Request: 
PUT /subscriptions/<subscriptionId>/resourceGroups/acctestRG-211129104546431506/providers/Microsoft.Sql/servers/acctestsqlserver211129104546431506/databases/acctestdb211129104546431506?api-version=2014-04-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.17.2 (amd64-darwin) go-autorest/v14.2.1 Azure-SDK-For-Go/v59.2.0 sql/2017-03-01-preview HashiCorp Terraform/1.0.10 (+https://www.terraform.io) Terraform Plugin SDK/2.8.0 terraform-provider-azurerm/acc pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 315
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: fbcfa19c-ed4a-d048-f9a0-ace304659c50
Accept-Encoding: gzip

{"location":"westeurope","properties":{"collation":"SQL_Latin1_General_CP1_CI_AS","createMode":"Default","edition":"Premium","maxSizeBytes":"1073741824","readScale":"Disabled","requestedServiceObjectiveId":"7203483a-c4fb-4304-9e9f-17c71c904f5d","requestedServiceObjectiveName":"P1","zoneRedundant":false},"tags":{}}
2021/11/29 10:51:29 [DEBUG] AzureRM Response for https://management.azure.com/subscriptions/<subscriptionId>/resourceGroups/acctestRG-211129104546431506/providers/Microsoft.Sql/servers/acctestsqlserver211129104546431506/databases/acctestdb211129104546431506?api-version=2014-04-01: 
HTTP/2.0 504 Gateway Timeout
Content-Length: 141
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Mon, 29 Nov 2021 09:51:31 GMT
Expires: -1
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: fbcfa19c-ed4a-d048-f9a0-ace304659c50
X-Ms-Failure-Cause: service
X-Ms-Request-Id: 30eb61c1-58d5-4e3d-80a4-906a08961189
X-Ms-Routing-Request-Id: WESTEUROPE:20211129T095131Z:30eb61c1-58d5-4e3d-80a4-906a08961189

{"error":{"code":"GatewayTimeout","message":"The gateway did not receive a response from 'Microsoft.Sql' within the specified time period."}}
2021/11/29 10:51:59 [DEBUG] AzureRM Request: 
PUT /subscriptions/<subscriptionId>/resourceGroups/acctestRG-211129104546431506/providers/Microsoft.Sql/servers/acctestsqlserver211129104546431506/databases/acctestdb211129104546431506?api-version=2014-04-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.17.2 (amd64-darwin) go-autorest/v14.2.1 Azure-SDK-For-Go/v59.2.0 sql/2017-03-01-preview HashiCorp Terraform/1.0.10 (+https://www.terraform.io) Terraform Plugin SDK/2.8.0 terraform-provider-azurerm/acc pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 315
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: fbcfa19c-ed4a-d048-f9a0-ace304659c50
Accept-Encoding: gzip

{"location":"westeurope","properties":{"collation":"SQL_Latin1_General_CP1_CI_AS","createMode":"Default","edition":"Premium","maxSizeBytes":"1073741824","readScale":"Disabled","requestedServiceObjectiveId":"7203483a-c4fb-4304-9e9f-17c71c904f5d","requestedServiceObjectiveName":"P1","zoneRedundant":false},"tags":{}}
2021/11/29 10:51:59 [DEBUG] AzureRM Response for https://management.azure.com/subscriptions/<subscriptionId>/resourceGroups/acctestRG-211129104546431506/providers/Microsoft.Sql/servers/acctestsqlserver211129104546431506/databases/acctestdb211129104546431506?api-version=2014-04-01: 
HTTP/2.0 400 Bad Request
Content-Length: 679
Cache-Control: no-store, no-cache
Content-Type: application/json
Dataserviceversion: 3.0;
Date: Mon, 29 Nov 2021 09:52:01 GMT
Preference-Applied: return-content
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: fbcfa19c-ed4a-d048-f9a0-ace304659c50
X-Ms-Ratelimit-Remaining-Subscription-Writes: 1198
X-Ms-Request-Id: 0ab302ab-83a1-4b0b-bcfc-026c09f2baea
X-Ms-Routing-Request-Id: WESTEUROPE:20211129T095202Z:8f34a8fc-c986-498e-b71a-d7d1fc55604e

{"code":"40802","message":"A service objective assignment on server \u0027acctestsqlserver211129104546431506\u0027 and database \u0027acctestdb211129104546431506\u0027 is already in progress. Please wait until the service objective assignment state for the database is marked as \u0027Completed\u0027.","target":null,"details":[{"code":"40802","message":"A service objective assignment on server \u0027acctestsqlserver211129104546431506\u0027 and database \u0027acctestdb211129104546431506\u0027 is already in progress. Please wait until the service objective assignment state for the database is marked as \u0027Completed\u0027.","target":null,"severity":"16"}],"innererror":[]}

It seems the time-out is triggering some action which blocks subsequent requests. GatewayTimeout is usually an indicator of bad error handling server side.

Linked issues

zhenglaizhang commented 2 years ago

@ericshape do you know which team I should contact for this issue?

ericshape commented 2 years ago

@aristosvo for these online service issues which impact your SLOs, I suggest contacting Azure Support and issuing an IcM ticket. Github API repo is not a good channel to provide Azure supports immedately. Our Azure supports are 24X7.

aristosvo commented 2 years ago

@ericshape I am not looking for support, at least not that I was aware of, these are issues which I faced during tests for the Terraform Provider for Azure.

I don't have a subscription which I can use with (paid) support for the community work I'm doing unfortunately, so if that's the way there is no channel for me to report these.

ericshape commented 2 years ago

I guess the error message provides hints {"code":"40802","message":"A service objective assignment on server \u0027acctestsqlserver211129104546431506\u0027 and database \u0027acctestdb211129104546431506\u0027 is already in progress. Please wait until the service objective assignment state for the database is marked as \u0027Completed\u0027.","target":null,"details":[{"code":"40802","message":"A service objective assignment on server \u0027acctestsqlserver211129104546431506\u0027 and database \u0027acctestdb211129104546431506\u0027 is already in progress. Please wait until the service objective assignment state for the database is marked as \u0027Completed\u0027.","target":null,"severity":"16"}],"innererror":[]}

The object assignment is in progress.

aristosvo commented 2 years ago

@ericshape True, but this is caused by the action which went timeout just before. This timeout is crucial in this behavior, as it still acts on the resource but feedback isn't given.

It's impossible to code against an API which has such behavior, and as such needs a fix. The problem is not the final error, but the GatewayTimeout just before.

ericshape commented 2 years ago

@aristosvo I have created an internal support ticket to track it. please send me (ji dot wang A_T_symbol microsoft_dot_com) your email and I can let you in the loop.

ghost commented 2 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureSQLGitHub.

Issue Details
@azureSQLGitHub When trying to switch Zone Redundancy a SQL database with package version 2017-03-01-preview of the Microsoft.Sql API, GatewayTimeouts are showing up: ```bash 2021/11/29 10:50:29 [DEBUG] AzureRM Request: PUT /subscriptions//resourceGroups/acctestRG-211129104546431506/providers/Microsoft.Sql/servers/acctestsqlserver211129104546431506/databases/acctestdb211129104546431506?api-version=2014-04-01 HTTP/1.1 Host: management.azure.com User-Agent: Go/go1.17.2 (amd64-darwin) go-autorest/v14.2.1 Azure-SDK-For-Go/v59.2.0 sql/2017-03-01-preview HashiCorp Terraform/1.0.10 (+https://www.terraform.io) Terraform Plugin SDK/2.8.0 terraform-provider-azurerm/acc pid-222c6c49-1b0a-5959-a213-6608f9eb8820 Content-Length: 315 Content-Type: application/json; charset=utf-8 X-Ms-Correlation-Request-Id: fbcfa19c-ed4a-d048-f9a0-ace304659c50 Accept-Encoding: gzip {"location":"westeurope","properties":{"collation":"SQL_Latin1_General_CP1_CI_AS","createMode":"Default","edition":"Premium","maxSizeBytes":"1073741824","readScale":"Disabled","requestedServiceObjectiveId":"7203483a-c4fb-4304-9e9f-17c71c904f5d","requestedServiceObjectiveName":"P1","zoneRedundant":false},"tags":{}} 2021/11/29 10:51:29 [DEBUG] AzureRM Response for https://management.azure.com/subscriptions//resourceGroups/acctestRG-211129104546431506/providers/Microsoft.Sql/servers/acctestsqlserver211129104546431506/databases/acctestdb211129104546431506?api-version=2014-04-01: HTTP/2.0 504 Gateway Timeout Content-Length: 141 Cache-Control: no-cache Content-Type: application/json; charset=utf-8 Date: Mon, 29 Nov 2021 09:51:31 GMT Expires: -1 Pragma: no-cache Strict-Transport-Security: max-age=31536000; includeSubDomains X-Content-Type-Options: nosniff X-Ms-Correlation-Request-Id: fbcfa19c-ed4a-d048-f9a0-ace304659c50 X-Ms-Failure-Cause: service X-Ms-Request-Id: 30eb61c1-58d5-4e3d-80a4-906a08961189 X-Ms-Routing-Request-Id: WESTEUROPE:20211129T095131Z:30eb61c1-58d5-4e3d-80a4-906a08961189 {"error":{"code":"GatewayTimeout","message":"The gateway did not receive a response from 'Microsoft.Sql' within the specified time period."}} 2021/11/29 10:51:59 [DEBUG] AzureRM Request: PUT /subscriptions//resourceGroups/acctestRG-211129104546431506/providers/Microsoft.Sql/servers/acctestsqlserver211129104546431506/databases/acctestdb211129104546431506?api-version=2014-04-01 HTTP/1.1 Host: management.azure.com User-Agent: Go/go1.17.2 (amd64-darwin) go-autorest/v14.2.1 Azure-SDK-For-Go/v59.2.0 sql/2017-03-01-preview HashiCorp Terraform/1.0.10 (+https://www.terraform.io) Terraform Plugin SDK/2.8.0 terraform-provider-azurerm/acc pid-222c6c49-1b0a-5959-a213-6608f9eb8820 Content-Length: 315 Content-Type: application/json; charset=utf-8 X-Ms-Correlation-Request-Id: fbcfa19c-ed4a-d048-f9a0-ace304659c50 Accept-Encoding: gzip {"location":"westeurope","properties":{"collation":"SQL_Latin1_General_CP1_CI_AS","createMode":"Default","edition":"Premium","maxSizeBytes":"1073741824","readScale":"Disabled","requestedServiceObjectiveId":"7203483a-c4fb-4304-9e9f-17c71c904f5d","requestedServiceObjectiveName":"P1","zoneRedundant":false},"tags":{}} 2021/11/29 10:51:59 [DEBUG] AzureRM Response for https://management.azure.com/subscriptions//resourceGroups/acctestRG-211129104546431506/providers/Microsoft.Sql/servers/acctestsqlserver211129104546431506/databases/acctestdb211129104546431506?api-version=2014-04-01: HTTP/2.0 400 Bad Request Content-Length: 679 Cache-Control: no-store, no-cache Content-Type: application/json Dataserviceversion: 3.0; Date: Mon, 29 Nov 2021 09:52:01 GMT Preference-Applied: return-content Server: Microsoft-HTTPAPI/2.0 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Content-Type-Options: nosniff X-Ms-Correlation-Request-Id: fbcfa19c-ed4a-d048-f9a0-ace304659c50 X-Ms-Ratelimit-Remaining-Subscription-Writes: 1198 X-Ms-Request-Id: 0ab302ab-83a1-4b0b-bcfc-026c09f2baea X-Ms-Routing-Request-Id: WESTEUROPE:20211129T095202Z:8f34a8fc-c986-498e-b71a-d7d1fc55604e {"code":"40802","message":"A service objective assignment on server \u0027acctestsqlserver211129104546431506\u0027 and database \u0027acctestdb211129104546431506\u0027 is already in progress. Please wait until the service objective assignment state for the database is marked as \u0027Completed\u0027.","target":null,"details":[{"code":"40802","message":"A service objective assignment on server \u0027acctestsqlserver211129104546431506\u0027 and database \u0027acctestdb211129104546431506\u0027 is already in progress. Please wait until the service objective assignment state for the database is marked as \u0027Completed\u0027.","target":null,"severity":"16"}],"innererror":[]} ``` It seems the time-out is triggering some action which blocks subsequent requests. GatewayTimeout is usually an indicator of bad error handling server side. ## Linked issues - Similar to #16838, except that I don't know how to circumvent this one (yet). - It causes failures in the acceptance tests for `azurerm`, which can be triggered by running `git clone https://github.com/hashicorp/terraform-provider-azurerm.git && cd terraform-provider-azurerm && go install && make acctests SERVICE='sql' TESTARGS='-run=TestAccSqlDatabase_zoneRedundant'`
Author: aristosvo
Assignees: -
Labels: `SQL`, `Service Attention`
Milestone: -