search

Azure / azure-cli

Azure Command-Line Interface
MIT License
3.98k stars 2.95k forks source link

Cannot add k8s extension on azure stack hci #27303

Closed kenneth0595 closed 1 year ago

kenneth0595 commented 1 year ago

Describe the bug

I have tried to add k8s extension on my az stack hci cluster but I have encountered "Operation returned an invalid status 'Bad Request'

How to resolve this issue?

Related command

az k8s-extension create --cluster-type appliances --cluster-name $resource_name --resource-group $resource_group --name hci-vmoperator --extension-type Microsoft.AZStackHCI.Operator --scope cluster --release-namespace helm-operator2 --configuration-settings Microsoft.CustomLocation.ServiceAccount=hci-vmoperator --config-protected-file $csv_path\ResourceBridge\hci-config.json --configuration-settings HCIClusterID=$hciClusterId --auto-upgrade true --debug

Errors

Content:

400 Request Header Or Cookie Too Large

400 Bad Request

Request Header Or Cookie Too Large
nginx

Issue script & Debug output

cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/78d54a33-be25-4cad-824e-dea184279fc9/resourceGroups/HCIGROUP/providers/Microsoft.ResourceConnector/appliances/CloudGroup01-arcbridge/providers/Microsoft.KubernetesConfiguration/extensions/hci-vmoperator?api-version=2022-11-01' cli.azure.cli.core.sdk.policies: Request method: 'PUT' cli.azure.cli.core.sdk.policies: Request headers: cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json' cli.azure.cli.core.sdk.policies: 'Content-Length': '3900' cli.azure.cli.core.sdk.policies: 'Accept': 'application/json' cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': '9fd6b9e5-489a-11ee-89c5-4aa8046592e0' cli.azure.cli.core.sdk.policies: 'CommandName': 'k8s-extension create' cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--cluster-type --cluster-name --resource-group --name --extension-type --scope --release-namespace --configuration-settings --config-protected-file --configuration-settings --auto-upgrade --debug' cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.51.0 (MSI) azsdk-python-azure-mgmt-kubernetesconfiguration/2.0.0 Python/3.10.10 (Windows-10-10.0.20349-SP0)' cli.azure.cli.core.sdk.policies: 'Authorization': '*****' cli.azure.cli.core.sdk.policies: Request body: cli.azure.cli.core.sdk.policies: {"identity": {"type": "SystemAssigned"}, "properties": {"extensionType": "microsoft.azstackhci.operator", "autoUpgradeMinorVersion": true, "scope": {"cluster": {"releaseNamespace": "helm-operator2"}}, "configurationSettings": {"Microsoft.CustomLocation.ServiceAccount": "hci-vmoperator", "HCIClusterID": "/Subscriptions/78d54a33-be25-4cad-824e-dea184279fc9/resourceGroups/HCIGROUP/providers/Microsoft.AzureStackHCI/clusters/CloudGroup01"}, "configurationProtectedSettings": {"secret.cloudFQDN": "ca-cloudagent.c5cloud.internal", "secret.loginString": "bmFtZTogbW9jLW9wZXJhdG9yCnRva2VuOiBleUpoYkdjaU9pSlNVekkxTmlJc0ltdHBaQ0k2SW1JMk4yTTRZbU13T1dNMU5qZ3pZVEEzWXpoaE1EZzNNMk0yTnpFMU5USTBaR1k1TlRBMk5EaGxOVEF5TmpReE9EazBOekJtTldJM1pqa3dOelZtT1dRaUxDSjBlWEFpT2lKS1YxUWlmUS5leUp1WVcxbElqb2liVzlqTFc5d1pYSmhkRzl5SWl3aVpYaHdJam94TnpJMU1EZzFOak15TENKcWRHa2lPaUl6TmpVNU9UQTVZeUlzSW1saGRDSTZNVFk1TXpVME9UWXpNaXdpYVhOeklqb2lkM056WkdGblpXNTBjM1pqSW4wLkpDNERQdXd6X2lvckdlWUxmVU1za0xiSDlTa2MyR2pOeWhBYlQyYW45QjNuTGlnTTh0Z1ZLVzNmM1FNSlJHQU9Tb0xPbHBEeG1ZWEtFRVBJMVJlRnB2cFBSQnUwZm9Fd25MZzVzY09rOWlOd2JFbEo2eVlsMzJiQzZKRVNmekppbG9kWGg0cXprN2kzaHFvOTBsWFlUNldaUzg5dktSbGxITXg3eEZacnJuaHdqdFoxYXZxOVd4NFZid0dtSHd1Z2RIeUNQYldoTnp0eTFEV2V1cGF4QnpJcTJ5eEktSnFoV3lkVUxqelJTaEpYaTJTeDR1ZFBlVVN2RWlKWXM4dGhWX2pCc2k5dzh1NHQteXVKbTRTdmE2dzh0bThUMU5HRlJQZTdLTUdYSUJGa2RGT01XSWVGTm5Yck5IbWxvVlVrYVExa2JJRG5VWXBKeVNFX0xMN2FpdwpjZXJ0aWZpY2F0ZTogTFMwdExTMUNSVWRKVGlCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2sxSlNVUnFWRU5EUVc1WFowRjNTVUpCWjBsSlVIUTRTR1ptY3pNeFFrVjNSRkZaU2t0dldrbG9kbU5PUVZGRlRFSlJRWGRMVkVWVFRVSkJSMEV4VlVVS1EyaE5TbUpYYkdwamJUbDZZakphTUUxU1RYZEZVVmxFVmxGUlJFVjNjRVJpUnpreFdrVkdibHBYTlRCTlFqUllSRlJKZWsxRWEzZE5WRUV5VFZSWmVBcE5NVzlZUkZSTmVrMUVaM2xQVkVFeVRWUlplRTB4YjNkS1ZFVlRUVUpCUjBFeFZVVkRhRTFLWWxkc2FtTnRPWHBpTWxvd1RWRTRkMFJSV1VSV1VWRkVDa1YzV2xSYVdFb3lXbGhKZDJkblJXbE5RVEJIUTFOeFIxTkpZak5FVVVWQ1FWRlZRVUUwU1VKRWQwRjNaMmRGUzBGdlNVSkJVVU0zYVhsTlZ6bDFaVmNLYUN0blR6bFpiVk5CVHpKYVJtMDBVM0ZYUTFWVVdIZGpZVVJXV21scWRuQnJWbXRTUnpWbWRYRkZWRE5ZVkVSaE9WcFVZMUZ0VTNGT1EwdE1XblJTYkFweGNrZHdXbkJCVG1aRE4xQndVMWgzY1VsRFdXTjFPSE5QYTJGblJ6TkxNMUpHUkdJMFJsRkNNbmd4Y21KT1UwVTJTM2hQTjJ3d1NtRkdhWEJvVEdVMENsTk1ia3hIUXpOd1FYQTVjbWd4VUVsWmFqZFROMWh3SzFORE9WTkxSM0YyUWpOSWVHSXhSRko2WldwcmJYVkxjRmxwTjIxTVlWcFFTMDVOTnl0cGNETUtZbkpxVERSemVuY3pZa2QxVVM5MVRFSlFkR1p6ZFZkQ01XUllUbVJqWlVWa2NuWlhhVU0yTlZoeFpHNVhMMnRRTDFsdFRuZzJiRzlEVUZCeUwwOTZad3BRTUhWeFJuZDZUemMyVW1obGRVOWhPR3hHTkhob1JXdG1hMlJoWkU1dWEwOU9ha1YwZHpkQ1pGTmtiREZGYmxaTFpGRnJLM2hPUmpCUGNuaG1XRmhNQ2pVM2NsWTJNa2hOVm5kNk0wRm5UVUpCUVVkcVoySjNkMmRpYTNkRVoxbEVWbEl3VUVGUlNDOUNRVkZFUVdkWFowMUNNRWRCTVZWa1NsRlJWMDFDVVVjS1EwTnpSMEZSVlVaQ2QwMURRbWRuY2tKblJVWkNVV05FUVZSQlRVSm5UbFpJVWsxQ1FXWTRSVUZxUVVGTlFqaEhRVEZWWkVsM1VWbE5RbUZCUmtzd1VBcFZTM2xxYWk5eE4xaHpWekZHTVV4NFQzZFhOVzFJVERoTlJYTkhRVEZWWkVWUlVrVk5SVXREUTFkNGRsa3lSbk5oUnpsNlpFbEpTbE5GVGtwVFJUbFVDbFpFUVhsbmFEVnFXVk14YW1KSE9URmFSMFp1V2xjMU1FeHRUVEZaTW5oMlpGZFJkV0ZYTlRCYVdFcDFXVmQ1U0VKSU9FRkJRVWRJUWtGdlFVRkJkM2NLUkVGWlNVdDNXVUpDVVZWSVEyZEpSVUZFUVU1Q1oydHhhR3RwUnpsM01FSkJVWE5HUVVGUFEwRlJSVUZPY25kbGIwOVFVREJSVHpsMU5pdGpXRTV0VlFwWmJVcDFkbWxCV0d4b2JYTjJaR0kwUm1wS04yWkJSVFpQYWl0U1NFMXlUV3BIZG1oRFpuRnRaRzVSY0hBd1VFUTNWMk5FTW1NNFJtVm1kblZMZW01UENqQkdNbkZWWWxnNFFVRk1WM0ZDY1cxck1HbFpiVzFRWlUxSFZXaHRiSG96ZG14Qk9HbEdWRE5KTkd4R2RIWkVaRGh2WlRGMk5uQnNLMU5RY0hScFJqTUtNa2RYZVc1Q2FVWXhZV1pYVW1oaU1WTm5Va1F2UjBwa1ZVNTRWWGRQUlhSRmFVbENTMHRIVkRoNlJYVlhkM2RCYldkSVluQnhUREZIVG10Vlp5dG5RZ3BKUWxsTVFsWkpOV1o1U2sxMFFtZGpaemR1TUhsRVYxaHFWVUl6UlZSRk5FbDFXV3gzVFU5d01HSXdZVlZFYzBzclJHaHBkWFpFTldGMlRUZG9aR0p3Q201MmRDOVVVakpUWm10MVJrOU1NelIyTjB0WE5YWnFNVlZyT1cxNmMwZzBjVkpFWjNwYVRGZEZSRU16Y2xGblRrNXRTRE5XUTNGUlozZzBPREJFVkZNS2MzYzlQUW90TFMwdExVVk9SQ0JEUlZKVVNVWkpRMEZVUlMwdExTMHRDZz09CmNsaWVudHR5cGU6ICIiCmNsb3VkZnFkbjogY2EtY2xvdWRhZ2VudC5jNWNsb3VkLmludGVybmFsCmNsb3VkcG9ydDogNTUwMDAKY2xvdWRhdXRocG9ydDogNjUwMDAKbG9jYXRpb246IE1vY0xvY2F0aW9uCnR5cGU6IENBLVNpZ25lZAo="}}} urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443 urllib3.connectionpool: https://management.azure.com:443 "PUT /subscriptions/78d54a33-be25-4cad-824e-dea184279fc9/resourceGroups/HCIGROUP/providers/Microsoft.ResourceConnector/appliances/CloudGroup01-arcbridge/providers/Microsoft.KubernetesConfiguration/extensions/hci-vmoperator?api-version=2022-11-01 HTTP/1.1" 400 226 cli.azure.cli.core.sdk.policies: Response status: 400 cli.azure.cli.core.sdk.policies: Response headers: cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache' cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache' cli.azure.cli.core.sdk.policies: 'Content-Length': '226' cli.azure.cli.core.sdk.policies: 'Content-Type': 'text/html' cli.azure.cli.core.sdk.policies: 'Expires': '-1' cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-writes': '1199' cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff' cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '3981f40c-7374-4e24-af51-45fef5c8e955' cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '3981f40c-7374-4e24-af51-45fef5c8e955' cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'EASTASIA:20230901T073908Z:3981f40c-7374-4e24-af51-45fef5c8e955' cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains' cli.azure.cli.core.sdk.policies: 'X-Cache': 'CONFIG_NOCACHE' cli.azure.cli.core.sdk.policies: 'X-MSEdge-Ref': 'Ref A: FBE2C5ACFE2C4665AF872700FFDDF99A Ref B: SEL221051503053 Ref C: 2023-09-01T07:39:06Z' cli.azure.cli.core.sdk.policies: 'Date': 'Fri, 01 Sep 2023 07:39:07 GMT' cli.azure.cli.core.sdk.policies: Response content: cli.azure.cli.core.sdk.policies:

400 Request Header Or Cookie Too Large **

400 Bad Request

Request Header Or Cookie Too Large
nginx

**

azext_k8s_extension.vendored_sdks._serialization: Ran into a deserialization error. Ignoring since this is failsafe deserialization Traceback (most recent call last): File "C:\Users\Administrator.C5CLOUD.azure\cliextensions\k8s-extension\azext_k8s_extension\vendored_sdks_serialization.py", line 1452, in _deserialize found_value = key_extractor(attr, attr_desc, data) File "C:\Users\Administrator.C5CLOUD.azure\cliextensions\k8s-extension\azext_k8s_extension\vendored_sdks_serialization.py", line 1196, in rest_key_extractor return working_data.get(key) AttributeError: 'str' object has no attribute 'get'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\Administrator.C5CLOUD.azure\cliextensions\k8s-extension\azext_k8s_extension\vendored_sdks_serialization.py", line 1525, in failsafe_deserialize return self(target_obj, data, content_type=content_type) File "C:\Users\Administrator.C5CLOUD.azure\cliextensions\k8s-extension\azext_k8s_extension\vendored_sdks_serialization.py", line 1392, in call return self._deserialize(target_obj, data) File "C:\Users\Administrator.C5CLOUD.azure\cliextensions\k8s-extension\azext_k8s_extension\vendored_sdks_serialization.py", line 1470, in _deserialize raise_with_traceback(DeserializationError, msg, err) File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/exceptions.py", line 78, in raise_with_traceback File "C:\Users\Administrator.C5CLOUD.azure\cliextensions\k8s-extension\azext_k8s_extension\vendored_sdks_serialization.py", line 1452, in _deserialize found_value = key_extractor(attr, attr_desc, data) File "C:\Users\Administrator.C5CLOUD.azure\cliextensions\k8s-extension\azext_k8s_extension\vendored_sdks_serialization.py", line 1196, in rest_key_extractor return working_data.get(key) azure.core.exceptions.DeserializationError: (", AttributeError: 'str' object has no attribute 'get'", 'Unable to deserialize to object: type', AttributeError("'str' object has no attribute 'get'")) cli.azure.cli.core.azclierror: Traceback (most recent call last): File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 663, in execute File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 726, in _run_jobs_serially File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 697, in _run_job File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 333, in call File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler File "C:\Users\Administrator.C5CLOUD.azure\cliextensions\k8s-extension\azext_k8s_extension\custom.py", line 217, in create_k8s_extension return sdk_no_wait( File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/util.py", line 716, in sdk_no_wait File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer File "C:\Users\Administrator.C5CLOUD.azure\cliextensions\k8s-extension\azext_k8s_extension\vendored_sdks\v2022_11_01\operations_extensions_operations.py", line 507, in begin_create raw_result = self._create_initial( # type: ignore File "C:\Users\Administrator.C5CLOUD.azure\cliextensions\k8s-extension\azext_k8s_extension\vendored_sdks\v2022_11_01\operations_extensions_operations.py", line 343, in _create_initial raise HttpResponseError(response=response, model=error, error_format=ARMErrorFormat) azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Bad Request' Content:

400 Request Header Or Cookie Too Large

400 Bad Request

Request Header Or Cookie Too Large
nginx

cli.azure.cli.core.azclierror: Operation returned an invalid status 'Bad Request' az_command_data_logger: Operation returned an invalid status 'Bad Request' cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x0000010883980550>] az_command_data_logger: exit code: 1 cli.main: Command ran in 5.319 seconds (init: 0.446, invoke: 4.873) telemetry.main: Begin splitting cli events and extra events, total events: 1 telemetry.client: Accumulated 0 events. Flush the clients. telemetry.main: Finish splitting cli events and extra events, cli events: 1 telemetry.save: Save telemetry record of length 3684 in cache telemetry.main: Begin creating telemetry upload process. telemetry.process: Creating upload process: "C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry__init__.pyc C:\Users\Administrator.C5CLOUD.azure" telemetry.process: Return from creating process telemetry.main: Finish creating telemetry upload process.

Expected behavior

{ "aksAssignedIdentity": null, "autoUpgradeMinorVersion": true, "configurationProtectedSettings": { "secret.cloudFQDN": "", "secret.loginString": "" }, "configurationSettings": { "HCIClusterID": "/Subscriptions/78d54a33-be25-4cad-824e-dea184279fc9/resourceGroups/RG-HCIGROUP/providers/Microsoft.AzureStackHCI/clusters/HCICT20221113", "Microsoft.CustomLocation.ServiceAccount": "hci-vmoperator" }, "customLocationSettings": null, "errorInfo": null, "extensionType": "microsoft.azstackhci.operator", "id": "/subscriptions/78d54a33-be25-4cad-824e-dea184279fc9/resourceGroups/RG-HCIGROUP/providers/Microsoft.ResourceConnector/appliances/HCICT20221113-arcbridge/providers/Microsoft.KubernetesConfiguration/extensions/hci-vmoperator", "identity": { "principalId": "089660ba-e428-487b-a2ef-b0520505880f", "tenantId": null, "type": "SystemAssigned" }, "installedVersion": null, "name": "hci-vmoperator", "packageUri": null, "provisioningState": "Succeeded", "releaseTrain": "Stable", "resourceGroup": "RG-HCIGROUP", "scope": { "cluster": { "releaseNamespace": "helm-operator2" }, "namespace": null }, "statuses": [], "systemData": { "createdAt": "2022-11-13T11:18:09.034216+00:00", "createdBy": null, "createdByType": null, "lastModifiedAt": "2022-11-13T11:18:09.034216+00:00", "lastModifiedBy": null, "lastModifiedByType": null }, "type": "Microsoft.KubernetesConfiguration/extensions", "version": "1.2.8" }

Environment Summary

azure-cli 2.51.0

core 2.51.0 telemetry 1.1.0

Extensions: k8s-extension 1.4.3

Dependencies: msal 1.24.0b1 azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe' Extensions directory 'C:\Users\Administrator.C5CLOUD.azure\cliextensions'

Python (Windows) 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

No response

yonzhan commented 1 year ago

Thank you for opening this issue, we will look into it.

azure-client-tools-bot-prd[bot] commented 1 year ago
Hi @kenneth0595 Find similar issue https://github.com/Azure/azure-cli/issues/24107.
Issue title "az aks create" fails with error "400 Request Header Or Cookie Too Large"
Create time 2022-10-05
Comment number 5

Possible solution: The issue is related to the maximum request header size set to 1K. The http web server configuration currently has a maximum request header size set to 1K. For users who are members of a lot of security groups, their tokens can be quite large, upwards of 100K+. This causes the headers for those requests to be very large. The http web server configuration rejected the request for being too large. The Product Group has fixed this issue in the background. They have rolled back the manifest change as mitigation, and have a work item filed to change the http web server configuration to allow for the increased request header size. You can try to recreate the AKS cluster and check if that works now.

Please confirm if this resolves your issue.

kenneth0595 commented 1 year ago

I have already removed it and recreated it several times but I have not made it.

kenneth0595 commented 1 year ago

I have resolved this issue.