search

Azure / ACS

Azure Container Service - Bug Tracker + Announcements
65 stars 27 forks source link

Commas in NSG rules break the ability to create services in ACS K8S clusters #114

Open samueljmello opened 6 years ago

samueljmello commented 6 years ago

Is this a request for help?: No

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm) Kubernetes 1.7.7

What happened: My organization needed to add some NSG rules to our cluster. We added several, each containing multiple ports (ex: 80,443) in comma-separated notation. After we did that, creating services resulted in them being stuck in a <pending> status. Inspecting the kube-controller-manager pod logs resulted in the follow error:

servicecontroller.go:753] Failed to process service. Retrying in 5m0s: Failed to create load balancer for service default/my-service-name: network.SecurityGroupsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="SecurityRuleParametersMissingForPortRanges" Message="Required security rule parameters are missing for security rule with Id: /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Network/networkSecurityGroups/<nsg-name>/securityRules/<rule-with-commas-in-destination-port>. Security rule must specify either DestinationPortRange or DestinationPortRanges." Details=[]

(Values with <> are values that have been omitted for security reasons)

For some reason, all NSG rules are being processed by K8S and the commas are causing errors. Removing commas and doing a single port (ex: 80) or port range (ex: 80-81) works fine. My best guess was that whatever command is processing these rules is using commas to separate each parameter, and the commas in the ports parameter is throwing the parameter count off.

What you expected to happen: A service to be created

How to reproduce it (as minimally and precisely as possible):

  1. Create an ACS cluster
  2. Go to the cluster NSG and add (or edit) a rule to contain multiple ports separated by commas (ex: 80,443)
  3. Create a new load balancer service with external access in Kubernetes

Anything else we need to know: A similar issue report was made for AKS. We found it when searching for anyone else with this problem. It can be viewed here.

Our work-around has been to create multiple rules for each port we want to allow or deny. This is not ideal, as we have to duplicate IP's and other information that may need to be updated, making it a cumbersome and lengthy process.

feiskyer commented 6 years ago

Confirmed this is an issue caused by old Azure SDK vendors. It doesn't support DestinationPortRanges. Kubernetes v1.7.X has been deprecated (refer support timeframes here) since v1.10 has been released. Could you upgrade the cluster to v1.9 or v1.10 to fix the issue?

samueljmello commented 6 years ago

@feiskyer When we create a new cluster in ACS, it defaults to version 1.7.

ACS should be deploying the newer version of Kubernetes if that's the case. We regularly upgrade our ACS clusters and re-deploy our applications, so that's not a problem. Unfortunately ACS doesn't provide you the ability to select any version newer than 1.7.