SubhashVasarapu-MSFT
commented
4 years ago @lucidqdreams , We are looking into this query and will update you as soon as possible.
Closed lucidqdreams closed 4 years ago
SubhashVasarapu-MSFT
commented
4 years ago @lucidqdreams , We are looking into this query and will update you as soon as possible.
lucidqdreams
commented
4 years ago my mistake I showed you service, not endpoints... in the screenshots. This will match up to IPs listed in the back end pools
lucidqdreams
commented
4 years ago any progress here? I actually have been trying to get cert-manager working as well which it was working when I went through the process manually and I dont think it used a URL path based request. though I'm not 100% sure about that. I'm trying to script it and now I have a 502 bad gateway trying to get the cert from a URL path based request.
lucidqdreams
commented
4 years ago it appears the health probes are failing for the service using the URLpath
If I change the path option to / the probe works
I thought the purpose of this was to allow you to mount a website to a URL/path? have I misunderstood this or is this how it is meant to work?
SubhashVasarapu-MSFT
commented
4 years ago Application Gateway Probe is bound to HTTP Settings. So probe to the Backend pool will look something like this (http/https://hostname:HTTPSettingsport/Path). Generally, Path here refers to Relative path of the probe. The valid path starts from '/'. You can also provide custom probe here where you need to mention complete URL path that is invoked by the application gateway to determine the health of the instance. For example, if you have a website http://contoso.com/, then the custom probe can be configured for "http://contoso.com/path/custompath.htm" for probe checks to have a successful HTTP response. You check this doc for you reference.
lucidqdreams
commented
4 years ago Right, but this is a probe setup by kubernetes. I don't have a shared instance so kubernetes should be controlling all features of the ingress via the application gateway without me having to update any settings in the application gateway. Also If I change anything the k8s aad pod should theoretically change it back.
What I understand the url path to be is that I can mask a back end service hosted in k8s and present It on the url/path. This should be able to be done through a yaml ingress template in k8s.
My observations are that this doesn't work as it should. Secondly what I noticed was that the probe for the service behind the url/path fails because its mapping to the service/path.
So which one of part of this statement is true?
You can map a url/path via app gateway to a k8s service as long as that service responds to that path. Eg a pod called aspnet which must have path /app that is aspnet/app can be mapped to URL/app. {OR} But a Pod aspnet/ at the root cannot be mapped to url/app
If the latter is true, how can I mount let's say the aspnet sample app to a path aspnet/app so that I can map the application gateway url/path to the pod/path
lucidqdreams
commented
4 years ago This text has no mention of powershell or probes. It's says I can add additional paths to expose other services from k8s directly
This talks about redirecting url/test/ to the path url/hello/ for the service go-server-service. So does this mean go-server-service has a native path inside the container called /hello/?
What would be useful is being able to map any url/path to any service without that path existing in the container. The instructions lead me to believe this is possible. Either its a product bug or the instructions need some additional clarification Or I've done something wrong.
lucidqdreams
commented
4 years ago I can share my script. This isn't perfect as it's still a work in progress and I had hoped to publish it once I had got it working. You cant run it end to end but if you run it in sections and it should get you to the URL mapping result I see above. This screenshot shows the annotations and tags showing the APP gw is controlled by K8s
#Azure Cloud Shell Bash
# base parameters
solutionname="flab"
subscription="Labs"
rg="$solutionname"
location="westus2"
vnetName="$solutionname-vnet"
aksSubnetName="kubernetes"
agwSubnetName="agw"
# these arent used if used for existing rg & vnet
vnetAddressSpace="10.9.0.0/16"
aksSubnetCIDR="10.9.1.0/24"
agwSubnetCIDR="10.9.2.0/24"
#AKS cluster parameters
aksClusterName="$rg-k8s"
aksDockerBridgeAddress='172.17.0.1/16'
aksDnsServiceIP='10.2.0.10'
aksServiceCIDR='10.2.0.0/24'
#Application Gateway parameters
agwName="$solutionname-agw"
agwPIP="$solutionname-pip"
agwSku="Standard_v2"
#AKS node pool 1
PrimaryNodePoolk8sVersion="1.15.7"
PrimaryNodePoolNodeSize="Standard_DS2_v2"
PrimaryNodePoolNodeCount=3
PrimaryNodePoolmaxpods=20
PrimaryNodePoolmincount=1
PrimaryNodePoolmaxcount=3
#AKS node pool 2
SecondaryNodePool="nodepool2"
SecondaryNodePoolk8sVersion="1.14.8"
SecondaryNodePoolNodeSize="Standard_DS2_v2"
SecondaryNodePoolCount=3
SecondaryNodemaxpods=10
SecondaryNodemincount=1
SecondaryNodemaxcount=3
#DNS Parameters
#certificate Manager
LetsEncryptEmail='my.email@gmail.com'
#Ingress Parameters
domainsuffix=".domain.com"
domaintext="$solutionname"
domainname="$domaintext$domainsuffix"
SecretName="$domaintext-tls"
SecretNameStage="$SecretName-stage"
URLPath="/app/*"
### Working Logice
echo "creating working directory in cloud shell"
cd ~
mkdir $rg-$aksClusterName
cd $rg-$aksClusterName
alias k=kubectl
echo "working directory and files '$rg-$aksClusterName'" > deployment.log
## Set up environment
az account set --subscription $subscription
subscriptionId=$(az account show --subscription $subscription | jq -r ".id")
echo "Subscription '$subscription' with ID '$subscriptionId' has been targeted" >> deployment.log
## Create Resource Group (testing)
az group create -l $location -n $rg
## Create virtual Network and AKS subnet (testing)
az network vnet create \
--name $vnetName \
--resource-group $rg \
--location $location \
--address-prefix $vnetAddressSpace \
--subnet-name $aksSubnetName \
--subnet-prefix $aksSubnetCIDR
# Create virtual agw subnet (testing)az
az network vnet subnet create \
--name $agwSubnetName \
--resource-group $rg \
--vnet-name $vnetName \
--address-prefix $agwSubnetCIDR
## Create Public IP Address
az network public-ip create --name $agwPIP --resource-group $rg --allocation-method 'Static' --sku 'Standard'
## Get resource group & subnet resource IDs
rgid=$(az group show --name $rg | jq -r ".id" )
echo "found resource group '$rgid'" >> deployment.log
aksSubnetId=$(az network vnet subnet show -g $rg -n $aksSubnetName --vnet-name $vnetName | jq -r '.id')
echo "found subnet '$aksSubnetId'" >> deployment.log
agwSubnetId=$(az network vnet subnet show -g $rg -n $agwSubnetName --vnet-name $vnetName | jq -r '.id')
echo "found subnet '$agwSubnetId'" >> deployment.log
## Application Gateway
echo "Creating app-gateway with '$location, $agwName, $rg, $agwSku, $agwSubnetName, $vnetName, $agwPIP'" >> deployment.log
az network application-gateway create \
--capacity 2 \
--frontend-port 80 \
--http-settings-cookie-based-affinity Disabled \
--http-settings-port 80 \
--http-settings-protocol Http \
--location $location \
--name $agwName \
--resource-group $rg \
--sku $agwSku \
--subnet $agwSubnetName \
--vnet-name $vnetName \
--public-ip-address $agwPIP \
--routing-rule-type basic \
--no-wait
## AKS Cluster
az aks create \
--resource-group $rg \
--name $aksClusterName \
--network-plugin azure \
--vnet-subnet-id $aksSubnetId \
--docker-bridge-address $aksDockerBridgeAddress \
--dns-service-ip $aksDnsServiceIP \
--service-cidr $aksServiceCIDR \
--generate-ssh-keys \
--kubernetes-version $PrimaryNodePoolk8sVersion \
--node-vm-size $PrimaryNodePoolNodeSize \
--node-count $PrimaryNodePoolNodeCount \
--enable-addons monitoring \
--location $location
--max-count $PrimaryNodePoolmaxcount \
--max-pods $PrimaryNodePoolmaxpods \
--min-count $PrimaryNodePoolmincount \
--enable-cluster-autoscaler
## Add Second NodePool
## There are just not enough IPs in a /25
#az aks nodepool add \
# --cluster-name $aksClusterName \
# --name $SecondaryNodePool \
# --resource-group $rg \
# --node-count $SecondaryNodePoolCount \
# --kubernetes-version $SecondaryNodePoolk8sVersion \
# --vnet-subnet-id $aksSubnetId
## Connect to AKS cluster
az aks get-credentials --resource-group $rg --name $aksClusterName
## kubesystem dashboard rbac acccess for dashboard
kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
#az aks browse --resource-group $rg --name $aksClusterName
## Create Managed Identity
aksidentity=$aksClusterName'identity'
agentpoolresourcegroup='MC_'$rg'_'$aksClusterName'_'$location
az identity create -g $agentpoolresourcegroup -n $aksidentity >> k8sidentity.json
az identity show -g $agentpoolresourcegroup -n $aksidentity
## Extract Parameters from managed Identity
clientid=$(jq -r ".clientId" k8sidentity.json)
identityresourceid=$(jq -r ".id" k8sidentity.json)
principalId=$(jq -r ".principalId" k8sidentity.json)
AppGatewayID=$( az network application-gateway show --resource-group $rg --name $agwName | jq -r ".id")
aksspnid=$(az aks show -g $rg -n $aksClusterName --query servicePrincipalProfile.clientId -o tsv)
echo "$clientid, $identityresourceid, $principalId, $AppGatewayID, $aksspnid" >> deployment.log
## AAD pod identity
wget https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml -O deployment-rbac.yaml
kubectl apply -f deployment-rbac.yaml
helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
helm repo update
## Managed Identity Role assignment (Application Gateway Contributor)
echo "Managed Identity Role assignment (Application Gateway Contributor) '$clientid, $AppGatewayID'" >> deployment.log
az role assignment create \
--role Contributor \
--assignee $clientid \
--scope $AppGatewayID
## Managed Identity Role assignment (Resource Group Reader)
echo "Managed Identity Role assignment (Resource Group Reader) '$clientid, $rgid'" >> deployment.log
az role assignment create \
--role Reader \
--assignee $clientid \
--scope $rgid
## create and update aadpodidentity YAML
echo 'create and update aadpodidentity YAML'
cat <<EOF > aadpodidentity.yaml
---
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
name: <aksidentity>
spec:
type: 0
ResourceID: <identityresourceid>
ClientID: <clientid>
EOF
sed -i "s|<clientid>|${clientid}|g" aadpodidentity.yaml
sed -i "s|<identityresourceid>|${identityresourceid}|g" aadpodidentity.yaml
sed -i "s|<aksidentity>|${aksidentity}|g" aadpodidentity.yaml
cat aadpodidentity.yaml >> deployment.log
kubectl apply -f aadpodidentity.yaml
## create and update azure identity binding YAML
selector="select_it"
azureidentitybinding="azure-identity-binding"
cat <<EOF > aadpodidentitybinding.yaml
---
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
name: <azureidentitybinding>
spec:
AzureIdentity: <aksidentity>
Selector: <selector>
EOF
sed -i "s|<azureidentitybinding>|${azureidentitybinding}|g" aadpodidentitybinding.yaml
sed -i "s|<aksidentity>|${aksidentity}|g" aadpodidentitybinding.yaml
sed -i "s|<selector>|${selector}|g" aadpodidentitybinding.yaml
cat aadpodidentitybinding.yaml
kubectl apply -f aadpodidentitybinding.yaml
## Add Helm app gateway ingress package and create helm file
## reference https://github.com/Azure/application-gateway-kubernetes-ingress/
wget https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/sample-helm-config.yaml -O helm-config.yaml
sed -i "s|<subscriptionId>|${subscriptionId}|g" helm-config.yaml
sed -i "s|<resourceGroupName>|${rg}|g" helm-config.yaml
sed -i "s|<applicationGatewayName>|${agwName}|g" helm-config.yaml
sed -i "s|<identityResourceId>|${identityresourceid}|g" helm-config.yaml
sed -i "s|<identityClientId>|${clientid}|g" helm-config.yaml
sed -i "s|enabled: false|enabled: true|g" helm-config.yaml
cat helm-config.yaml
helm install -f helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure --generate-name
## Im manually creating this but you could automate
##
# Get the resource-id of the public ip
#PUBLICIPID=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$IP')].[id]" --output tsv)
# Update public ip address with DNS name
#az network public-ip update --ids $PUBLICIPID --dns-name $DNSNAME
# Display the FQDN
#az network public-ip show --ids $PUBLICIPID --query "[dnsSettings.fqdn]" --output tsv
#https://docs.microsoft.com/en-us/azure/aks/ingress-tls
# az network dns record-set a add-record \
# --resource-group myResourceGroup \
# --zone-name MY_CUSTOM_DOMAIN \
# --record-set-name * \
# --ipv4-address MY_EXTERNAL_IP
###
## Cert manager v0.13.0
# add custome resource definitions
wget https://raw.githubusercontent.com/jetstack/cert-manager/v0.13.0/deploy/manifests/00-crds.yaml -O 00-crds.yaml
kubectl apply --validate=false -f 00-crds.yaml
# install cert manager
kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
cert-manager \
--namespace cert-manager \
--version v0.13.0 \
jetstack/cert-manager
kubectl get pods --namespace cert-manager
###### letsencrypt-staging
cat <<EOF > letsencrypt-staging.yaml
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: <registeredemail>
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource used to store the account's private key.
name: letsencrypt-staging
# Add a single challenge solver, HTTP01 using nginx
solvers:
- http01:
ingress:
class: azure/application-gateway
EOF
sed -i "s|<registeredemail>|${LetsEncryptEmail}|g" letsencrypt-staging.yaml
cat letsencrypt-staging.yaml
kubectl apply -f letsencrypt-staging.yaml
###### letsencrypt-production
cat <<EOF > letsencrypt-production.yaml
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: <registeredemail>
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource used to store the account's private key.
name: letsencrypt-prod
# Add a single challenge solver, HTTP01 using nginx
solvers:
- http01:
ingress:
class: azure/application-gateway
EOF
sed -i "s|<registeredemail>|${LetsEncryptEmail}|g" letsencrypt-production.yaml
kubectl apply -f letsencrypt-production.yaml
## enale Live Data for AKS cluster
cat <<EOF > Live-Data-AKS-cluster.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: containerHealth-log-reader
rules:
- apiGroups: ["", "metrics.k8s.io", "extensions", "apps"]
resources:
- "pods/log"
- "events"
- "nodes"
- "pods"
- "deployments"
- "replicasets"
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: containerHealth-read-logs-global
roleRef:
kind: ClusterRole
name: containerHealth-log-reader
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: User
name: clusterUser
apiGroup: rbac.authorization.k8s.io
EOF
kubectl apply -f Live-Data-AKS-cluster.yaml
### Environment Configured Here.
### Sample Applications and Ingress
## Guestbook (Sample Application)
wget https://raw.githubusercontent.com/kubernetes/examples/master/guestbook/all-in-one/guestbook-all-in-one.yaml -O guestbook-all-in-one.yaml
kubectl apply -f guestbook-all-in-one.yaml
## ASP net (Sample Application)
cat <<EOF > aspnet-app.yaml
---
apiVersion: v1
kind: Pod
metadata:
name: aspnetapp
labels:
app: aspnetapp
spec:
containers:
- image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
name: aspnetapp-image
ports:
- containerPort: 80
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: aspnetapp
spec:
selector:
app: aspnetapp
ports:
- protocol: TCP
port: 80
targetPort: 80
EOF
kubectl apply -f aspnet-app.yaml
## Ingress
## Guestbook Application App gateway Ingress (staging)
cat <<EOF > letsencrypt-guestbook-ing-stage.yaml
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: guestbook-letsencrypt
annotations:
cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: azure/application-gateway
spec:
tls:
- hosts:
- <domainname>
secretName: <SecretNameStage>
rules:
- host: <domainname>
http:
paths:
- backend:
serviceName: frontend
servicePort: 80
EOF
sed -i "s|<domainname>|${domainname}|g" letsencrypt-guestbook-ing-stage.yaml
sed -i "s|<SecretNameStage>|${SecretNameStage}|g" letsencrypt-guestbook-ing-stage.yaml
cat letsencrypt-guestbook-ing-stage.yaml
kubectl apply -f letsencrypt-guestbook-ing-stage.yaml
## Guestbook Application App gateway Ingress (Production)
cat <<EOF > letsencrypt-guestbook-ing-prod.yaml
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: guestbook-letsencrypt
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: azure/application-gateway
spec:
tls:
- hosts:
- <domainname>
secretName: <SecretName>
rules:
- host: <domainname>
http:
paths:
- backend:
serviceName: frontend
servicePort: 80
EOF
sed -i "s|<domainname>|${domainname}|g" letsencrypt-guestbook-ing-prod.yaml
sed -i "s|<SecretName>|${SecretName}|g" letsencrypt-guestbook-ing-prod.yaml
cat letsencrypt-guestbook-ing-prod.yaml
kubectl apply -f letsencrypt-guestbook-ing-prod.yaml
## URL path / Guestbook Application / ASP Net App gateway Ingress (Production)
cat <<EOF > letsencrypt-guestbook-ing-prod-paths.yaml
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: guestbook-letsencrypt
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: azure/application-gateway
spec:
tls:
- hosts:
- <domainname>
secretName: <SecretName>
rules:
- host: <domainname>
http:
paths:
- path: <URLPath>
backend:
serviceName: aspnetapp
servicePort: 80
- backend:
serviceName: frontend
servicePort: 80
EOF
sed -i "s|<URLPath>|${URLPath}|g" letsencrypt-guestbook-ing-prod-paths.yaml
sed -i "s|<domainname>|${domainname}|g" letsencrypt-guestbook-ing-prod-paths.yaml
sed -i "s|<SecretName>|${SecretName}|g" letsencrypt-guestbook-ing-prod-paths.yaml
cat letsencrypt-guestbook-ing-prod-paths
kubectl apply -f letsencrypt-guestbook-ing-prod-paths.yaml@lucidqdreams , How about checking you ASP.net app on your local machine. Could you please confirm whether you are able to host your app locally. (something like http://ASP.net:80)
lucidqdreams
commented
4 years ago This is why I am using public sample Microsoft containers so there is no question about the container validity itself. You can run this container in your own environment to test, also why I provided the entire script. The image ( - image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp") works fine, as I showed in the first comment.
Simply by changing the service that is mounted to the root I can see the container running the default webpage. All I change is the ingress template. The services working just fine, the mapping fails regardless of what service is backing it.
from this
- path: /app/*
backend:
serviceName: aspnetapp
servicePort: 80
- backend:
serviceName: frontend
servicePort: 80https://test.mydomain.com/ --> maps to --> frontend:80 (works) https://test.mydomain.com/app/ --> maps to --> Aspnetapp:80 (fails)
to this
- path: /app/*
backend:
serviceName: frontend
servicePort: 80
- backend:
serviceName: aspnetapp
servicePort: 80https://test.mydomain.com/ --> maps to --> Aspnetapp:80 (works) https://test.mydomain.com/app/ --> maps to --> frontend:80 (fails)
and the web page displays. It's only when I put the service behind the URL Path setting that is doesn't display. The ingress template should contain all the rules to update the application gateway to display the services.
The documentation leads me to believe its possible
lucidqdreams
commented
4 years ago 
I have even managed to get this working with external DNS, this image is showing that both containers are running just fine and responding through app gw at the same time. The problem is still that the URL path mapping doesnt redirect to the container.
This is great that this works however, the clients' requirement is to use the URL path on one URL.
SubhashVasarapu-MSFT
commented
4 years ago @lucidqdreams , From observations it looks like you have mentioned the path parameter as,
/path/
In-order to access contoso.com/path, then you need to define path as /path or /path.
And also, Health probe is failing because of back-end sending 404.You need to specify the path in health probe where you are trying to reach the web page(ASP.net).
We are closing this issue for now.If there are further questions regarding this matter, please reply and we will gladly continue the discussion.
lucidqdreams
commented
4 years ago Please reopen this case and escalate it to another AKS tech resource or I will have to open another issue with all this content. You have not answered the question.
I have tried to specify all sorts of paths /path/ and /path/* and none of them work. I even showed you the code and it looks exactly like what you have above
As I have already stated I should not be configuring the port probes, that is the job of the ingress controller as AKS manages the AGIC. My observations were that the port probes were failing and I have shown that with screenshots and even given you a script that will demonstrate this
Please reopen and escalate
trying to create a URL ingress path seems to fail to reach the service at the URL plus path /other/*
I am using the guestbook and aspnetapp applications. I publish them both and they work just fine. I would like to access one service at the root url and one service at a path off the root url.
I don't change anything else except the backend part of the rule.
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: guestbook-letsencrypt-prod annotations: cert-manager.io/cluster-issuer: letsencrypt-prod kubernetes.io/ingress.class: azure/application-gateway spec: tls:
to this
When I view the URL, I get the service either the guestbook or the asp net app, whichever one is configured. Which tells me the services are running just fine. However, both configurations give me the same result when I try to go to path-based URL... URL/app/ I get "502 Bad Gateway Microsoft-Azure-Application-Gateway/v2"
it looks like the yaml does configure the application gateway correctly. Some screenshots to help explain
back end pools
services
aspnet
guestbook
path-based rule
this all looks correct and it was configured through k8s yaml templates, and both services respond on the base URL but not the path-based URL.
The instructions on this page tell me this should work.
my results as follows
then I change
I see the change in the gateway
refresh the browsers
any thoughts what's going on here? has someone else tested this works?
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.