sauryadas
commented
3 years ago https://github.com/Azure/azure-arc-kubernetes-preview/issues
is the right place to file arc/extensions issues..
@shashankbarsin Will you please take a look ?
Closed jamie2605 closed 2 years ago
sauryadas
commented
3 years ago https://github.com/Azure/azure-arc-kubernetes-preview/issues
is the right place to file arc/extensions issues..
@shashankbarsin Will you please take a look ?
shashankbarsin
commented
3 years ago @jamie2605 what commands did you run to get the following errors?
2021-01-22T14:36:03.7461476Z Creating...........
2021-01-22T14:36:04.0023562Z FailedProvisioning operation finished, operation "Failed"
2021-01-22T14:36:04.4579326Z Traceback (most recent call last):
2021-01-22T14:36:04.4580446Z File "./attach.py", line 34, in <module>Full repro steps would be helpful.
About clusterconnect-agent, it's a feature independent from the cluster extensions feature. So crashes on clusterconnect-agent pod shouldn't impact Azure ML extension and its jobs. Having said that we do want to get to the root cause of why that pod is crashing. Please run the following command and share the logs:
kubectl logs clusterconnect-agent-556cc7d5fd-m5p59 -c clusterconnect-agent -p
jamie2605
commented
3 years ago Thanks @shashankbarsin for the reply.
Those logs are shared above in the log.txt spoiler, but will post them again at the end of this post.
So what I did to get to that state:
Setup kind and have it use our company proxy:
export KIND_EXPERIMENTAL_DOCKER_NETWORK=bridge
export HTTP_PROXY=http://172.17.0.1:3128 HTTPS_PROXY=$HTTP_PROXY http_proxy=$HTTP_PROXY https_proxy=$HTTP_PROXY
kind create cluster
kubectl config set-context kind
Install helm as described on install helm https://helm.sh/docs/intro/install/
Setup az
git clone https://github.com/Azure/azure-arc-kubernetes-preview.git
az extension add --source ./extensions/connectedk8s-0.3.2-py2.py3-none-any.whl --yes
az extension add --source ./extensions/k8s_extension-0.1.0-py2.py3-none-any.whl --yes
az provider register --namespace Microsoft.KubernetesConfiguration (IT did that)
az feature register --namespace Microsoft.KubernetesConfiguration --name extensions (IT did that)
Connect the cluster
az connectedk8s connect -n pocworkstation-arc-dev -g atpdeep-arc-devThe command above already gives me a lot of repetitions of
W0122 14:08:43.594362 716508 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinitionand at the end a Error: timed out waiting for the condition, but since the cluster showed up in the portal fine and the pods reported running (except for that one clusterconnect-agent pod), I thought maybe the AML part would work anyways.
So for the AzureML specific parts of installation and attachment
pip install --upgrade azureml-sdk
pip install azureml-contrib-pipeline-steps # Without this, the k8s preview failed to install
pip install --disable-pip-version-check --extra-index-url https://azuremlsdktestpypi.azureedge.net/azureml-contrib-k8s-preview/D58E86006C65 azureml-contrib-k8s
# Attaching (attach.py script from above)
target_name = "pocarc"
ws = Workspace(os.environ["subscription_id"],
os.environ["resource_group"],
os.environ["workspace_name"])
k8s_config = {
}
attach_config = ArcKubernetesCompute.attach_configuration(
cluster_name="pocworkstation-arc-dev",
resource_group="atpdeep-arc-dev",
resource_id="/subscriptions/***/resourceGroups/atpdeep-arc-dev/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev",
aml_k8s_config=k8s_config
)
arc_target = ArcKubernetesCompute.attach(ws, target_name, attach_config)
arc_target.wait_for_completion(show_output=True)
And then again the output of kubectl logs -p clusterconnect-agent-556cc7d5fd-m5p59 clusterconnect-agent --namespace azure-arc
Running ConnectProxy Agent
MICROSOFT SOFTWARE LICENSE TERMS
MICROSOFT Azure Arc for Kubernetes
__________________________________
This software is licensed to you as part of your or your company's subscription license for Microsoft Azure Services. You may only use the software with Microsoft Azure Services and subject to the terms and conditions of the agreement under which you obtained Microsoft Azure Services. If you do not have an active subscription license for Microsoft Azure Services, you may not use the software. Microsoft Azure Legal Information: https://azure.microsoft.com/en-us/support/legal/
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"RelayConfigFetchStarting ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 14:59:54"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ListProxyConnectionDetailStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 14:59:55"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 14:59:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 14:59:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 14:59:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:00:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:01:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:02:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:03:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:09"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:19"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:29"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:39"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.<GetToken>b__4_0() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 50\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider) }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:49"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:59"}
{"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 61\n at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)\n at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider)\n at Polly.Retry.RetryPolicy.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken)\n at Polly.Policy.Implementation(Action`2 action, Context context, CancellationToken cancellationToken)\n at Polly.Policy.Execute(Action`2 action, Context context, CancellationToken cancellationToken)\n at Polly.Policy.Execute(Action action)\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.GetToken() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 52\n at ServiceCaller.Caller.NotificationServiceCaller.FetchConnectionString() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/ServiceCaller/NotificationServiceCaller.cs:line 104\n at ServiceCaller.Caller.NotificationServiceCaller.FetchData() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/ServiceCaller/NotificationServiceCaller.cs:line 48\n at ConnectedProxyAgent.Program.FetchAndDumpConfig(Object data) in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/ConnectedProxyAgent/Program.cs:line 178\n at ConnectedProxyAgent.Program.Main(String[] args) in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/ConnectedProxyAgent/Program.cs:line 78 }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 15:04:59"}
Unhandled exception. System.InvalidOperationException: Status not populated
at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 61
at Polly.Policy.<>c__DisplayClass108_0.<Execute>b__0(Context ctx, CancellationToken ct)
at Polly.Policy.<>c__DisplayClass138_0.<Implementation>b__0(Context ctx, CancellationToken token)
at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider)
at Polly.Retry.RetryPolicy.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken)
at Polly.Policy.Implementation(Action`2 action, Context context, CancellationToken cancellationToken)
at Polly.Policy.Execute(Action`2 action, Context context, CancellationToken cancellationToken)
at Polly.Policy.Execute(Action action)
at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.GetToken() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ManagedIdentityTokenProvider.cs:line 52
at ServiceCaller.Caller.NotificationServiceCaller.FetchConnectionString() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/ServiceCaller/NotificationServiceCaller.cs:line 104
at ServiceCaller.Caller.NotificationServiceCaller.FetchData() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/ServiceCaller/NotificationServiceCaller.cs:line 48
at ConnectedProxyAgent.Program.FetchAndDumpConfig(Object data) in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/ConnectedProxyAgent/Program.cs:line 178
at ConnectedProxyAgent.Program.Main(String[] args) in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/ConnectedProxyAgent/Program.cs:line 78
./entrypoint.sh: line 8: 8 Aborted (core dumped) dotnet ConnectedProxyAgent.dllBtw: should I open a new thread over in https://github.com/Azure/azure-arc-kubernetes-preview/issues so that you can close this one instead or do you want to keep the thread here?
shashankbarsin
commented
3 years ago @jamie2605 this looks like an issue of missing proxy values from agent pods. Proxy values have to be passed in explicitly as part of az connectedk8s connect command. More info in our docs here. This is needed despite the HTTP_PROXY environment variable set on the laptop because agent pods on the cluster don't utilize host network settings for their egress communication.
To address this issue, I'd recommend following steps:
az connectedk8s connect command with proxy parameters specifiedaz connectedk8s connect -n pocworkstation-arc-dev -g atpdeep-arc-dev --proxy-https http://172.17.0.1:3128 --proxy-http http://172.17.0.1:3128 --proxy-skip-range <podCIDRRange>,kubernetes.default.svcThanks @shashankbarsin, that got me a bit further, now kubectl get pods --namespace azure-arc gives me
NAME READY STATUS RESTARTS AGE
cluster-metadata-operator-57b8b479-6nlpj 2/2 Running 0 13m
clusterconnect-agent-5c575d6b85-856rb 2/2 Running 0 13m
clusteridentityoperator-67998b5448-qfcx6 2/2 Running 0 13m
config-agent-5667944fdd-nxr8v 2/2 Running 0 13m
controller-manager-59c67c4b7b-r28qn 2/2 Running 0 13m
extension-manager-5d9799d7dd-f82fh 2/2 Running 0 13m
flux-logs-agent-6c588fdd9c-n2859 1/1 Running 0 13m
metrics-agent-88fd84c57-mc8bc 2/2 Running 0 13m
resource-sync-agent-6454b4c86d-l6rxs 2/2 Running 0 13mand az connectedk8s show --name atpdeep-pocworkstation-arc-dev --resource-group atpdeep-arc-dev
gives me
{
"aadProfile": {
"clientAppId": "",
"serverAppId": "",
"tenantId": ""
},
"agentPublicKeyCertificate": "***",
"agentVersion": "0.2.43",
"connectivityStatus": "Connected",
"distribution": "kind",
"id": "/subscriptions/***/resourceGroups/atpdeep-arc-dev/providers/Microsoft.Kubernetes/connectedClusters/atpdeep-pocworkstation-arc-dev",
"identity": {
"principalId": "***",
"tenantId": "***",
"type": "SystemAssigned"
},
"infrastructure": "generic",
"kubernetesVersion": "1.19.1",
"lastConnectivityTime": "2021-01-26T06:42:33.225000+00:00",
"location": "westeurope",
"managedIdentityCertificateExpirationTime": "2021-04-26T06:32:00+00:00",
"name": "atpdeep-pocworkstation-arc-dev",
"offering": null,
"provisioningState": "Succeeded",
"resourceGroup": "atpdeep-arc-dev",
"tags": {
"createdDate": "2021-01-26T06:37:13.8876632Z",
"stage": "",
"workstream": ""
},
"totalCoreCount": 8,
"totalNodeCount": 1,
"type": "Microsoft.Kubernetes/connectedClusters"
}However, the attaching to AzureML still fails with the same problem
target_name = "pocarc"
ws = Workspace(os.environ["subscription_id"],
os.environ["resource_group"],
os.environ["workspace_name"])
k8s_config = {
}
attach_config = ArcKubernetesCompute.attach_configuration(
cluster_name="pocworkstation-arc-dev",
resource_group="atpdeep-arc-dev",
resource_id=f"/subscriptions/{os.environ['subscription_id']}/resourceGroups/atpdeep-arc-dev/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev",
aml_k8s_config=k8s_config
)
arc_target = ArcKubernetesCompute.attach(ws, target_name, attach_config)
arc_target.wait_for_completion(show_output=True)
gives
Creating...........
FailedProvisioning operation finished, operation "Failed"
Traceback (most recent call last):
File "./attach.py", line 34, in <module>
arc_target.wait_for_completion(show_output=True)
File "/agent/_work/1/s/infrastructure/arcpoc/venv/lib/python3.6/site-packages/azureml/core/compute/compute.py", line 575, in wait_for_completion
raise e
File "/agent/_work/1/s/infrastructure/arcpoc/venv/lib/python3.6/site-packages/azureml/core/compute/compute.py", line 568, in wait_for_completion
'{}'.format(self.provisioning_state, error_response))
azureml.exceptions._azureml_exception.ComputeTargetException: ComputeTargetException:
Message: Compute object provisioning polling reached non-successful terminal state, current provisioning state: Failed
Provisioning operation error:
{'code': 'InternalServerError', 'message': 'An internal server error occurred. If the problem persists, contact support.'}
InnerException None
ErrorResponse
{
"error": {
"message": "Compute object provisioning polling reached non-successful terminal state, current provisioning state: Failed\nProvisioning operation error:\n{'code': 'InternalServerError', 'message': 'An internal server error occurred. If the problem persists, contact support.'}"
}
}
So I guess now we are at the correct location as this now really seems to be an AML Kubernetes problem @sauryadas ? Or is there any other way to verify that the kubernetes cluster is correctly attached than the output of az connectedk8s show?
sauryadas
commented
3 years ago @adressel we require any egress traffic right ?
From the above it looks like they have an outbound proxy set up following these instructions https://docs.microsoft.com/en-in/azure/azure-arc/kubernetes/connect-cluster#connect-using-an-outbound-proxy-server
sauryadas
commented
3 years ago @jamie2605 can you please try without the proxy while we investigate the issue ?
jamie2605
commented
3 years ago @sauryadas: Unfortunately, that is not possible, I cannot get transparent Internet access for that workstation, as it is located in an office building and I cannot bypass the company network in any way....
sauryadas
commented
3 years ago looks like we have a bug that we fixed and need to roll out to West Europe by the end of the week.. we have it working in one of our canary regions but you wont have access to it ..
jamie2605
commented
3 years ago I tried again today, but unfortunately, this issue still persists. Do I have to update the extension or recreate the Azure Arc cluster for this to work? Just reattaching definitely does not work...
Adarsh-Ramanathan
commented
3 years ago @jamie2605, can you navigate to the resource providers tab on the left blade in your subscription, and confirm that the Microsoft.Relay provider is registered?
jamie2605
commented
3 years ago I put in a request to our IT to enable that feature, hopefully we will get it enabled quickly, will report then.
jamie2605
commented
3 years ago Ok, so a little progress, but still not running out of proxy problems yet.
I was now able to attach the cluster, and azureml seems to deploy some pods:
kubectl get pods --namespace azureml
NAME READY STATUS RESTARTS AGE
aml-mpi-operator-8467b75889-krhv7 1/1 Running 0 175m
aml-operator-849464dd65-b8bh5 1/1 Running 53 175m
aml-pytorch-operator-6f875cb6db-dqx6f 1/1 Running 0 175m
aml-tf-job-operator-99f7bcfdd-4qtw6 1/1 Running 0 175m
azureml-connector-admission-799865fc6-vd4bd 0/1 ContainerCreating 0 175m
azureml-connector-admission-init-g9wn4 0/1 Error 0 175m
azureml-connector-admission-init-jbmcl 0/1 Error 0 172m
azureml-connector-admission-init-rcmc4 0/1 Error 0 172m
azureml-connector-admission-init-s88h8 0/1 Error 0 174m
azureml-connector-controllers-f88b75d6f-kqxgh 1/1 Running 0 175m
azureml-connector-scheduler-578df4c749-sndhv 1/1 Running 0 175m
compute-exporter-7f7f7b87c8-4mkbm 1/1 Running 0 175m
job-exporter-hjm5r 1/1 Running 0 175m
metric-reporter-7cfc9dd98c-2jcfq 1/1 Running 0 175m
prometheus-deployment-9685f74d9-r9vhf 1/1 Running 0 175m
relay-server-cd96494cc-9x5l9 0/1 CrashLoopBackOff 32 175m
rest-server-7f9cb5566-9r5mh 1/1 Running 0 175mSince the relay server seems to be affected the worst, I took a look at it:
kubectl logs -p relay-server-cd96494cc-9x5l9 relay-server --namespace azureml
2021-02-05 10:25:07.0934|ERROR|AzureRelayListener|Start|Microsoft.Azure.Relay.RelayException: Unable to connect to the remote server. TrackingId:c0efae36-7a24-4168-be11-b0b7f129f3ae, Address:sb://atpdeeppocworkstationarcdev5137536128327538518.servicebus.windows.net/aml-k8s-jobs-connection, Timestamp:02/05/2021 10:25:07
---> System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Azure.Relay.WebSockets.NetCore21.WebSocketHandle.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)
at Microsoft.Azure.Relay.WebSockets.NetCore21.WebSocketHandle.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)
at Microsoft.Azure.Relay.WebSockets.NetCore21.ClientWebSocket.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken)
at Microsoft.Azure.Relay.HybridConnectionListener.ControlConnection.ConnectAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.Azure.Relay.HybridConnectionListener.ControlConnection.ConnectAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Relay.HybridConnectionListener.ControlConnection.CloseOrAbortWebSocketAsync(Task`1 connectTask, Boolean abort, WebSocketCloseStatus closeStatus, String statusDescription, CancellationToken cancellationToken)
at Microsoft.Azure.Relay.HybridConnectionListener.ControlConnection.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Relay.HybridConnectionListener.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Relay.HybridConnectionListener.OpenAsync(TimeSpan timeout)
at Microsoft.Azure.Relay.AspNetCore.AzureRelayListener.Start()Also, there seems to be more problems in other services:
kubectl get events --namespace=azureml
LAST SEEN TYPE REASON OBJECT MESSAGE
16m Normal Pulling pod/aml-operator-849464dd65-b8bh5 Pulling image "amlk8s.azurecr.io/public/azureml/amlk8s/dockerimage/westeurope/preview/amlop:1.0.0"
106s Warning Unhealthy pod/aml-operator-849464dd65-b8bh5 Liveness probe failed: Get "http://10.244.0.17:8080/healthz": dial tcp 10.244.0.17:8080: connect: connection refused
6m41s Warning BackOff pod/aml-operator-849464dd65-b8bh5 Back-off restarting failed container
109s Warning FailedMount pod/azureml-connector-admission-799865fc6-vd4bd MountVolume.SetUp failed for volume "admission-certs" : secret "aml-volcano-admission-secret" not found
10m Warning FailedMount pod/azureml-connector-admission-799865fc6-vd4bd Unable to attach or mount volumes: unmounted volumes=[admission-certs], unattached volumes=[admission-certs azureml-connector-admission-token-tcrsd]: timed out waiting for the condition
6m5s Warning FailedMount pod/azureml-connector-admission-799865fc6-vd4bd Unable to attach or mount volumes: unmounted volumes=[admission-certs], unattached volumes=[azureml-connector-admission-token-tcrsd admission-certs]: timed out waiting for the condition
46m Normal Pulling pod/relay-server-cd96494cc-9x5l9 Pulling image "amlk8s.azurecr.io/public/azureml/amlk8s/dockerimage/westeurope/preview/aml-relay-server:1.0.0"
102s Warning BackOff pod/relay-server-cd96494cc-9x5l9 Back-off restarting failed containerHow can I tell your pods to use the proxy?
sauryadas
commented
3 years ago Thanks .. Adarsh is looking into how we can specify proxy in the relay server ..
jamie2605
commented
3 years ago Hm, is there still work going into this @sauryadas ?
I saw https://github.com/Azure/AML-Kubernetes/issues/48, is that related? For the rest, this issue seems to have stalled?
sauryadas
commented
3 years ago @jamie2605 we have been working on a fix. Will keep you posted on the rollout plan
sauryadas
commented
3 years ago @jamie2605 still rolling out and validating .. will keep you posted ...
sauryadas
commented
3 years ago @jamie2605 Still working on this. Looking at how blobfuse can work through a proxy. Apologies, but this probably is going to take longer to fix than we thought. will keep you updated.
jamie2605
commented
3 years ago Ok, that is unfortunate but thanks for the info. We would be happy to have this feature for testing without mounting datastores for now, but I guess you need it to mount the workspaceblobstore?
sauryadas
commented
3 years ago Thanks @jamie2605 . @adrosa is working on implementing a workaround for another issue discovered. Will keep you posted
sauryadas
commented
3 years ago @criher What does Daimlers certificate look like ? Is it signed by a recognized CA? Or is it self-signed/signed by a CA that's unlikely to be in a default trust list?
criher
commented
3 years ago @sauryadas The certificate landscape on our side might look a little more complicated than I initially thought. I have the opportunity to discuss it with our experts tomorrow. Is it a specific certificate that you need the answer for or is it relevant for all certificates involved in proxy communication?
adrosa
commented
3 years ago Just the certificates involved in proxy communication.
From: @.> Sent: Wednesday, April 14, 2021 7:08 AM To: @.> Cc: Ada Rosa @.>; @.> Subject: Re: [Azure/AML-Kubernetes] Cannot attach Azure Arc Kubernetes cluster (#41)
@sauryadashttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsauryadas&data=04%7C01%7Cada.rosa%40microsoft.com%7C9dc29c20d9b54080fff408d8ff4ec27a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637540061034214483%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vwW%2BYCeIHE3pkZ0%2FbQl%2BZaOaMS1Sxww3Z3E32I2baJs%3D&reserved=0 The certificate landscape on our side might look a little more complicated than I initially thought. I have the opportunity to discuss it with our experts tomorrow. Is it a specific certificate that you need the answer for or is it relevant for all certificates involved in proxy communication?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2FAML-Kubernetes%2Fissues%2F41%23issuecomment-819547035&data=04%7C01%7Cada.rosa%40microsoft.com%7C9dc29c20d9b54080fff408d8ff4ec27a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637540061034224475%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=zGXmpjcCbYUQpZ1tayv5XGXUbZZeXq1SxXK9JBbG8gw%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAOTUSGFM5SGWPDL4PCX24UTTIWOVJANCNFSM4WOULKMQ&data=04%7C01%7Cada.rosa%40microsoft.com%7C9dc29c20d9b54080fff408d8ff4ec27a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637540061034234469%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5w008Y3%2Fy8XOj5IQ%2FVtPM3NZn2uQPRkg4XS3jJiG4JE%3D&reserved=0.
criher
commented
3 years ago Update: no critical certificates should be involved. If certificates are involved in proxy communication, they should be signed by a "default" CA.
sauryadas
commented
3 years ago Thanks for the certificate info. We have made our fixes and waiting now for the blobfuse team to fix their issue
Adarsh-Ramanathan
commented
3 years ago @criher, the blobfuse team has a fix in the pipeline; however, they're not confident it'll make it into their next release. I can get us a build that we can sideload into the worker nodes on your cluster to unblock you for the time being; is that going to be a workable solution, or do you need packages to be signed and published?
sauryadas
commented
3 years ago @criher thoughts?
criher
commented
3 years ago @Adarsh-Ramanathan I guess it would be a viable solution that we'd appreciate to begin with. That way, we could verify that we don't have any other blockers. In the long run, published packages are definitely preferred/required.
Adarsh-Ramanathan
commented
3 years ago @criher, could you run through these steps, please? (feel free to reach out to me if you need assistance)
az extension add --source k8s_extension-0.1PP.15-py3-none-any.whl --yesaz connectedk8s delete --name <arc_cluster_name> --resource-group <resource_group> --subscription <subscription> Wait for all arc pods and the azure-arc namespace to be deleted.az connectedk8s connect --name <arc_cluster_name> --resource-group <resource_group> --subscription <subscription> --location eastus2euap --proxy-https http://<proxy_host>:<proxy_port> --proxy-http http://<proxy_host>:<proxy_port> --proxy-skip-range <proxy_host>,localhost,<pod_network_CIDR>,10.0.0.0/8,kubernetes.default.svc,.svc.cluster.local --proxy-cert /path/to/proxy-certificate.crt It is important that the cluster is in eastus2euap, that you add ".svc.cluster.local" to the proxy skip argument, and that you specify the path to the proxy's cert file.az k8s-extension create --subscription <subscription> --resource-group <resource_group> -c <arc_cluster_name> --cluster-type connectedClusters --extension-type Microsoft.AzureML.Kubernetes -n azureml-connector --release-train preview --config enableTraining=True --version 1.0.60sauryadas
commented
3 years ago @criher .. we will roll out to West Europe today and ping you
sauryadas
commented
3 years ago @criher can you try in west europe now?
sauryadas
commented
3 years ago @criher any updates?
criher
commented
3 years ago Still having issues. However, slightly different behavior. Before, aml runs got directly a 404 error when deployed to the cluster. Now they are just stuck in "Queued" forever.
I am about to cross check that I didn't miss any of the details in the description.
Details on the current state:
kubectl get pods -n azureml
NAME READY STATUS RESTARTS AGE
aml-mpi-operator-5484c86b65-rw7xk 1/1 Running 0 65m
aml-operator-754b5995c8-cvpbc 0/1 CrashLoopBackOff 21 65m
aml-pytorch-operator-79bf8d8955-ch648 1/1 Running 0 65m
aml-tf-job-operator-99f7bcfdd-kc9xg 1/1 Running 0 65m
azureml-connector-admission-799865fc6-22n9l 0/1 ContainerCreating 0 65m
azureml-connector-admission-init-2wgts 0/1 Error 0 65m
azureml-connector-admission-init-gmxpr 0/1 Error 0 63m
azureml-connector-admission-init-js7zw 0/1 Error 0 62m
azureml-connector-admission-init-slxt2 0/1 Error 0 62m
azureml-connector-controllers-f88b75d6f-2rs68 1/1 Running 0 65m
azureml-connector-kube-state-metrics-599f98d995-p8ws9 1/1 Running 0 65m
azureml-connector-scheduler-578df4c749-856fn 1/1 Running 0 65m
job-exporter-shg8z 1/1 Running 0 65m
prom-operator-768f56f56-4px5m 2/2 Running 0 65m
prometheus-prom-prometheus-0 3/3 Running 1 64m
relay-server-75464fbc4-n4xx4 1/1 Running 0 65m
rest-server-6fcbfdf7f-qxf7p 1/1 Running 0 65mThe connectors, don't show any logs, the operator however does:
kubectl logs -p aml-operator-754b5995c8-cvpbc --namespace azureml
[2021-05-06 14:49:54,412] utils.common [INFO ] Couldn't get valid response from IMDS_ENDPOINT, setting azcloud region to AzurePublicCloud.
[2021-05-06 14:49:54,412] utils.common [INFO ] Azure Cloud region: AzurePublicCloud.
[2021-05-06 14:49:54,423] kopf.activities.star [INFO ] Activity 'configure' succeeded.
[2021-05-06 14:49:54,425] kopf.reactor.activit [INFO ] Initial authentication has been initiated.
[2021-05-06 14:49:54,426] utils.config [ERROR ] Update metrics-mapping-data failed. metrics-mapping-data:{}
[2021-05-06 14:49:54,429] kopf.activities.auth [INFO ] Activity 'login_via_pykube' succeeded.
Traceback (most recent call last):
File "/src/utils/config.py", line 77, in update_config_map
ConfigMapHelper.data[fname] = json.loads(content)
File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[2021-05-06 14:49:54,430] kopf.activities.auth [INFO ] Activity 'login_via_client' succeeded.
[2021-05-06 14:49:54,431] kopf.reactor.activit [INFO ] Initial authentication has finished.
[2021-05-06 14:49:54,448] kopf.engines.peering [WARNING ] Default peering object not found, falling back to the standalone mode.Also the events indicate that there might still be some connection issues:
kubectl get events --namespace=azureml
LAST SEEN TYPE REASON OBJECT MESSAGE
6m58s Normal Pulling pod/aml-operator-754b5995c8-cvpbc Pulling image "amlk8s.azurecr.io/public/azureml/amlk8s/docker/preview/amlop:1.0.60"
58m Normal Created pod/aml-operator-754b5995c8-cvpbc Created container the-only-one
58m Normal Started pod/aml-operator-754b5995c8-cvpbc Started container the-only-one
42m Warning Unhealthy pod/aml-operator-754b5995c8-cvpbc Liveness probe failed: Get "http://10.244.0.37:8080/healthz": dial tcp 10.244.0.37:8080: connect: connection refused
59m Normal Killing pod/aml-operator-754b5995c8-cvpbc Container the-only-one failed liveness probe, will be restarted
59m Normal Pulled pod/aml-operator-754b5995c8-cvpbc Successfully pulled image "amlk8s.azurecr.io/public/azureml/amlk8s/docker/preview/amlop:1.0.60" in 320.916781ms
58m Normal Pulled pod/aml-operator-754b5995c8-cvpbc Successfully pulled image "amlk8s.azurecr.io/public/azureml/amlk8s/docker/preview/amlop:1.0.60" in 214.724456ms
2m3s Warning BackOff pod/aml-operator-754b5995c8-cvpbc Back-off restarting failed container
63s Warning FailedMount pod/azureml-connector-admission-799865fc6-22n9l MountVolume.SetUp failed for volume "admission-certs" : secret "aml-volcano-admission-secret" not found
16m Warning FailedMount pod/azureml-connector-admission-799865fc6-22n9l Unable to attach or mount volumes: unmounted volumes=[admission-certs], unattached volumes=[admission-certs azureml-connector-admission-token-88w54]: timed out waiting for the condition
27m Warning FailedMount pod/azureml-connector-admission-799865fc6-22n9l Unable to attach or mount volumes: unmounted volumes=[admission-certs], unattached volumes=[azureml-connector-admission-token-88w54 admission-certs]: timed out waiting for the condition
60m Normal Pulled pod/azureml-connector-admission-init-gmxpr Successfully pulled image "mcr.microsoft.com/azureml/cmk8s/oss-component:private-vc-webhook-manager" in 35.784617215s
60m Normal Created pod/azureml-connector-admission-init-gmxpr Created container main
59m Normal Started pod/azureml-connector-admission-init-gmxpr Started container main
59m Normal Scheduled pod/azureml-connector-admission-init-js7zw Successfully assigned azureml/azureml-connector-admission-init-js7zw to kind-control-plane
59m Normal Pulling pod/azureml-connector-admission-init-js7zw Pulling image "mcr.microsoft.com/azureml/cmk8s/oss-component:private-vc-webhook-manager"
59m Normal Pulled pod/azureml-connector-admission-init-js7zw Successfully pulled image "mcr.microsoft.com/azureml/cmk8s/oss-component:private-vc-webhook-manager" in 254.028613ms
59m Normal Created pod/azureml-connector-admission-init-js7zw Created container main
59m Normal Started pod/azureml-connector-admission-init-js7zw Started container main
59m Normal Scheduled pod/azureml-connector-admission-init-slxt2 Successfully assigned azureml/azureml-connector-admission-init-slxt2 to kind-control-plane
59m Normal Pulling pod/azureml-connector-admission-init-slxt2 Pulling image "mcr.microsoft.com/azureml/cmk8s/oss-component:private-vc-webhook-manager"
59m Normal Pulled pod/azureml-connector-admission-init-slxt2 Successfully pulled image "mcr.microsoft.com/azureml/cmk8s/oss-component:private-vc-webhook-manager" in 147.538372ms
59m Normal Created pod/azureml-connector-admission-init-slxt2 Created container main
59m Normal Started pod/azureml-connector-admission-init-slxt2 Started container main
59m Normal SuccessfulCreate job/azureml-connector-admission-init Created pod: azureml-connector-admission-init-slxt2
59m Normal SuccessfulCreate job/azureml-connector-admission-init Created pod: azureml-connector-admission-init-js7zw
58m Warning BackoffLimitExceeded job/azureml-connector-admission-init Job has reached the specified backoff limit
60m Normal Pulled pod/prometheus-prom-prometheus-0 Successfully pulled image "quay.io/prometheus/prometheus:v2.21.0" in 1m36.885955339s
59m Normal Created pod/prometheus-prom-prometheus-0 Created container prometheus
59m Normal Started pod/prometheus-prom-prometheus-0 Started container prometheus
59m Normal Pulling pod/prometheus-prom-prometheus-0 Pulling image "quay.io/prometheus-operator/prometheus-config-reloader:v0.42.1"
59m Normal Pulled pod/prometheus-prom-prometheus-0 Successfully pulled image "quay.io/prometheus-operator/prometheus-config-reloader:v0.42.1" in 2.818209709s
59m Normal Created pod/prometheus-prom-prometheus-0 Created container prometheus-config-reloader
59m Normal Started pod/prometheus-prom-prometheus-0 Started container prometheus-config-reloader
59m Normal Pulling pod/prometheus-prom-prometheus-0 Pulling image "docker.io/jimmidyson/configmap-reload:v0.4.0"
59m Normal Pulled pod/prometheus-prom-prometheus-0 Successfully pulled image "docker.io/jimmidyson/configmap-reload:v0.4.0" in 3.617080249s
59m Normal Created pod/prometheus-prom-prometheus-0 Created container rules-configmap-reloader
59m Normal Started pod/prometheus-prom-prometheus-0 Started container rules-configmap-reloader
59m Normal Pulled pod/prometheus-prom-prometheus-0 Container image "quay.io/prometheus/prometheus:v2.21.0" already present on machine@criher does look like you got a e2e test job running?
criher
commented
3 years ago Yes, thanks to the kind help of @Adarsh-Ramanathan, I managed to run a basic job on a local kind cluster. So communication through the proxy works. Next step will be to attach our bigger cluster. I'll keep you posted, if we run into new issues.
Bozhong68
commented
2 years ago closing staled issue
We want to use the AML Kubernetes Preview using an on-premise cluster (for the first proof-of-concept this is just a kind cluster) connected to Azure by means of Azure Arc.
If I try to connect the Azure Arc cluster to an AML workspace I get the error message
However, I am not sure whether the problem actually lies with AML-Kubernetes or the Azure Arc Kubernetes cluster. I checked
kubectl get pods --namespace azure-arcand the many restarts of the clusterconnect agent were worrying, so I checked
kubectl get events --namespace=azure-arcThe (redacted) log of the cluster connect agent
kubectl logs -p clusterconnect-agent-556cc7d5fd-m5p59 clusterconnect-agent --namespace azure-arc >> log.txtis in the spoiler belowlog.txt
``` Running ConnectProxy Agent MICROSOFT SOFTWARE LICENSE TERMS MICROSOFT Azure Arc for Kubernetes __________________________________ This software is licensed to you as part of your or your company's subscription license for Microsoft Azure Services. You may only use the software with Microsoft Azure Services and subject to the terms and conditions of the agreement under which you obtained Microsoft Azure Services. If you do not have an active subscription license for Microsoft Azure Services, you may not use the software. Microsoft Azure Legal Information: https://azure.microsoft.com/en-us/support/legal/ {"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"RelayConfigFetchStarting ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 14:59:54"} {"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ListProxyConnectionDetailStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 14:59:55"} {"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Information","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchStarted ","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 14:59:59"} {"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Error","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ClusterIdentityCRDTokenFetchFailed { message = Retrying token fetch in 00:00:10 seconds }","AgentVersion":"0.2.28","AgentTimestamp":"01/22/2021 14:59:59"} {"ArmId":"subscriptions/***/resourceGroups/***/providers/Microsoft.Kubernetes/connectedClusters/pocworkstation-arc-dev","Location":"westeurope","AgentName":"ConnectProxyAgent","Role":"ClusterConfigAgent","LogLevel":"Warning","Environment":"prod","LogType":"ConnectAgentTrace","CorrelationId":"","Message":"ThrowingException { source = , exception = System.InvalidOperationException: Status not populated\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ClusterIdentityCRDClient.GetTokenFromStatus() in /home/vsts/work/1/s/gopath/src/github.com/Azure/ClusterConfigurationAgent/ConnectedProxyAgent/DataPlaneInteraction/msi/ClusterIdentityCRDClient.cs:line 76\n at Microsoft.Azure.Arc.Kubernetes.Agent.TokenProvider.ManagedIdentityTokenProvider.So I am not sure who to contact for this: The Azure Arc Team, people maintaining connectedk8s or k8s_extensions or you guys.... Since it seems that @wmpauli already talked to you about this, I am going to start here and would appreciate if you could forward me to the correct people if the above doesn't make sense to you.
@wmpauli: Sorry for the delay in reporting this, it took me a while to get to last year's state again on this topic and some other things got in between