irperez
commented
2 years ago A few comments with this one as I worked with @ddzubow on it.
- We wiped out our .kube folder and reconnected.
- We wiped out the deployment in kubernetes and redeployed.
- Istio is configured with mTLS. Not using any other features.
- Using nginx-ingress as our ingress controller
- Using cert-manager for certificate management
- Here is the deployment of the pod itself.
apiVersion: apps/v1
kind: Deployment
metadata:
name: connectapi
namespace: ddzubow
labels:
app: connectapi
app.kubernetes.io/component: api
app.kubernetes.io/managed-by: kubernetes
app.kubernetes.io/name: connectapi
app.kubernetes.io/version: '20210929.1'
version: '20210929.1'
annotations:
azure-pipelines/jobName: '"Deploy ddzubow"'
azure-pipelines/org: 'https://build.hidden.com/tfs/ITA/'
azure-pipelines/pipeline: '"Connect.API.Deploy"'
azure-pipelines/pipelineId: '"369"'
azure-pipelines/project: Eviti UX
azure-pipelines/run: '20210929.1'
azure-pipelines/runuri: 'https://build.hidden.com/tfs/ITA/Eviti UX/_build/results?buildId=76782'
spec:
replicas: 1
selector:
matchLabels:
app: connectapi
template:
metadata:
creationTimestamp: null
labels:
app: connectapi
app.kubernetes.io/component: api
app.kubernetes.io/managed-by: kubernetes
app.kubernetes.io/name: connectapi
app.kubernetes.io/version: '20210929.1'
environment: development
version: '20210929.1'
annotations:
apparmor.security.beta.kubernetes.io/pod: runtime/default
container.apparmor.security.beta.kubernetes.io/connectapi: runtime/default
spec:
volumes:
- name: tmp
emptyDir: {}
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: ev-dev-kv-keys
containers:
- name: connectapi
image: 'hidden.azurecr.io/connect-api:20210929.1'
ports:
- name: http
containerPort: 9090
protocol: TCP
envFrom:
- configMapRef:
name: database-config
- configMapRef:
name: appurl-config
- secretRef:
name: rabbitmq-svr-default-user
env:
- name: ASPNETCORE_URLS
value: 'http://+:9090'
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 5m
memory: 10Mi
volumeMounts:
- name: tmp
mountPath: /tmp
- name: secrets-store-inline
readOnly: true
mountPath: /mnt/secrets-store
readinessProbe:
httpGet:
path: api/connect.api/health/ready
port: 9090
scheme: HTTP
initialDelaySeconds: 5
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
securityContext:
capabilities:
drop:
- ALL
privileged: false
runAsUser: 2000
runAsNonRoot: true
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
securityContext: {}
schedulerName: default-scheduler
tolerations:
- key: kubernetes.azure.com/scalesetpriority
operator: Equal
value: spot
effect: NoSchedule
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
daniv-msft
This is a urgent issue, as it is blocking multiple members of our team.
This is a new issue that popped up in the last day or two. Previously, everything was working fine and no configuration changes were made within Bridge. When I hit the play button to start a debugging session in Visual Studio 2019, I get the below error. It's very non-descript.
bridge-mindarcocli logs: 2021-09-29T19:26:04.6070205Z | MindaroCli | TRACE | Event: Command.Start{"properties":{"arguments":"prep-connect --namespace ddzubow --output json","isRoutingEnabled":"false"},"metrics":null} \nOperation context: {"clientRequestId":null,"correlationId":"4eed9fc6-ac18-4b23-b6b6-63e09c892a75:611a5ea270d8","requestId":null,"userSubscriptionId":null,"startTime":"2021-09-29T19:26:04.2657856Z","userAgent":"LPK/2.1.20210515.1-887601f7 VisualStudio/16.11.31702.278","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210421.2","requestHeaders":{},"loggingProperties":{"applicationName":"MindaroCli","deviceOperatingSystem":"Microsoft Windows 10.0.19042","framework":".NET Core 3.1.19","macAddressHash":"f9771b7903aa8bf07ae55f5e7ecfe6c583910ea2efafe0c05575dd8a8d8bfe3d","processId":21588,"targetEnvironment":"Production","commandId":"611a5ea270d8"}}
2021-09-29T19:26:04.7423560Z | MindaroCli | TRACE | Running Microsoft.BridgeToKubernetes.Exe.Commands.Connect.PrepConnectCommand...
2021-09-29T19:26:04.7464980Z | MindaroCli | TRACE | Initializing progress reporter...
2021-09-29T19:26:06.0492501Z | MindaroCli | ERROR | Dependency: Prep Connect {"target":null,"success":false,"duration":null,"properties":{"requestId":"null","clientRequestId":"null","correlationRequestId":"null"}}
2021-09-29T19:26:06.7146822Z | MindaroCli | ERROR | An unexpected error occurred: 'Value cannot be null. (Parameter 'source')'\n
2021-09-29T19:26:06.7162779Z | MindaroCli | ERROR | To see our active issues or file a bug report, please visit https://aka.ms/bridge-to-k8s-report.\n
2021-09-29T19:26:06.7180875Z | MindaroCli | ERROR | For diagnostic information, see logs at 'C:\Users\ddzubow\AppData\Local\Temp\Bridge To Kubernetes'.\n
2021-09-29T19:26:06.7366334Z | MindaroCli | ERROR | Logging handled exception: System.ArgumentNullException: {"ClassName":"System.ArgumentNullException","Message":"Value cannot be null.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":" at System.Linq.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument)\r\n at System.Linq.Enumerable.SelectMany[TSource,TResult](IEnumerable{"properties":{"arguments":"prep-connect --namespace ddzubow --output json","result":"Failed","failureReason":"Value cannot be null. (Parameter 'source')"},"metrics":{"duration":2226.0}}
1 source, Func2 selector)\r\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager.<>cDisplayClass31_2.<_CollectServicesToRouteAsync>b2(V1Endpoints e)\r\n at System.Linq.Enumerable.WhereListIterator1.MoveNext()\r\n at System.Linq.Enumerable.Any[TSource](IEnumerable1 source)\r\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager.<>cDisplayClass31_0.<<_CollectServicesToRouteAsync>b0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager.GetReachableServicesAsync(String namespaceName, ILocalProcessConfig localProcessConfig, IProgress1 progress, CancellationToken cancellationToken)\r\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass13_0.<<GetElevationRequestsAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func1 func, FailureConfig failureConfig)\r\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.GetElevationRequestsAsync(IProgress`1 progress, CancellationToken cancellationToken)\r\n at Microsoft.BridgeToKubernetes.Exe.Commands.Connect.PrepConnectCommand.ExecuteAsync()\r\n at Microsoft.BridgeToKubernetes.Exe.CliApp.RunCommandAsync(String[] args, CancellationToken cancellationToken)\r\n at Microsoft.BridgeToKubernetes.Exe.CliApp.ExecuteAsync(String[] args, CancellationToken cancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147467261,"Source":"System.Linq","WatsonBuckets":null,"ParamName":"source"} 2021-09-29T19:26:06.7430648Z | MindaroCli | TRACE | Event: Command.Endbridge-library logs: 2021-09-29T19:26:04.7801317Z | Library | TRACE | Pulling kubeconfig...\nOperation context:{"clientRequestId":null,"correlationId":"4eed9fc6-ac18-4b23-b6b6-63e09c892a75:611a5ea270d8:fc6d9e51f10d","requestId":null,"userSubscriptionId":null,"startTime":"2021-09-29T19:26:04.7185608Z","userAgent":"LPK/2.1.20210515.1-887601f7 VisualStudio/16.11.31702.278","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210421.2","requestHeaders":{},"loggingProperties":{"applicationName":"Library","deviceOperatingSystem":"Microsoft Windows 10.0.19042","framework":".NET Core 3.1.19","macAddressHash":"f9771b7903aa8bf07ae55f5e7ecfe6c583910ea2efafe0c05575dd8a8d8bfe3d","processId":21588,"targetEnvironment":"Production"}}
2021-09-29T19:26:05.0546233Z | Library | TRACE | Event: CloudProvider {"properties":{"clusterFQDNDomain":"AzurePublic"},"metrics":null}
2021-09-29T19:26:05.6440921Z | Library | TRACE | Dependency: Kubernetes {"target":"ListServicesInNamespaceAsync","success":true,"duration":null,"properties":{}}
2021-09-29T19:26:05.8659007Z | Library | TRACE | Dependency: Kubernetes {"target":"ListPodsInNamespaceAsync","success":true,"duration":null,"properties":{}}
2021-09-29T19:26:05.8701128Z | Library | TRACE | Detected headless service with selectors. Using selectors to look for endpoints...\nOperation context: {"clientRequestId":null,"correlationId":"4eed9fc6-ac18-4b23-b6b6-63e09c892a75:611a5ea270d8:f7975d157ccd","requestId":null,"userSubscriptionId":null,"startTime":"2021-09-29T19:26:04.7185608Z","userAgent":"LPK/2.1.20210515.1-887601f7 VisualStudio/16.11.31702.278","requestHttpMethod":null,"requestUri":null,"version":"1.0.20210421.2","requestHeaders":{},"loggingProperties":{"applicationName":"Library","deviceOperatingSystem":"Microsoft Windows 10.0.19042","framework":".NET Core 3.1.19","macAddressHash":"f9771b7903aa8bf07ae55f5e7ecfe6c583910ea2efafe0c05575dd8a8d8bfe3d","processId":21588,"targetEnvironment":"Production","isRoutingEnabled":false}}
2021-09-29T19:26:05.8889485Z | Library | TRACE | Detected headless service with selectors. Using selectors to look for endpoints...
2021-09-29T19:26:05.9115336Z | Library | TRACE | Dependency: Kubernetes {"target":"ListEndpointsInNamespaceAsync","success":true,"duration":null,"properties":{}}
2021-09-29T19:26:05.9777340Z | Library | TRACE | Dependency: Kubernetes {"target":"ListEndpointsInNamespaceAsync","success":true,"duration":null,"properties":{}}
2021-09-29T19:26:05.9815619Z | Library | TRACE | Found 2 endpoint(s) matching headless service 'rabbitmq-svr-nodes'
2021-09-29T19:26:05.9932139Z | Library | TRACE | Event: KubernetesRemoteEnvironmentManager-GetReachableServices {"properties":{"result":"Failed"},"metrics":{"durationInMs":850.0}}
2021-09-29T19:26:05.9986941Z | Library | TRACE | Event: ConnectManagementClient-GetElevationRequests {"properties":{"result":"Failed"},"metrics":{"durationInMs":885.0}}
2021-09-29T19:26:05.9998689Z | Library | ERROR | Failed to get elevation requests.
2021-09-29T19:26:06.0356755Z | Library | ERROR | Logging handled exception: System.ArgumentNullException: {"ClassName":"System.ArgumentNullException","Message":"Value cannot be null.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":" at System.Linq.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument)\r\n at System.Linq.Enumerable.SelectMany[TSource,TResult](IEnumerable
1 source, Func2 selector)\r\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager.<>cDisplayClass31_2.<_CollectServicesToRouteAsync>b2(V1Endpoints e)\r\n at System.Linq.Enumerable.WhereListIterator1.MoveNext()\r\n at System.Linq.Enumerable.Any[TSource](IEnumerable1 source)\r\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager.<>cDisplayClass31_0.<<_CollectServicesToRouteAsync>b0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at Microsoft.BridgeToKubernetes.Library.Connect.KubernetesRemoteEnvironmentManager.GetReachableServicesAsync(String namespaceName, ILocalProcessConfig localProcessConfig, IProgress1 progress, CancellationToken cancellationToken)\r\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ConnectManagementClient.<>c__DisplayClass13_0.<<GetElevationRequestsAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at Microsoft.BridgeToKubernetes.Library.ManagementClients.ManagementClientExceptionStrategy.RunWithHandlingAsync[T](Func1 func, FailureConfig failureConfig)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147467261,"Source":"System.Linq","WatsonBuckets":null,"ParamName":"source"}Microsoft.VisualStudio.Kubernetes.Debugging
{ "timestamp": "2021-09-29 15-42-04.498", "message": "Begin a connect session", "exception": { "type": "Microsoft.VisualStudio.Kubernetes.Debugging.RpcContracts.DscException", "message": "Something went wrong.", "stack": " at Microsoft.VisualStudio.Kubernetes.Debugging.ServiceHub.DscClient.d15`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.VisualStudio.Kubernetes.Debugging.ServiceHub.DscClient.d 17.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.VisualStudio.Kubernetes.Debugging.ServiceHub.Connect.ConnectService.<>c__DisplayClass9_0.<b0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.VisualStudio.Kubernetes.Debugging.RpcContracts.Telemetry.KubeConnectScopeBase`1.d 27
1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Microsoft.VisualStudio.Kubernetes.Debugging.ServiceHub.Connect.ConnectService.<GetElevationRequestsAsync>d__9.MoveNext()", "code": -2146233088, "inner": { "type": "Microsoft.VisualStudio.Kubernetes.Debugging.RpcContracts.DscException", "message": "An unexpected error occurred: 'Value cannot be null. (Parameter 'source')'\r\nTo see our active issues or file a bug report, please visit https://aka.ms/bridge-to-k8s-report.\r\nFor diagnostic information, see logs at 'C:\\Users\\ddzubow\\AppData\\Local\\Temp\\Bridge To Kubernetes'.\r\n", "stack": " at Microsoft.VisualStudio.Kubernetes.Debugging.ServiceHub.DscClient.<InvokeAsync>d__151.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at Microsoft.VisualStudio.Telemetry.WindowsErrorReporting.WatsonReport.GetClrWatsonExceptionInfo(Exception exceptionObject)", "code": -2146233088, "inner": null } }, "operationName": "ConnectionManagerService.ConnectAsync", "stage": "End", "elapsedTime": "00:00:05.3235339", "completionState": "Error" }2021-09-29T19-41-59.1319070.log
Environment Details Client used: Visual Studio 16.11.3 Bridge version:
Operating System: Windows