search

neuvector / neuvector

Apache License 2.0
1.07k stars 202 forks source link

Frequent timeouts and forced logoff #600

Closed Eric-Domeier closed 1 year ago

Eric-Domeier commented 1 year ago

Environment Platform: Rancher Kubernetes/Platform Version(s): RKE2 Nodes: OpenSuse LEAP 15.4

Describe the bug Clicking around in Neuvector UI eventually leads to random timeout/logoff within a few minutes.

To Reproduce Steps to reproduce the behavior: Install Neuvector integration from the Rancher UI https://ranchermanager.docs.rancher.com/integrations-in-rancher/neuvector

Expected behavior Be able to interact with neuvector without timing out.

Screenshots image image

UI Issue Desktop (please complete the following information):**

Additional context

UI is accessed via rancher UI and logs in via SSO, after it timeouts unable to reaccess neuvector for several minutes.

Neuvector Manager pod logs

2023-01-06 19:05:24,863|INFO |MANAGER|com.neu.api.AuthenticationService(apply:826): post path auth
2023-01-06 19:05:26,074|INFO |MANAGER|com.neu.api.AuthenticationService(login$1:817): login with SUSE cookie: true
2023-01-06 19:05:26,074|INFO |MANAGER|com.neu.api.AuthenticationService(login$1:818): Client ip 10.0.200.39
2023-01-06 19:05:26,074|INFO |MANAGER|com.neu.api.AuthenticationService(login$1:819): admin login
2023-01-06 19:05:26,076|INFO |MANAGER|com.neu.api.AuthenticationService(apply:147): saml-g: servername is empty
2023-01-06 19:05:26,096|INFO |MANAGER|com.neu.api.AuthenticationService(apply:571): Getting EULA
2023-01-06 19:05:26,905|INFO |MANAGER|com.neu.api.AuthenticationService(apply:601): Getting license
2023-01-06 19:05:26,906|INFO |MANAGER|com.neu.api.ClusterService(apply:28): Getting cluster..
2023-01-06 19:05:27,224|INFO |MANAGER|com.neu.api.DashboardService(apply:213): Url: https://neuvector-svc-controller.cattle-neuvector-system:10443/v1/internal/system
2023-01-06 19:05:27,246|INFO |MANAGER|com.neu.api.DashboardService(apply:223): internalSystemData: InternalSystemData(Metrics(0,323,8,24,0,0,24,36,24,0,3,0,0,Discover,Kubernetes-Rancher,0,9,0,0,30,0,0),[Lcom.neu.model.Exposure;@4888fff0,[Lcom.neu.model.Exposure;@565345ae,None)
2023-01-06 19:05:27,246|INFO |MANAGER|com.neu.api.DashboardService(apply:1880): Exposure density: 0.35714285714285715
2023-01-06 19:05:27,395|INFO |MANAGER|com.neu.api.DashboardService(apply:1244): Dashboard notifications - Start: 2023-01-06T19:05:27.395Z
2023-01-06 19:05:27,409|INFO |MANAGER|com.neu.api.DashboardService(apply:1293): Dashboard notifications - Multiple API call end: 2023-01-06T19:05:27.409Z
2023-01-06 19:05:27,409|INFO |MANAGER|com.neu.api.DashboardService(apply:1294): Dashboard notifications - Multiple API call duration: 14
2023-01-06 19:05:27,410|INFO |MANAGER|com.neu.api.DashboardService(apply:1302): Dashboard notifications - Calculation Start: 2023-01-06T19:05:27.410Z
2023-01-06 19:05:27,413|INFO |MANAGER|com.neu.api.DashboardService(apply:1319): Dashboard notifications - End: 2023-01-06T19:05:27.413Z
2023-01-06 19:05:27,413|INFO |MANAGER|com.neu.api.DashboardService(apply:1320): Dashboard notifications - Calculation Duration: 3
2023-01-06 19:05:27,414|INFO |MANAGER|com.neu.api.DashboardService(apply:1324): Dashboard notifications - Duration: 18
2023-01-06 19:05:27,431|INFO |MANAGER|com.neu.api.DashboardService(apply:285): dashboard scores Start: 2023-01-06T19:05:27.431Z
2023-01-06 19:05:27,560|INFO |MANAGER|com.neu.api.DashboardService(apply:385): Dashboard scores - Multiple API call end: 2023-01-06T19:05:27.560Z
2023-01-06 19:05:27,560|INFO |MANAGER|com.neu.api.DashboardService(apply:386): Dashboard scores - Multiple API call duration: 129
2023-01-06 19:05:27,561|INFO |MANAGER|com.neu.api.DashboardService(apply:394): Dashboard scores - Calculation Start: 2023-01-06T19:05:27.561Z
2023-01-06 19:05:27,585|INFO |MANAGER|com.neu.api.DashboardService(apply:1704): groupSet: 19
2023-01-06 19:05:27,587|INFO |MANAGER|com.neu.api.DashboardService(apply:1419): serviceMaps.serviceMap: Map(nv.rancher.cattle-system -> ServiceStateIn(cattle-system,nv.rancher.cattle-system,Some(Discover),,[Lcom.neu.model.Pod;@638de5f3,false,container), nv.gitjob.cattle-fleet-system -> ServiceStateIn(cattle-fleet-system,nv.gitjob.cattle-fleet-system,Some(Discover),,[Lcom.neu.model.Pod;@ad0580e,false,container), nv.ui-plugin-operator.cattle-ui-plugin-system -> ServiceStateIn(cattle-ui-plugin-system,nv.ui-plugin-operator.cattle-ui-plugin-system,Some(Discover),,[Lcom.neu.model.Pod;@5a851db,false,container), containers -> ServiceStateIn(,containers,None,,[Lcom.neu.model.Pod;@499a1af5,false,container), nv.fleet-controller.cattle-fleet-system -> ServiceStateIn(cattle-fleet-system,nv.fleet-controller.cattle-fleet-system,Some(Discover),,[Lcom.neu.model.Pod;@45b4cc48,false,container), nv.fleet-agent.cattle-fleet-local-system -> ServiceStateIn(cattle-fleet-local-system,nv.fleet-agent.cattle-fleet-local-system,Some(Discover),,[Lcom.neu.model.Pod;@4d2cecd8,false,container), nv.rancher-webhook.cattle-system -> ServiceStateIn(cattle-system,nv.rancher-webhook.cattle-system,Some(Discover),,[Lcom.neu.model.Pod;@a4d4a2f,false,container))
2023-01-06 19:05:27,588|INFO |MANAGER|com.neu.api.DashboardService(apply:1420): serviceMaps.serviceMap.isEmpty: false
2023-01-06 19:05:27,589|INFO |MANAGER|com.neu.api.DashboardService(apply:1437): vulContainersByDomain: [Workload(c40295dd982f2ad5a960a0cfe0f137bce97efaf3d4acf2ba69a7003eaadeea4a,fleet-agent-7bcc7d69cf-8qmsm,discover,fleet-agent.cattle-fleet-local-system,,cattle-fleet-local-system,None,None,Some(ScanSummary4Dashboard(finished,0,0)),Some([Lcom.neu.model.WorkloadChildren;@702cfc96)), Workload(1df5f439fb70949c874d38066a616bae8c53ec2378a7caf819138a8e68104fda,fleet-controller-7f58774c8-fqs4g,discover,fleet-controller.cattle-fleet-system,,cattle-fleet-system,None,None,Some(ScanSummary4Dashboard(finished,0,0)),Some([Lcom.neu.model.WorkloadChildren;@5a4c8467)), Workload(45dcb02c9c245c5d1e146d6a61efa4ea8c999c565d45cc35a27ee80926ec49ef,gitjob-849c6cc45-hj2qh,discover,gitjob.cattle-fleet-system,,cattle-fleet-system,None,None,Some(ScanSummary4Dashboard(finished,0,0)),Some([Lcom.neu.model.WorkloadChildren;@3a30b133)), Workload(9b5a2fd02facd56d92e7795d2c47636dda15ddf8ca284df55e42f7939c541e52,rancher-84565759dd-4zn8f,discover,rancher.cattle-system,,cattle-system,None,None,Some(ScanSummary4Dashboard(finished,0,0)),Some([Lcom.neu.model.WorkloadChildren;@645622e8)), Workload(f4c82b99c763951cc8a5c642173ebe680853bfb99e73bd1e187aa03dad1aa0de,rancher-84565759dd-ktzm5,discover,rancher.cattle-system,,cattle-system,None,None,Some(ScanSummary4Dashboard(finished,0,0)),Some([Lcom.neu.model.WorkloadChildren;@7309bab8)), Workload(0ffb5c7a2ba2a6422350ddc596708df5edd71b0373bbabb1f32c6e07ed48373a,rancher-84565759dd-z62fk,discover,rancher.cattle-system,,cattle-system,None,None,Some(ScanSummary4Dashboard(finished,0,0)),Some([Lcom.neu.model.WorkloadChildren;@4dd86702)), Workload(aa510b5abe85d3ae945d71f9a00188c6ed8271b14d6549f9cd8e2cd5877341c6,rancher-webhook-577b778f8f-9qxx9,discover,rancher-webhook.cattle-system,,cattle-system,None,None,Some(ScanSummary4Dashboard(finished,0,0)),Some([Lcom.neu.model.WorkloadChildren;@725eeea3)), Workload(5e4aa3662d4ec17ba0ae8c77266dcdba1c6d8c5c82016f1efad7423ee1471dcb,ui-plugin-operator-7f5b7bdb47-29rmg,discover,ui-plugin-operator.cattle-ui-plugin-system,,cattle-ui-plugin-system,None,None,Some(ScanSummary4Dashboard(finished,0,0)),Some([Lcom.neu.model.WorkloadChildren;@274ad362))]
2023-01-06 19:05:27,590|INFO |MANAGER|com.neu.api.DashboardService(apply:1572): ingressConversations: 2
2023-01-06 19:05:27,591|INFO |MANAGER|com.neu.api.DashboardService(apply:1573): egressConversations: 3
2023-01-06 19:05:27,592|INFO |MANAGER|com.neu.api.DashboardService(apply:560): Dashbaord scores - End: 2023-01-06T19:05:27.592Z
2023-01-06 19:05:27,593|INFO |MANAGER|com.neu.api.DashboardService(apply:561): Dashbaord scores - Calculation Duration: 31
2023-01-06 19:05:27,593|INFO |MANAGER|com.neu.api.DashboardService(apply:565): Dashbaord scores - Duration: 161
2023-01-06 19:06:06,141|INFO |MANAGER|com.neu.api.NotificationService(apply:505): getting graph data
2023-01-06 19:06:06,176|INFO |MANAGER|com.neu.api.NotificationService(apply:510): Parsing graph data
2023-01-06 19:06:06,191|INFO |MANAGER|com.neu.api.NotificationService(apply:514): Sending data
2023-01-06 19:06:06,195|INFO |MANAGER|com.neu.api.NotificationService(apply:515): blacklist:  None
2023-01-06 19:06:11,006|INFO |MANAGER|com.neu.api.GroupService(apply:677): Getting sensors
2023-01-06 19:08:04,393|INFO |MANAGER|com.neu.api.NotificationService(com$neu$api$NotificationService$$getNotifications:663): notifications Start: 2023-01-06T19:08:04.393Z
2023-01-06 19:08:04,584|INFO |MANAGER|com.neu.api.NotificationService(apply:56): Getting ip locations
2023-01-06 19:09:19,282|INFO |MANAGER|com.neu.api.DashboardService(apply:1244): Dashboard notifications - Start: 2023-01-06T19:09:19.281Z
2023-01-06 19:09:19,479|WARN |MANAGER|com.neu.client.RestClient$(handleError:323): spray.httpx.UnsuccessfulResponseException: Status: 401 Unauthorized
Body: {"code":3,"error":"Authentication failed","message":"Authentication failed"}
        at spray.httpx.ResponseTransformation$$anonfun$unmarshal$1.apply(ResponseTransformation.scala:39)
        at spray.httpx.ResponseTransformation$$anonfun$unmarshal$1.apply(ResponseTransformation.scala:31)
        at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)
        at scala.util.Try$.apply(Try.scala:192)
        at scala.util.Success.map(Try.scala:237)
        at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
        at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
williamlin-suse commented 1 year ago

Your user profile has timeout setting as 300 seconds, correct? You could change it to a bigger timeout value

Eric-Domeier commented 1 year ago

Ah, yeah that seems to make it better so far thanks :) I didn't notice that option there.