search

futurewei-cloud / alcor

Alcor: Cloud native SDN platform powered by Kubernetes and Istio
MIT License
32 stars 33 forks source link

[DPM] DPM Failed Handle Request #668

Open yanmo96 opened 3 years ago

yanmo96 commented 3 years ago

When run alcor-network-performance test 20 times with 3 concurrency. Each run create 100 networks, and put 1 instance per network. Rally will give the following error:

Traceback (most recent call last):
  File "/root/rally_venv/lib/python3.6/site-packages/rally/task/runner.py", line 69, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/opt/rally/plugins/neutron_network_performance.py", line 63, in run
    self._delete_subnet(subnet)
  File "/root/rally_venv/lib/python3.6/site-packages/rally_openstack/task/scenarios/neutron/utils.py", line 204, in _delete_subnet
    self.neutron.delete_subnet(subnet["subnet"]["id"])
  File "/root/rally_venv/lib/python3.6/site-packages/rally/task/service.py", line 114, in wrapper
    return func(instance, *args, **kwargs)
  File "/root/rally_venv/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/root/rally_venv/lib/python3.6/site-packages/rally_openstack/common/services/network/neutron.py", line 574, in delete_subnet
    self.client.delete_subnet(subnet_id)
  File "/root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 888, in delete_subnet
    return self.delete(self.subnet_path % (subnet))
  File "/root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 353, in delete
    headers=headers, params=params)
  File "/root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 334, in retry_request
    headers=headers, params=params)
  File "/root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 297, in do_request
    self._handle_fault_response(status_code, replybody, resp)
  File "/root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 272, in _handle_fault_response
    exception_handler_v20(status_code, error_body)
  File "/root/rally_venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20
    request_ids=request_ids)
neutronclient.common.exceptions.InternalServerError: 500 : [{"timestamp":"2021-07-14T20:47:19.615+0000","status":500,"error":"Internal Server Error","message":"DPM Failed Handle Request","path":"/project/d323f4568474427dbe92a5c4a986ec05/subnets/15782b34-eff6-4f2b-a3f7-82651b7be479/routetable"}]

Full Rally report and log:

DPM log from K8s:

{"rsType":"ROUTER","opType":"INFO","rsOpTypes":null,"ports_internal":null,"vpcs_internal":null,"subnets_internal":null,"security_groups_internal":null,"neighbor_info":null,"neighbor_table":null,"routers_internal":[{"routerConfiguration":{"revisionNumber":"1","formatVersion":"1","subnetRoutingTables":[{"subnetId":"15782b34-eff6-4f2b-a3f7-82651b7be479","routingRules":[],"subnet_id":"15782b34-eff6-4f2b-a3f7-82651b7be479","routing_rules":[]}],"requestId":"8fdc25b1-5b75-470c-9d7f-5dd4b6f20cb2","hostDvrMac":"","messageType":"FULL","format_version":"1","revision_number":"1","request_id":"8fdc25b1-5b75-470c-9d7f-5dd4b6f20cb2","id":null,"message_type":"FULL","host_dvr_mac":"","subnet_routing_tables":[{"subnetId":"15782b34-eff6-4f2b-a3f7-82651b7be479","routingRules":[],"subnet_id":"15782b34-eff6-4f2b-a3f7-82651b7be479","routing_rules":[]}]},"operationType":"INFO","operation_type":"INFO","configuration":{"revisionNumber":"1","formatVersion":"1","subnetRoutingTables":[{"subnetId":"15782b34-eff6-4f2b-a3f7-82651b7be479","routingRules":[],"subnet_id":"15782b34-eff6-4f2b-a3f7-82651b7be479","routing_rules":[]}],"requestId":"8fdc25b1-5b75-470c-9d7f-5dd4b6f20cb2","hostDvrMac":"","messageType":"FULL","format_version":"1","revision_number":"1","request_id":"8fdc25b1-5b75-470c-9d7f-5dd4b6f20cb2","id":null,"message_type":"FULL","host_dvr_mac":"","subnet_routing_tables":[{"subnetId":"15782b34-eff6-4f2b-a3f7-82651b7be479","routingRules":[],"subnet_id":"15782b34-eff6-4f2b-a3f7-82651b7be479","routing_rules":[]}]}}]}]
2021-07-14 20:47:18.956 DEBUG 1 --- [nio-8080-exec-7] org.apache.tomcat.util.http.Parameters   : Set query string encoding to UTF-8
2021-07-14 20:47:18.957 DEBUG 1 --- [nio-8080-exec-7] o.apache.catalina.valves.RemoteIpValve   : Incoming request /network-configuration with originalRemoteAddr [172.16.190.98], originalRemoteHost=[172.16.190.98], originalSecure=[false], originalScheme=[http], originalServerName=[dataplanemanager-service.default.svc.cluster.local], originalServerPort=[9010] will be seen as newRemoteAddr=[172.16.190.98], newRemoteHost=[172.16.190.98], newSecure=[false], newScheme=[http], newServerName=[dataplanemanager-service.default.svc.cluster.local], newServerPort=[9010]
2021-07-14 20:47:18.957 DEBUG 1 --- [nio-8080-exec-7] o.a.c.authenticator.AuthenticatorBase    : Security checking request POST /network-configuration
2021-07-14 20:47:18.957 DEBUG 1 --- [nio-8080-exec-7] org.apache.catalina.realm.RealmBase      :   No applicable constraints defined
2021-07-14 20:47:18.957 DEBUG 1 --- [nio-8080-exec-7] o.a.c.authenticator.AuthenticatorBase    : Not subject to any constraint
2021-07-14 20:47:18.957 DEBUG 1 --- [nio-8080-exec-7] org.apache.tomcat.util.http.Parameters   : Set encoding to UTF-8
2021-07-14 20:47:18.957 DEBUG 1 --- [nio-8080-exec-7] o.s.web.servlet.DispatcherServlet        : POST "/network-configuration", parameters={}
2021-07-14 20:47:18.958 DEBUG 1 --- [thread-pool-4-1] i.g.n.s.i.n.u.i.InternalThreadLocalMap   : -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2021-07-14 20:47:18.958 DEBUG 1 --- [thread-pool-4-1] i.g.n.s.i.n.u.i.InternalThreadLocalMap   : -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2021-07-14 20:47:18.958 DEBUG 1 --- [nio-8080-exec-7] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to com.futurewei.alcor.dataplane.controller.DpmController#createNetworkConfiguration(NetworkConfiguration)
2021-07-14 20:47:18.958 DEBUG 1 --- [nio-8080-exec-7] m.m.a.RequestResponseBodyMethodProcessor : Read "application/json;charset=UTF-8" to [NetworkConfiguration{rsType=ROUTER, opType=INFO, portEntities=null, vpcs=null, subnets=null, securit (truncated)...]
2021-07-14 20:47:18.959 DEBUG 1 --- [nio-8080-exec-7] c.f.alcor.common.stats.StatisticsAspect  : Calculating duration of com.futurewei.alcor.dataplane.controller.DpmController.createNetworkConfiguration()...
2021-07-14 20:47:18.959 DEBUG 1 --- [nio-8080-exec-7] c.f.alcor.common.stats.StatisticsAspect  : Calculating duration of com.futurewei.alcor.dataplane.cache.SubnetPortsCache.getSubnetPorts()...
2021-07-14 20:47:18.960  INFO 1 --- [nio-8080-exec-7] c.f.alcor.common.stats.StatisticsAspect  : com.futurewei.alcor.dataplane.cache.SubnetPortsCache.getSubnetPorts() startTime: 20570661317719151ns, endTime: 20570661318685254ns, duration: 0ms
2021-07-14 20:47:18.960 DEBUG 1 --- [thread-pool-4-2] c.f.a.d.client.grpc.DataPlaneClientImpl  : 
2021-07-14 20:47:18.984 DEBUG 1 --- [thread-pool-4-1] i.g.n.s.i.n.u.i.PlatformDependent        : org.jctools-core.MpscChunkedArrayQueue: available
2021-07-14 20:47:19.000 DEBUG 1 --- [thread-pool-4-1] io.opencensus.tags.Tags                  : Couldn't load full implementation for TagsComponent, now trying to load lite implementation.

java.lang.ClassNotFoundException: io.opencensus.impl.tags.TagsComponentImpl
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[na:na]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[na:na]
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:92) ~[AlcorDataPlaneManager-0.1.0.jar:0.1.0-SNAPSHOT]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
    at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
    at java.base/java.lang.Class.forName(Class.java:398) ~[na:na]
    at io.opencensus.tags.Tags.loadTagsComponent(Tags.java:99) ~[opencensus-api-0.21.0.jar!/:0.21.0]
    at io.opencensus.tags.Tags.<clinit>(Tags.java:35) ~[opencensus-api-0.21.0.jar!/:0.21.0]
    at io.grpc.internal.CensusStatsModule.<init>(CensusStatsModule.java:91) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.getEffectiveInterceptors(AbstractManagedChannelImplBuilder.java:528) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:512) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.newChannel(DataPlaneClientImpl.java:197) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.doSendGoalState(DataPlaneClientImpl.java:132) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.sendGoalState(DataPlaneClientImpl.java:125) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.lambda$doSendGoalStates$0(DataPlaneClientImpl.java:100) ~[classes!/:0.1.0-SNAPSHOT]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.48.Final.jar!/:4.1.48.Final]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2021-07-14 20:47:19.001 DEBUG 1 --- [thread-pool-4-1] io.opencensus.tags.Tags                  : Couldn't load lite implementation for TagsComponent, now using default implementation for TagsComponent.

java.lang.ClassNotFoundException: io.opencensus.impllite.tags.TagsComponentImplLite
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[na:na]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[na:na]
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:92) ~[AlcorDataPlaneManager-0.1.0.jar:0.1.0-SNAPSHOT]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
    at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
    at java.base/java.lang.Class.forName(Class.java:398) ~[na:na]
    at io.opencensus.tags.Tags.loadTagsComponent(Tags.java:112) ~[opencensus-api-0.21.0.jar!/:0.21.0]
    at io.opencensus.tags.Tags.<clinit>(Tags.java:35) ~[opencensus-api-0.21.0.jar!/:0.21.0]
    at io.grpc.internal.CensusStatsModule.<init>(CensusStatsModule.java:91) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.getEffectiveInterceptors(AbstractManagedChannelImplBuilder.java:528) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:512) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.newChannel(DataPlaneClientImpl.java:197) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.doSendGoalState(DataPlaneClientImpl.java:132) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.sendGoalState(DataPlaneClientImpl.java:125) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.lambda$doSendGoalStates$0(DataPlaneClientImpl.java:100) ~[classes!/:0.1.0-SNAPSHOT]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.48.Final.jar!/:4.1.48.Final]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2021-07-14 20:47:19.009 DEBUG 1 --- [thread-pool-4-1] io.opencensus.stats.Stats                : Couldn't load full implementation for StatsComponent, now trying to load lite implementation.

java.lang.ClassNotFoundException: io.opencensus.impl.stats.StatsComponentImpl
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[na:na]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[na:na]
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:92) ~[AlcorDataPlaneManager-0.1.0.jar:0.1.0-SNAPSHOT]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
    at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
    at java.base/java.lang.Class.forName(Class.java:398) ~[na:na]
    at io.opencensus.stats.Stats.loadStatsComponent(Stats.java:97) ~[opencensus-api-0.21.0.jar!/:0.21.0]
    at io.opencensus.stats.Stats.<clinit>(Stats.java:34) ~[opencensus-api-0.21.0.jar!/:0.21.0]
    at io.grpc.internal.CensusStatsModule.<init>(CensusStatsModule.java:93) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.getEffectiveInterceptors(AbstractManagedChannelImplBuilder.java:528) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:512) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.newChannel(DataPlaneClientImpl.java:197) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.doSendGoalState(DataPlaneClientImpl.java:132) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.sendGoalState(DataPlaneClientImpl.java:125) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.lambda$doSendGoalStates$0(DataPlaneClientImpl.java:100) ~[classes!/:0.1.0-SNAPSHOT]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.48.Final.jar!/:4.1.48.Final]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2021-07-14 20:47:19.010 DEBUG 1 --- [thread-pool-4-1] io.opencensus.stats.Stats                : Couldn't load lite implementation for StatsComponent, now using default implementation for StatsComponent.

java.lang.ClassNotFoundException: io.opencensus.impllite.stats.StatsComponentImplLite
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[na:na]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[na:na]
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:92) ~[AlcorDataPlaneManager-0.1.0.jar:0.1.0-SNAPSHOT]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
    at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
    at java.base/java.lang.Class.forName(Class.java:398) ~[na:na]
    at io.opencensus.stats.Stats.loadStatsComponent(Stats.java:110) ~[opencensus-api-0.21.0.jar!/:0.21.0]
    at io.opencensus.stats.Stats.<clinit>(Stats.java:34) ~[opencensus-api-0.21.0.jar!/:0.21.0]
    at io.grpc.internal.CensusStatsModule.<init>(CensusStatsModule.java:93) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.getEffectiveInterceptors(AbstractManagedChannelImplBuilder.java:528) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:512) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.newChannel(DataPlaneClientImpl.java:197) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.doSendGoalState(DataPlaneClientImpl.java:132) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.sendGoalState(DataPlaneClientImpl.java:125) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.lambda$doSendGoalStates$0(DataPlaneClientImpl.java:100) ~[classes!/:0.1.0-SNAPSHOT]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.48.Final.jar!/:4.1.48.Final]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2021-07-14 20:47:19.031 DEBUG 1 --- [thread-pool-4-2] io.opencensus.trace.Tracing              : Couldn't load full implementation for TraceComponent, now trying to load lite implementation.

java.lang.ClassNotFoundException: io.opencensus.impl.trace.TraceComponentImpl
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[na:na]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[na:na]
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:92) ~[AlcorDataPlaneManager-0.1.0.jar:0.1.0-SNAPSHOT]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
    at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
    at java.base/java.lang.Class.forName(Class.java:398) ~[na:na]
    at io.opencensus.trace.Tracing.loadTraceComponent(Tracing.java:95) ~[opencensus-api-0.21.0.jar!/:0.21.0]
    at io.opencensus.trace.Tracing.<clinit>(Tracing.java:37) ~[opencensus-api-0.21.0.jar!/:0.21.0]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.getEffectiveInterceptors(AbstractManagedChannelImplBuilder.java:539) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:512) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.newChannel(DataPlaneClientImpl.java:197) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.doSendGoalState(DataPlaneClientImpl.java:132) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.sendGoalState(DataPlaneClientImpl.java:125) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.lambda$doSendGoalStates$0(DataPlaneClientImpl.java:100) ~[classes!/:0.1.0-SNAPSHOT]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.48.Final.jar!/:4.1.48.Final]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2021-07-14 20:47:19.032 DEBUG 1 --- [thread-pool-4-2] io.opencensus.trace.Tracing              : Couldn't load lite implementation for TraceComponent, now using default implementation for TraceComponent.

java.lang.ClassNotFoundException: io.opencensus.impllite.trace.TraceComponentImplLite
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[na:na]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[na:na]
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:92) ~[AlcorDataPlaneManager-0.1.0.jar:0.1.0-SNAPSHOT]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
    at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
    at java.base/java.lang.Class.forName(Class.java:398) ~[na:na]
    at io.opencensus.trace.Tracing.loadTraceComponent(Tracing.java:108) ~[opencensus-api-0.21.0.jar!/:0.21.0]
    at io.opencensus.trace.Tracing.<clinit>(Tracing.java:37) ~[opencensus-api-0.21.0.jar!/:0.21.0]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.getEffectiveInterceptors(AbstractManagedChannelImplBuilder.java:539) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:512) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.newChannel(DataPlaneClientImpl.java:197) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.doSendGoalState(DataPlaneClientImpl.java:132) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.sendGoalState(DataPlaneClientImpl.java:125) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.lambda$doSendGoalStates$0(DataPlaneClientImpl.java:100) ~[classes!/:0.1.0-SNAPSHOT]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.48.Final.jar!/:4.1.48.Final]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2021-07-14 20:47:19.064 DEBUG 1 --- [thread-pool-4-2] io.grpc.LoadBalancerRegistry             : Service loader found Provider{policy=round_robin, priority=5, available=true}
2021-07-14 20:47:19.064 DEBUG 1 --- [thread-pool-4-2] io.grpc.LoadBalancerRegistry             : Service loader found PickFirstLoadBalancerProvider{policy=pick_first, priority=5, available=true}
2021-07-14 20:47:19.139 DEBUG 1 --- [thread-pool-4-2] io.grpc.Context                          : Storage override doesn't exist. Using default

java.lang.ClassNotFoundException: io.grpc.override.ContextStorageOverride
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[na:na]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[na:na]
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:92) ~[AlcorDataPlaneManager-0.1.0.jar:0.1.0-SNAPSHOT]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
    at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
    at java.base/java.lang.Class.forName(Class.java:315) ~[na:na]
    at io.grpc.Context$LazyStorage.createStorage(Context.java:144) ~[grpc-context-1.23.0.jar!/:1.23.0]
    at io.grpc.Context$LazyStorage.<clinit>(Context.java:133) ~[grpc-context-1.23.0.jar!/:1.23.0]
    at io.grpc.Context.storage(Context.java:121) ~[grpc-context-1.23.0.jar!/:1.23.0]
    at io.grpc.Context.current(Context.java:181) ~[grpc-context-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor.interceptCall(CensusTracingModule.java:386) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[grpc-api-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor.interceptCall(CensusStatsModule.java:691) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[grpc-api-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.ManagedChannelImpl.newCall(ManagedChannelImpl.java:826) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.ForwardingManagedChannel.newCall(ForwardingManagedChannel.java:63) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:127) ~[grpc-stub-1.23.0.jar!/:1.23.0]
    at com.futurewei.alcor.schema.GoalStateProvisionerGrpc$GoalStateProvisionerBlockingStub.pushNetworkResourceStates(GoalStateProvisionerGrpc.java:253) ~[schema-0.1.0-SNAPSHOT.jar!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.doSendGoalState(DataPlaneClientImpl.java:137) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.sendGoalState(DataPlaneClientImpl.java:125) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.lambda$doSendGoalStates$0(DataPlaneClientImpl.java:100) ~[classes!/:0.1.0-SNAPSHOT]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.48.Final.jar!/:4.1.48.Final]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2021-07-14 20:47:19.152 DEBUG 1 --- [thread-pool-4-2] io.perfmark.PerfMark                     : Error during PerfMark.<clinit>

java.lang.ClassNotFoundException: io.perfmark.impl.SecretPerfMarkImpl$PerfMarkImpl
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[na:na]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[na:na]
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:92) ~[AlcorDataPlaneManager-0.1.0.jar:0.1.0-SNAPSHOT]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
    at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
    at java.base/java.lang.Class.forName(Class.java:315) ~[na:na]
    at io.perfmark.PerfMark.<clinit>(PerfMark.java:37) ~[perfmark-api-0.17.0.jar!/:na]
    at io.grpc.internal.ClientCallImpl.<init>(ClientCallImpl.java:100) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.ManagedChannelImpl$RealChannel.newCall(ManagedChannelImpl.java:861) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.ServiceConfigInterceptor.interceptCall(ServiceConfigInterceptor.java:153) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[grpc-api-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor.interceptCall(CensusTracingModule.java:388) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[grpc-api-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor.interceptCall(CensusStatsModule.java:691) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156) ~[grpc-api-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.ManagedChannelImpl.newCall(ManagedChannelImpl.java:826) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.internal.ForwardingManagedChannel.newCall(ForwardingManagedChannel.java:63) ~[grpc-core-1.23.0.jar!/:1.23.0]
    at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:127) ~[grpc-stub-1.23.0.jar!/:1.23.0]
    at com.futurewei.alcor.schema.GoalStateProvisionerGrpc$GoalStateProvisionerBlockingStub.pushNetworkResourceStates(GoalStateProvisionerGrpc.java:253) ~[schema-0.1.0-SNAPSHOT.jar!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.doSendGoalState(DataPlaneClientImpl.java:137) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.sendGoalState(DataPlaneClientImpl.java:125) ~[classes!/:0.1.0-SNAPSHOT]
    at com.futurewei.alcor.dataplane.client.grpc.DataPlaneClientImpl.lambda$doSendGoalStates$0(DataPlaneClientImpl.java:100) ~[classes!/:0.1.0-SNAPSHOT]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.48.Final.jar!/:4.1.48.Final]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
yanmo96 commented 3 years ago

Dahai's PR #664 could solve this issue, will test it out.

yanmo96 commented 3 years ago

After pull the PR #664 to test, the above issue from rally test is gone