search

apache / inlong

Apache InLong - a one-stop, full-scenario integration framework for massive data
https://inlong.apache.org/
Apache License 2.0
1.39k stars 531 forks source link

[Bug][Manager] Failed to handle request on path: /inlong/manager/openapi/dataproxy/getIpList/inlong_agent_system by user: admin #11412

Open PeterZh6 opened 9 hours ago

PeterZh6 commented 9 hours ago

What happened

Summary

When starting Inlong with Docker, the manager continuously throws an exception, leading to system malfunction. This issue was previously mentioned in Issue #11071. However, the associated PR only addressed a null check without resolving the core issue. Reopening this ticket for further investigation.

Description

After deploying Inlong with Docker, the manager produces the following error:

[ ] 2024-10-27 07:19:22.293 -ERROR [http-nio-8083-exec-4] m.s.c.InlongClusterServiceImpl:938 - group not found by groupId=inlong_agent_system 
[ ] 2024-10-27 07:19:22.294 -ERROR [http-nio-8083-exec-4] w.c.ControllerExceptionHandler:129 - failed to handle request on path: /inlong/manager/openapi/dataproxy/getIpList/inlong_agent_system by user: admin

This error prevents the manager from functioning, rendering the entire system unusable.

Click to view complete log 
[ ] 2024-10-27 07:19:22.293 -ERROR [http-nio-8083-exec-4] m.s.c.InlongClusterServiceImpl:938 - group not found by groupId=inlong_agent_system 
[ ] 2024-10-27 07:19:22.294 -ERROR [http-nio-8083-exec-4] w.c.ControllerExceptionHandler:129 - failed to handle request on path: /inlong/manager/openapi/dataproxy/getIpList/inlong_agent_system by user: admin 
org.apache.inlong.manager.common.exceptions.BusinessException: group not found by groupId=inlong_agent_system
        at org.apache.inlong.manager.service.cluster.InlongClusterServiceImpl.getDataProxyNodes(InlongClusterServiceImpl.java:939) ~[manager-service-2.0.0.jar:2.0.0]
        at org.apache.inlong.manager.service.cluster.InlongClusterServiceImpl$$FastClassBySpringCGLIB$$5ac56ec1.invoke() ~[manager-service-2.0.0.jar:2.0.0]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.34.jar:5.3.34]
        at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) ~[spring-aop-5.3.27.jar:5.3.27]
        at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) ~[spring-aop-5.3.27.jar:5.3.27]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:704) ~[spring-aop-5.3.27.jar:5.3.27]
        at org.apache.inlong.manager.service.cluster.InlongClusterServiceImpl$$EnhancerBySpringCGLIB$$8a4f27dd.getDataProxyNodes() ~[manager-service-2.0.0.jar:2.0.0]
        at sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_342]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_342]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.27.jar:5.3.27]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-5.3.27.jar:5.3.27]
        at com.sun.proxy.$Proxy210.getDataProxyNodes(Unknown Source) ~[?:?]
        at org.apache.inlong.manager.web.controller.openapi.DataProxyController.getIpList(DataProxyController.java:59) ~[manager-web.jar:2.0.0]
        at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_342]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_342]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.34.jar:5.3.34]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.34.jar:5.3.34]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:555) ~[tomcat-embed-core-9.0.75.jar:4.0.FR]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) ~[tomcat-embed-core-9.0.75.jar:4.0.FR]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.inlong.manager.web.filter.TenantInsertionFilter.doFilter(TenantInsertionFilter.java:71) ~[manager-web.jar:2.0.0]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.inlong.manager.web.auth.tenant.TenantAuthenticatingFilter.doFilter(TenantAuthenticatingFilter.java:81) ~[manager-web.jar:2.0.0]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.inlong.manager.web.auth.openapi.OpenAPIFilter.doFilter(OpenAPIFilter.java:84) ~[manager-web.jar:2.0.0]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) ~[shiro-core-1.13.0.jar:1.13.0]
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) ~[shiro-core-1.13.0.jar:1.13.0]
        at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) ~[shiro-core-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:370) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.inlong.manager.web.trace.TraceFilter.doFilter(TraceFilter.java:80) ~[manager-web.jar:2.0.0]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.inlong.manager.web.filter.EmptyTenantFilter.doFilter(EmptyTenantFilter.java:50) ~[manager-web.jar:2.0.0]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.inlong.manager.web.filter.HttpServletRequestFilter.doFilter(HttpServletRequestFilter.java:45) ~[manager-web.jar:2.0.0]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.34.jar:5.3.34]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.34.jar:5.3.34]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.34.jar:5.3.34]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.34.jar:5.3.34]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.34.jar:5.3.34]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.34.jar:5.3.34]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]

Here is a screenshot explaining what's going on.

abe493934a82bc986851aa0d5ac3e9d

What you expected to happen

In version 1.13.0 of Inlong Docker, this issue does not occur. The latest version seems to introduce a change where the Inlong Agent module queries the inlong_agent_system group, which has not been created. This results in a request the manager cannot handle.

Possible reasons of this issue

In PR #10757, a Sender object was introduced with group_id=INLONG_AGENT_SYSTEM, created with DefaultMessageSender. Within this setup:

How to reproduce

  1. Pull the latest Inlong code.
  2. Deploy Inlong using Docker:
    • Navigate to docker/docker-compose and follow README.md instructions.
    • Copy apache_inlong_manager.sql and apache_inlong_audit_mysql.sql into a sql directory under docker/docker-compose.
  3. Run docker-compose up -d.
  4. Enter the Inlong manager container with docker exec -it manager and inspect logs/manager-all.log, or use the Logs option in Docker Desktop if available.
  5. Locate the error messages as detailed above.

Environment

WSL2 Ubuntu 22.04.4 LTS Docker Engine v25.0.3

InLong version

master

InLong Component

InLong Manager

Are you willing to submit PR?

Code of Conduct

fuweng11 commented 2 hours ago

This exception will not leading to system malfunction. What problem do you want to solve?

PeterZh6 commented 38 minutes ago

This exception will not leading to system malfunction. What problem do you want to solve?

Thanks for your attention :)

Firstly, this exception keeps popping up once the manager container is started. It is not only annoying as it fills the screen with exception message, but also does contains unwanted behaviour as mentioned above.

Secondly, the configuration of data syncronization job fails immediately after it is approved, while this problem didn't exist on the 1.13.0 version of docker image. I am not sure whether I'm doing it right. It would be great if you could try to configure a data syncronization job with docker deployment of inlong personally and check whether this issue will cause the system to malfunction.

I am try to reproduce this example but encountered this issue.