search

DataLinkDC / dinky

Dinky is a real-time data development platform based on Apache Flink, enabling agile data development, deployment and operation.
http://www.dinky.org.cn
Apache License 2.0
3.07k stars 1.13k forks source link

[Bug] The dispatcher returned from the ServletContext does not support asynchronous dispatching #3737

Closed fengDianDemaNong closed 2 weeks ago

fengDianDemaNong commented 3 weeks ago

Search before asking

What happened

a90926d7350bcd2ac917b7fde59500bI encounter this situation when submitting a jar using k8s session mode These are some of the parameters submitted EXECUTE JAR WITH ( 'uri'='rs:/lib/flink-doris-connector-1.17-1.6.2.jar', 'main-class'='org.apache.doris.flink.tools.cdc.CdcTools', 'args'=' --database ods \ --single-sink true \ --table-prefix odsrecircle \ --mysql-conf hostname=xxxx.xxx.xxx.xxx \ --mysql-conf port=3306 \ --mysql-conf username=xxxxxx \ --mysql-conf password=xxxxxx \ --mysql-conf database-name=recircle_business \ --sink-conf fenodes=xxxx.xxx.xxx.xxx:8030 \ --sink-conf benodes=xxxx.xxx.xxx.xxx:8040 \ --sink-conf username=root \ --sink-conf password=xxxxxxxxxxxxxx \ --sink-conf jdbc-url=jdbc:mysql://xxxx.xxx.xxx.xxx:9030 \ --sink-conf sink.label-prefix=rc_bus \ --table-conf replication_num=1' );

[dinky] 2024-08-23 09:43:58.592 INFO 36 --- [nio-8888-exec-5] org.dinky.job.builder.JobUDFBuilder: A total of 0 UDF have been Init. [dinky] 2024-08-23 09:43:58.592 INFO 36 --- [nio-8888-exec-5] org.dinky.job.builder.JobUDFBuilder: Initializing Flink UDF...Finish [dinky] 2024-08-23 09:43:58.596 INFO 36 --- [nio-8888-exec-5] org.dinky.utils.KerberosUtil: Simple authentication mode [dinky] 2024-08-23 09:43:58.654 INFO 36 --- [ionShutdownHook] org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler: Stopping... [dinky] 2024-08-23 09:43:58.654 INFO 36 --- [ionShutdownHook] org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler: BrokerAvailabilityEvent[available=false, SimpleBrokerMessageHandler [org.springframework.messaging.simp.broker.DefaultSubscriptionRegistry@6407ed17]] [dinky] 2024-08-23 09:43:58.654 INFO 36 --- [ionShutdownHook] org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler: Stopped. [dinky] 2024-08-23 09:43:58.655 INFO 36 --- [ionShutdownHook] org.springframework.boot.web.embedded.tomcat.GracefulShutdown: Commencing graceful shutdown. Waiting for active requests to complete

[dinky] 2024-08-23 09:44:28.665 INFO 36 --- [ionShutdownHook] org.springframework.context.support.DefaultLifecycleProcessor: Failed to shut down 1 bean with phase value 2147483647 within timeout of 30000ms: [webServerGracefulShutdown] [dinky] 2024-08-23 09:44:28.705 INFO 36 --- [tomcat-shutdown] org.springframework.boot.web.embedded.tomcat.GracefulShutdown: Graceful shutdown aborted with one or more requests still active [dinky] 2024-08-23 09:44:32.784 WARN 36 --- [nio-8888-exec-3] org.dinky.context.SseSessionContextHolder: Failed to complete sseEmitter, Error: The dispatcher returned from the ServletContext does not support asynchronous dispatching [dinky] 2024-08-23 09:44:32.845 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.845 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.845 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.846 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.846 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.846 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.846 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:ResponseBodyEmitter has already completed [dinky] 2024-08-23 09:44:32.847 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:37.804 INFO 36 --- [ionShutdownHook] com.alibaba.druid.pool.DruidDataSource: {dataSource-1} closing ... [dinky] 2024-08-23 09:44:37.810 INFO 36 --- [ionShutdownHook] com.alibaba.druid.pool.DruidDataSource: {dataSource-1} closed

What you expected to happen

[dinky] 2024-08-23 09:43:58.592 INFO 36 --- [nio-8888-exec-5] org.dinky.job.builder.JobUDFBuilder: A total of 0 UDF have been Init. [dinky] 2024-08-23 09:43:58.592 INFO 36 --- [nio-8888-exec-5] org.dinky.job.builder.JobUDFBuilder: Initializing Flink UDF...Finish [dinky] 2024-08-23 09:43:58.596 INFO 36 --- [nio-8888-exec-5] org.dinky.utils.KerberosUtil: Simple authentication mode [dinky] 2024-08-23 09:43:58.654 INFO 36 --- [ionShutdownHook] org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler: Stopping... [dinky] 2024-08-23 09:43:58.654 INFO 36 --- [ionShutdownHook] org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler: BrokerAvailabilityEvent[available=false, SimpleBrokerMessageHandler [org.springframework.messaging.simp.broker.DefaultSubscriptionRegistry@6407ed17]] [dinky] 2024-08-23 09:43:58.654 INFO 36 --- [ionShutdownHook] org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler: Stopped. [dinky] 2024-08-23 09:43:58.655 INFO 36 --- [ionShutdownHook] org.springframework.boot.web.embedded.tomcat.GracefulShutdown: Commencing graceful shutdown. Waiting for active requests to complete

[dinky] 2024-08-23 09:44:28.665 INFO 36 --- [ionShutdownHook] org.springframework.context.support.DefaultLifecycleProcessor: Failed to shut down 1 bean with phase value 2147483647 within timeout of 30000ms: [webServerGracefulShutdown] [dinky] 2024-08-23 09:44:28.705 INFO 36 --- [tomcat-shutdown] org.springframework.boot.web.embedded.tomcat.GracefulShutdown: Graceful shutdown aborted with one or more requests still active [dinky] 2024-08-23 09:44:32.784 WARN 36 --- [nio-8888-exec-3] org.dinky.context.SseSessionContextHolder: Failed to complete sseEmitter, Error: The dispatcher returned from the ServletContext does not support asynchronous dispatching [dinky] 2024-08-23 09:44:32.845 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.845 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.845 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.846 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.846 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.846 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.846 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:ResponseBodyEmitter has already completed [dinky] 2024-08-23 09:44:32.847 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:37.804 INFO 36 --- [ionShutdownHook] com.alibaba.druid.pool.DruidDataSource: {dataSource-1} closing ... [dinky] 2024-08-23 09:44:37.810 INFO 36 --- [ionShutdownHook] com.alibaba.druid.pool.DruidDataSource: {dataSource-1} closed

How to reproduce

These are some of the parameters submitted EXECUTE JAR WITH ( 'uri'='rs:/lib/flink-doris-connector-1.17-1.6.2.jar', 'main-class'='org.apache.doris.flink.tools.cdc.CdcTools', 'args'=' --database ods \ --single-sink true \ --table-prefix odsrecircle \ --mysql-conf hostname=xxxx.xxx.xxx.xxx \ --mysql-conf port=3306 \ --mysql-conf username=xxxxxx \ --mysql-conf password=xxxxxx \ --mysql-conf database-name=recircle_business \ --sink-conf fenodes=xxxx.xxx.xxx.xxx:8030 \ --sink-conf benodes=xxxx.xxx.xxx.xxx:8040 \ --sink-conf username=root \ --sink-conf password=xxxxxxxxxxxxxx \ --sink-conf jdbc-url=jdbc:mysql://xxxx.xxx.xxx.xxx:9030 \ --sink-conf sink.label-prefix=rc_bus \ --table-conf replication_num=1' );

Anything else

No response

Version

dev

Are you willing to submit PR?

Code of Conduct

github-actions[bot] commented 3 weeks ago

Hello @fengDianDemaNong, this issue is about web, so I assign it to @Zzm0809. If you have any questions, you can comment and reply.

你好 @fengDianDemaNong, 这个 issue 是关于 web 的，所以我把它分配给了 @Zzm0809。如有任何问题，可以评论回复。

Zzm0809 commented 3 weeks ago

遇到了什么问题？。上述日志未发现有任何问题 只是sse相关日志，未发现与任务相关有用的日志

github-actions[bot] commented 3 weeks ago

Hello @fengDianDemaNong, we need you to provide reproduction steps so that we can help you troubleshoot the problem. Issues are marked `Need Reproduce`.

你好 @fengDianDemaNong, 我们需要你提供复现步骤以便于我们帮你排查问题，Issues 标注为 `Need Reproduce` 。

fengDianDemaNong commented 3 weeks ago

Hello @fengDianDemaNong, we need you to provide reproduction steps so that we can help you troubleshoot the problem. Issues are marked Need Reproduce.

你好 @fengDianDemaNong, 我们需要你提供复现步骤以便于我们帮你排查问题，Issues 标注为 Need Reproduce 。 报错内容：[dinky] 2024-08-23 09:44:32.784 WARN 36 --- [nio-8888-exec-3] org.dinky.context.SseSessionContextHolder: Failed to complete sseEmitter, Error: The dispatcher returned from the ServletContext does not support asynchronous dispatching

复现步骤： 在页面上提交jar包，到k8s session mode 集群（我flink部署在k8s上） 代码： EXECUTE JAR WITH ( 'uri'='rs:/lib/flink-doris-connector-1.17-1.6.2.jar', 'main-class'='org.apache.doris.flink.tools.cdc.CdcTools', 'args'=' --database ods --single-sink true --table-prefix odsrecircle --mysql-conf hostname=xxxx.xxx.xxx.xxx --mysql-conf port=3306 --mysql-conf username=xxxxxx --mysql-conf password=xxxxxx --mysql-conf database-name=recircle_business --sink-conf fenodes=xxxx.xxx.xxx.xxx:8030 --sink-conf benodes=xxxx.xxx.xxx.xxx:8040 --sink-conf username=root --sink-conf password=xxxxxxxxxxxxxx --sink-conf jdbc-url=jdbc:mysql://xxxx.xxx.xxx.xxx:9030 --sink-conf sink.label-prefix=rc_bus --table-conf replication_num=1' );

现象： dinky提交一会之后，会直接挂掉，dinky服务直接挂了。

输出日志： [dinky] 2024-08-23 09:43:58.592 INFO 36 --- [nio-8888-exec-5] org.dinky.job.builder.JobUDFBuilder: A total of 0 UDF have been Init. [dinky] 2024-08-23 09:43:58.592 INFO 36 --- [nio-8888-exec-5] org.dinky.job.builder.JobUDFBuilder: Initializing Flink UDF...Finish [dinky] 2024-08-23 09:43:58.596 INFO 36 --- [nio-8888-exec-5] org.dinky.utils.KerberosUtil: Simple authentication mode [dinky] 2024-08-23 09:43:58.654 INFO 36 --- [ionShutdownHook] org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler: Stopping... [dinky] 2024-08-23 09:43:58.654 INFO 36 --- [ionShutdownHook] org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler: BrokerAvailabilityEvent[available=false, SimpleBrokerMessageHandler [org.springframework.messaging.simp.broker.DefaultSubscriptionRegistry@6407ed17]] [dinky] 2024-08-23 09:43:58.654 INFO 36 --- [ionShutdownHook] org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler: Stopped. [dinky] 2024-08-23 09:43:58.655 INFO 36 --- [ionShutdownHook] org.springframework.boot.web.embedded.tomcat.GracefulShutdown: Commencing graceful shutdown. Waiting for active requests to complete

[dinky] 2024-08-23 09:44:28.665 INFO 36 --- [ionShutdownHook] org.springframework.context.support.DefaultLifecycleProcessor: Failed to shut down 1 bean with phase value 2147483647 within timeout of 30000ms: [webServerGracefulShutdown] [dinky] 2024-08-23 09:44:28.705 INFO 36 --- [tomcat-shutdown] org.springframework.boot.web.embedded.tomcat.GracefulShutdown: Graceful shutdown aborted with one or more requests still active [dinky] 2024-08-23 09:44:32.784 WARN 36 --- [nio-8888-exec-3] org.dinky.context.SseSessionContextHolder: Failed to complete sseEmitter, Error: The dispatcher returned from the ServletContext does not support asynchronous dispatching [dinky] 2024-08-23 09:44:32.845 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.845 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.845 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.846 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.846 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.846 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:32.846 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:ResponseBodyEmitter has already completed [dinky] 2024-08-23 09:44:32.847 ERROR 36 --- [ scheduler-9] org.dinky.context.SseSessionContextHolder: Error sending sse data:Failed write [dinky] 2024-08-23 09:44:37.804 INFO 36 --- [ionShutdownHook] com.alibaba.druid.pool.DruidDataSource: {dataSource-1} closing ... [dinky] 2024-08-23 09:44:37.810 INFO 36 --- [ionShutdownHook] com.alibaba.druid.pool.DruidDataSource: {dataSource-1} closed

我查了一下：Servlet 异步提交需要加一个参数asyncSuppored=true 这是关于这个问题的网上文章： https://blog.csdn.net/yzf913214/article/details/53915378 https://www.jianshu.com/p/a65f6f122ffc

fengDianDemaNong commented 3 weeks ago

这是执行： image

Zzm0809 commented 3 weeks ago

可以尝试升级版本到 1.1.0,该版本修复了 execute jar 语法在解析参数列表时会导致 oom 的问题

fengDianDemaNong commented 3 weeks ago

可以尝试升级版本到 1.1.0,该版本修复了 execute jar 语法在解析参数列表时会导致 oom 的问题

1.10 sql都不能执行，下一个版本大概什么时候发布

Zzm0809 commented 3 weeks ago

可以尝试升级版本到 1.1.0,该版本修复了 execute jar 语法在解析参数列表时会导致 oom 的问题

1.10 sql都不能执行，下一个版本大概什么时候发布

什么原因无法执行

fengDianDemaNong commented 3 weeks ago

可以尝试升级版本到 1.1.0,该版本修复了 execute jar 语法在解析参数列表时会导致 oom 的问题

1.10 sql都不能执行，下一个版本大概什么时候发布

什么原因无法执行

执行 show databases 都报错，找不到集群地址，我看有人提了对应bug，在dev分支，代码还没合并。

Zzm0809 commented 3 weeks ago

可以尝试升级版本到 1.1.0,该版本修复了 execute jar 语法在解析参数列表时会导致 oom 的问题

1.10 sql都不能执行，下一个版本大概什么时候发布

什么原因无法执行

执行 show databases 都报错，找不到集群地址，我看有人提了对应bug，在dev分支，代码还没合并。

那建议排查你的环境问题, 另外我说的什么原因无法执行 是说你的 1.1.0 的 sql 不能执行的原因, 如果不能执行,是否是没有按照文档说明的步骤执行呢? 根据日志无法解决吗??

fengDianDemaNong commented 3 weeks ago

可以尝试升级版本到 1.1.0,该版本修复了 execute jar 语法在解析参数列表时会导致 oom 的问题

1.10 sql都不能执行，下一个版本大概什么时候发布

什么原因无法执行

执行 show databases 都报错，找不到集群地址，我看有人提了对应bug，在dev分支，代码还没合并。

那建议排查你的环境问题, 另外我说的什么原因无法执行 是说你的 1.1.0 的 sql 不能执行的原因, 如果不能执行,是否是没有按照文档说明的步骤执行呢? 根据日志无法解决吗??

不是环境的问题，是这个问题https://github.com/DataLinkDC/dinky/issues/3704 。我看已经有人修复了，但需要下个版本才能更新修复

fengDianDemaNong commented 3 weeks ago

可以尝试升级版本到 1.1.0,该版本修复了 execute jar 语法在解析参数列表时会导致 oom 的问题

1.10 sql都不能执行，下一个版本大概什么时候发布

什么原因无法执行

执行 show databases 都报错，找不到集群地址，我看有人提了对应bug，在dev分支，代码还没合并。

那建议排查你的环境问题, 另外我说的什么原因无法执行 是说你的 1.1.0 的 sql 不能执行的原因, 如果不能执行,是否是没有按照文档说明的步骤执行呢? 根据日志无法解决吗??

不是环境的问题，是这个问题#3704 。我看已经有人修复了，但需要下个版本才能更新修复

刚刚用1.1.0 版本试了，提交jar包还是一样问题，服务挂了 image

Zzm0809 commented 3 weeks ago

刚刚用1.1.0 版本试了，提交jar包还是一样问题，服务挂了

目前无法复现挂掉的这个问题

fengDianDemaNong commented 3 weeks ago

刚刚用1.1.0 版本试了，提交jar包还是一样问题，服务挂了

目前无法复现挂掉的这个问题

方便的话我们加下V（qiang928855230）我们具体交流

Zzm0809 commented 3 weeks ago

刚刚用1.1.0 版本试了，提交jar包还是一样问题，服务挂了

目前无法复现挂掉的这个问题

方便的话我们加下V（qiang928855230）我们具体交流

官网有进微信群方式, 请进入社区交流群与社区人员联系

github-actions[bot] commented 2 weeks ago

Hi @, Since this issue was marked as requiring more information, no response was received for 7 days. The issue is now closed. If you have any questions, please comment and reply.

Hi @, 由于该 issue 被标记为需要更多信息，却 7 天未收到回应。现关闭 issue，若有任何问题，可评论回复。