apolloconfig / apollo

Apollo is a reliable configuration management system suitable for microservice configuration management scenarios.
https://www.apolloconfig.com
Apache License 2.0
29.18k stars 10.21k forks source link

configservice在处理客户端请求时,出现500报错 #5190

Closed weifly closed 2 months ago

weifly commented 3 months ago

各位大牛好!最近遇到一个问题,还请帮忙分析下原因,描述如下:

公司运行了apollo配置中心,configservice集群中包含8个服务实例。有一个appid的客户端数量在5000左右。7月29日晚上,这个appid发布了新版配置,大量的客户端拉取新版配置,产生了惊群效应。现象是configservice的cpu100%了,所有服务实例都自动重启了,其中有一个服务实例在处理客户端请求时,出现少量的500报错,不能自动恢复,在手工重启这个实例后,不再报错。

apollo configservice版本:v1.0.0 出现500错误的接口 长轮询 /notifications/v2 查询配置 /configs/{appId}/{clusterName}/{namespace:.+}

其中一类错误: Already value [org.springframework.orm.jpa.EntityManagerHolder@705437c9] for key [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean@36a860dd] bound to thread [http-nio-8080-exec-147] image

另一类错误: EntityManager is closed; nested exception is java.lang.IllegalStateException: EntityManager is closed image

我在本地无法复现这个问题,看了apollo及spring的源码也没有找出详细错误原因,还请帮忙分析下,多谢!

BlackBear2003 commented 3 months ago

Hi weifly, maybe it has something related with the code here

https://github.com/apolloconfig/apollo/blob/49bd8ccab93c2484dacabd9d0709bb49efbdfc4b/apollo-configservice/src/main/java/com/ctrip/framework/apollo/configservice/controller/NotificationControllerV2.java#L188

By the way, can you provide more details?

stale[bot] commented 2 months ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

stale[bot] commented 2 months ago

This issue has been automatically closed because it has not had activity in the last 7 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.