应用重启报错：Sync config failed 和 Long polling failed 和 Get config services failed

[reignsr]

• [x] 我已经检查过discussions
• [x] 我已经搜索过issues
• [x] 我已经仔细检查过FAQ

问题描述

• 版本 Apollo-1.4.0

• 环境 Apollo服务端部署在虚机（centos），客户端应用（springboot）部署在k8s集群，应用启动时和运行中都会出现如下警告信息。

• 日志： |2021-10-19 14:58:09.544| WARN| main|com.ctrip.framework.apollo.internals.AbstractConfigRepository:27||||||||Sync config failed, will retry. Repository class com.ctrip.framework.apollo.internals.RemoteConfigRepository, reason: Get config services failed from http://meta.ts-accs.epay/services/config?appId=fcc-fses&ip=10.34.192.12 [Cause: java.io.IOException: Server returned HTTP response code: 503 for URL: http://meta.ts-accs.epay/services/config?appId=fcc-fses&ip=10.34.192.12 [Cause: Server returned HTTP response code: 503 for URL: http://meta.ts-accs.epay/services/config?appId=fcc-fses&ip=10.34.192.12 [Cause: Server returned HTTP response code: 503 for URL: http://meta.ts-accs.epay/services/config?appId=fcc-fses&ip=10.34.192.12]]]| |2021-10-19 14:58:32.589| WARN|Apollo-RemoteConfigLongPollService-1|com.ctrip.framework.apollo.internals.RemoteConfigLongPollService:193||||||||Long polling failed, will retry in 1 seconds. appId: fcc-fses, cluster: default, namespaces: fcc-fses-cnuwx-provider, long polling url: null, reason: Get config services failed from http://meta.ts-accs.epay/services/config?appId=fcc-fses&ip=10.34.192.12 [Cause: Could not complete get operation [Cause: Read timed out]]| |2021-10-19 14:58:43.586| WARN| main|com.ctrip.framework.apollo.internals.LocalFileConfigRepository:168||||||||Sync config from upstream repository class com.ctrip.framework.apollo.internals.RemoteConfigRepository failed, reason: Get config services failed from http://meta.ts-accs.epay/services/config?appId=fcc-fses&ip=10.34.192.12 [Cause: Could not complete get operation [Cause: Read timed out]]|

• 补充说明

  1. curl命令结果如下： ~ $ curl -v http://meta.ts-accs.epay/services/config?appId=fcc-fses&ip=10.34.192.12 ~ $ Trying 172.18.106.19:80... TCP_NODELAY set Connected to meta.ts-accs.epay (172.18.106.19) port 80 (#0) GET /services/config?appId=fcc-fses HTTP/1.1 Host: meta.ts-accs.epay User-Agent: curl/7.67.0 Accept: /* Mark bundle as not supporting multiuse HTTP/1.1 503 Service Unavailable content-length: 91 content-type: text/plain date: Tue, 19 Oct 2021 07:09:32 GMT server: envoy Connection #0 to host meta.ts-accs.epay left intact upstream connect error or disconnect/reset before headers. reset reason: connection failure
  2. 重启configservice后，再次重启应用，以上问题得以缓解；
  3. 以上问题近几个月频繁出现，很多业务系统都出现过，并非偶现的个例问题，有时应用连续重启几次可以启动成功；
  4. 确认appid、apollo.meta、env等参数设置正确无误；

问题总结

1. 如上问题是否能判断是哪方面原因，能否指引个诊断方向？
2. eureka注册中心参数或者linux操作系统内核参数等，有没有建议的调优方案？

[nobodyiam]

It looks like 503 is returned by the load balancer, which indicates the config service is probably unhealthy. You may check the cpu/mem/gc stats.

[reignsr]

问题定位，操作系统内核参数设置问题，导致 tcp连接 packets reject 和 sockets drop，调整后恢复正常。