cloudwu / skynet

A lightweight online game framework
MIT License
13.33k stars 4.2k forks source link

skynet底层死循环,经常性出现,还没找到复现原因,请云风大大帮忙看看~ #1562

Closed gels123 closed 2 years ago

gels123 commented 2 years ago

我们游戏的登录服、游戏服,都有这种情况发生,出现的问题的是worker线程: top查看: image pstack查看: image 出现上面的死循环后,loginslave就会readline阻塞: image

木有思路怎么解决, 也不好找出怎么触发的

zhangshiqian1214 commented 2 years ago

你估计不怎么关注提问和wiki, 你这样上来就贴图的, 云大是不会回复的, 要用markdown语法提问

terry8210 commented 2 years ago

请问什么版本呢?程序大概跑了多久出现?

gels123 commented 2 years ago

skynet 1.4.0版本,内网的一台内存不怎么足够(内存15G,使用15G,available 内存只有60M左右)的机器上,大概30分钟--2小时出现一次,外网内存足够的环境,大概半个月出现一次。比较奇怪,怀疑与可用内存有关。

gels123 commented 2 years ago

请问什么版本呢?程序大概跑了多久出现? 今天内网出现的概率好高,每30分钟、2小时就出现一次

terry8210 commented 2 years ago

跑什么业务呢?是在做压测吗?

terry8210 commented 2 years ago

你外网外网内存足够的环境,大概半个月出现一次,当出现的时候,内存的占用如何呢?

gels123 commented 2 years ago

跑什么业务呢?是在做压测吗?

没有的,只是普通的qa测试功能,业务不怎么繁忙。

gels123 commented 2 years ago

你外网外网内存足够的环境,大概半个月出现一次,当出现的时候,内存的占用如何呢?

外网是亚马逊云,8核16G的centos7.9,出现了一次死机自动重启的情况,从云监控上看,cpu使用正常,如下图 lQLPDhtIsR57gOfNAlvNBXywZwE3DzqKprwCR8WErUAMAA_1404_603 内存占用木有监控图表,图上中间断了的几分钟,就是死机重启的时间,怀疑是所有线程把cpu吃满了,导致了重启。

BanKnight commented 2 years ago

死循环很好解决阿,主动coredump,然后查看调用栈。 这里麻烦的是,gdb看到的都是c++栈,你要自己转到lua去

然后我印象中,skynet底层有死循环检测的

gels123 commented 2 years ago

死循环很好解决阿,主动coredump,然后查看调用栈。 这里麻烦的是,gdb看到的都是c++栈,你要自己转到lua去

image 我gdb attack; thread 线程序号;的时候,发现 执行n没反应。skynet有lua层的死循环检测,但是没有c层的。

terry8210 commented 2 years ago

你们有C的服务?

BanKnight commented 2 years ago

你是怎么看出来是死循环的,我看你的栈不像是死循环, 更像是有一个service不停的在发包,导致另外一个service处理不过来。

gels123 commented 2 years ago

你们有C的服务? 没有的,只有一些so的动态库。

cloudwu commented 2 years ago
  1. 所有文本信息,请不要贴图。用 markdown 发。
  2. worker 如果处理某个服务无法退出, log 中有 endless 提示;或 overload 提示。你应该确认是哪个服务的问题,C 层的 worker 没有意义。
  3. 一切问题都应该先更新到最新版本再做测试。
gels123 commented 2 years ago
  1. 所有文本信息,请不要贴图。用 markdown 发。
  2. worker 如果处理某个服务无法退出, log 中有 endless 提示;或 overload 提示。你应该确认是哪个服务的问题,C 层的 worker 没有意义。
  3. 一切问题都应该先更新到最新版本再做测试。

好的,感谢云风大大。

gels123 commented 2 years ago

你是怎么看出来是死循环的,我看你的栈不像是死循环, 更像是有一个service不停的在发包,导致另外一个service处理不过来。

不排除这种可能性。登录服的业务非常简单,出问题的是loginslave,telnet stat指令,消息队列数量正常,没有非常大。下会再出现的时候,我用markdown发出来,大家一起帮忙看看

BanKnight commented 2 years ago

我印象中,skynet中是有提供这方面的支持的(消息队列的包数量就是其中一个)。 你可以利用这一点,做出自己的应对或者记录,另外你也不要只看特定service的,可以统一加上。

gels123 commented 2 years ago
  1. telnet下stat输入如下,clusterd TIMEOUT了 :00000004 cpu:0.167047 message:12526 mqlen:0 task:0 :00000006 cpu:0.180531 message:12525 mqlen:0 task:0 :00000007 cpu:0.175914 message:12970 mqlen:0 task:0 :00000008 cpu:0.15737 message:12528 mqlen:0 task:0 :00000009 cpu:0.17317 message:12525 mqlen:0 task:0 :0000000b cpu:15.053134 message:29120 mqlen:0 task:1 :0000000c TIMEOUT :0000000d cpu:0.695918 message:12750 mqlen:0 task:0 :0000000e cpu:0.189708 message:14161 mqlen:0 task:0 :0000000f cpu:0.171058 message:12813 mqlen:0 task:0 :00000010 cpu:0.161672 message:12535 mqlen:0 task:0 :00000011 cpu:0.161176 message:12530 mqlen:0 task:1 :00000012 cpu:0.164745 message:12530 mqlen:0 task:1 :00000013 cpu:0.16202 message:12530 mqlen:0 task:1 :00000014 cpu:0.150618 message:12530 mqlen:0 task:1 :00000015 cpu:0.151364 message:12535 mqlen:0 task:0 :00000016 cpu:0.15679 message:12530 mqlen:0 task:1 :00000017 cpu:0.158439 message:12530 mqlen:0 task:1 :00000018 cpu:0.155964 message:12530 mqlen:0 task:1 :00000019 cpu:0.166917 message:12530 mqlen:0 task:1 :0000001a cpu:27.9016 message:790208 mqlen:0 task:4 :0000001b cpu:0.17208 message:12593 mqlen:0 task:1 :0000001c cpu:14.777129 message:392176 mqlen:0 task:1 :0000001d cpu:0.15881 message:12525 mqlen:0 task:0 :0000001e cpu:0.225021 message:12599 mqlen:0 task:1 :0000001f cpu:0.223384 message:12524 mqlen:0 task:0 :00000020 cpu:14.062971 message:388891 mqlen:0 task:1 :00000021 cpu:29.6589 message:766956 mqlen:0 task:4 :00000022 cpu:10.082068 message:392043 mqlen:0 task:2 :00000023 cpu:7.905762 message:388843 mqlen:0 task:2 :00000024 cpu:8.143836 message:388841 mqlen:0 task:2 :00000025 cpu:7.871144 message:389001 mqlen:0 task:2 :00000026 cpu:7.867759 message:389221 mqlen:0 task:2 :00000027 cpu:9.045355 message:388849 mqlen:0 task:2 :00000028 cpu:7.780188 message:388845 mqlen:0 task:2 :00000029 cpu:7.532821 message:388843 mqlen:0 task:2 :0000002a cpu:7.820151 message:388839 mqlen:0 task:2 :0000002b cpu:7.448197 message:388839 mqlen:0 task:2 :0000002c cpu:6.903775 message:388839 mqlen:0 task:2 :0000002d cpu:7.862837 message:388839 mqlen:0 task:2 :0000002e cpu:7.479924 message:388839 mqlen:0 task:2 :0000002f cpu:7.688312 message:388839 mqlen:0 task:2 :00000030 cpu:8.379022 message:388839 mqlen:0 task:2 :00000031 cpu:6.67336 message:388839 mqlen:0 task:2 :00000032 cpu:8.876747 message:391355 mqlen:0 task:2 :00000033 cpu:8.814409 message:389311 mqlen:0 task:2 :00000034 cpu:6.683588 message:389237 mqlen:0 task:2 :00000035 cpu:6.16605 message:388839 mqlen:0 task:2 :00000036 cpu:6.98609 message:388839 mqlen:0 task:2 :00000037 cpu:5.949252 message:388839 mqlen:0 task:2 :00000038 cpu:7.69556 message:388839 mqlen:0 task:2 :00000039 cpu:7.313885 message:388839 mqlen:0 task:2 :0000003a cpu:6.672888 message:388839 mqlen:0 task:2 :0000003b cpu:8.72753 message:388839 mqlen:0 task:2 :0000003c cpu:6.155578 message:388839 mqlen:0 task:2 :0000003d cpu:8.627446 message:388839 mqlen:0 task:2 :0000003e cpu:6.453674 message:389263 mqlen:0 task:2 :0000003f cpu:6.393783 message:388839 mqlen:0 task:2 :00000040 cpu:8.785346 message:388839 mqlen:0 task:2 :00000041 cpu:6.474854 message:388839 mqlen:0 task:2 :00000042 cpu:6.058518 message:388839 mqlen:0 task:2 :00000043 cpu:6.235863 message:388838 mqlen:0 task:2 :00000044 cpu:8.314483 message:388839 mqlen:0 task:2 :00000045 cpu:6.31897 message:388839 mqlen:0 task:2 :00000046 cpu:6.373016 message:388839 mqlen:0 task:2 :00000047 cpu:6.259525 message:388839 mqlen:0 task:2 :00000048 cpu:7.984371 message:388839 mqlen:0 task:2 :00000049 cpu:6.293048 message:388839 mqlen:0 task:2 :0000004a cpu:8.46165 message:388839 mqlen:0 task:2 :0000004b cpu:6.742866 message:388839 mqlen:0 task:2 :0000004c cpu:6.523421 message:388839 mqlen:0 task:2 :0000004d cpu:6.328213 message:388839 mqlen:0 task:2 :0000004e cpu:8.276054 message:388839 mqlen:0 task:2 :0000004f cpu:6.474633 message:388839 mqlen:0 task:2 :00000050 cpu:8.194498 message:388839 mqlen:0 task:2 :00000051 cpu:6.232458 message:388839 mqlen:0 task:2 :00000052 cpu:6.304563 message:388838 mqlen:0 task:2 :00000053 cpu:7.349345 message:388839 mqlen:0 task:2 :00000054 cpu:8.733903 message:388838 mqlen:0 task:2 :00000055 cpu:7.884209 message:388949 mqlen:0 task:2 :00000056 cpu:6.207544 message:388839 mqlen:0 task:2 :00000057 cpu:6.563968 message:388839 mqlen:0 task:2 :00000058 cpu:6.260773 message:389669 mqlen:0 task:2 :00000059 cpu:6.838968 message:389009 mqlen:0 task:2 :0000005a cpu:6.270751 message:388839 mqlen:0 task:2 :0000005b cpu:6.861052 message:388839 mqlen:0 task:2 :0000005c cpu:8.658639 message:388839 mqlen:0 task:2 :0000005d cpu:6.23105 message:388839 mqlen:0 task:2 :0000005e cpu:8.434644 message:389650 mqlen:0 task:2 :0000005f cpu:6.061344 message:388947 mqlen:0 task:2 :00000060 cpu:6.806046 message:388839 mqlen:0 task:2 :00000061 cpu:6.554135 message:388839 mqlen:0 task:2 :00000062 cpu:6.631871 message:388839 mqlen:0 task:2 :00000063 cpu:8.473407 message:389147 mqlen:0 task:2 :00000064 cpu:6.403462 message:388839 mqlen:0 task:2 :00000065 cpu:8.102824 message:389753 mqlen:0 task:2 :00000066 cpu:6.547498 message:388839 mqlen:0 task:2 :00000067 cpu:8.399626 message:388875 mqlen:0 task:2 :00000068 cpu:7.192537 message:388839 mqlen:0 task:2 :00000069 cpu:7.113198 message:388839 mqlen:0 task:2 :0000006a cpu:6.508983 message:389029 mqlen:0 task:2 :0000006b cpu:8.271921 message:388839 mqlen:0 task:2 :0000006c cpu:6.617448 message:388839 mqlen:0 task:2 :0000006d cpu:8.843478 message:389265 mqlen:0 task:2 :0000006e cpu:7.033075 message:389630 mqlen:0 task:2 :0000006f cpu:6.572082 message:388839 mqlen:0 task:2 :00000070 cpu:8.258566 message:388839 mqlen:0 task:2 :00000071 cpu:6.348525 message:388839 mqlen:0 task:2 :00000072 cpu:6.780269 message:388838 mqlen:0 task:2 :00000073 cpu:6.600263 message:389625 mqlen:0 task:2 :00000074 cpu:6.623408 message:388839 mqlen:0 task:2 :00000075 cpu:8.431521 message:389447 mqlen:0 task:2 :00000076 cpu:6.316694 message:389691 mqlen:0 task:2 :00000077 cpu:8.396647 message:389225 mqlen:0 task:2 :00000078 cpu:6.564499 message:388839 mqlen:0 task:2 :00000079 cpu:6.412008 message:388839 mqlen:0 task:2 :0000007a cpu:6.470043 message:389236 mqlen:0 task:2 :0000007b cpu:6.417378 message:388839 mqlen:0 task:2 :0000007c cpu:7.883303 message:388839 mqlen:0 task:2 :0000007d cpu:6.307698 message:389237 mqlen:0 task:2 :0000007e cpu:6.350322 message:388839 mqlen:0 task:2 :0000007f cpu:6.470858 message:389663 mqlen:0 task:2 :00000080 cpu:6.684873 message:389237 mqlen:0 task:2 :00000081 cpu:6.281103 message:388839 mqlen:0 task:2 :00000082 cpu:6.00146 message:388839 mqlen:0 task:2 :00000083 cpu:14.556299 message:394265 mqlen:0 task:32 :00000084 cpu:21.052253 message:849829 mqlen:0 task:5 :00000085 cpu:22.229525 message:765086 mqlen:0 task:3 :00000086 cpu:8.803616 message:388713 mqlen:0 task:1 :00000087 cpu:0.213463 message:12530 mqlen:0 task:0 :00000088 cpu:0.613353 message:12529 mqlen:0 task:0 :00000089 cpu:95.11157 message:3084149 mqlen:0 task:11 :0000008a cpu:21.589623 message:764970 mqlen:0 task:4 :0000008b cpu:21.500536 message:764994 mqlen:0 task:3 :0000008c cpu:21.567608 message:765087 mqlen:0 task:3 :0000008d cpu:7.968797 message:388713 mqlen:0 task:1 :0000008e cpu:0.191713 message:12544 mqlen:0 task:1 :0000008f cpu:0.191962 message:12526 mqlen:0 task:0 :00000090 cpu:0.191572 message:12526 mqlen:0 task:0 :00000091 cpu:0.197112 message:12526 mqlen:0 task:0 :00000092 cpu:0.184789 message:12526 mqlen:0 task:0 :00000093 cpu:0.190861 message:12526 mqlen:0 task:0 :00000094 cpu:0.190755 message:12526 mqlen:0 task:0 :00000095 cpu:0.192368 message:12526 mqlen:0 task:0 :00000096 cpu:0.191544 message:12526 mqlen:0 task:0 :00000097 cpu:0.194506 message:12526 mqlen:0 task:0 :00000098 cpu:0.19202 message:12526 mqlen:0 task:0 :00000099 cpu:0.191322 message:12526 mqlen:0 task:0 :0000009a cpu:0.187716 message:12526 mqlen:0 task:0 :0000009b cpu:0.193172 message:12526 mqlen:0 task:0 :0000009c cpu:0.197532 message:12526 mqlen:0 task:0 :0000009d cpu:0.198753 message:12526 mqlen:0 task:0 :0000009e cpu:0.197176 message:12526 mqlen:0 task:0 :0000009f cpu:24.582646 message:764977 mqlen:0 task:4 :000000a0 cpu:0.209175 message:12528 mqlen:0 task:0 :000000a1 cpu:8.06128 message:388713 mqlen:0 task:2 :000000a2 cpu:7.663776 message:388713 mqlen:0 task:2 :000000a3 cpu:7.618858 message:388713 mqlen:0 task:2 :000000a4 cpu:7.121652 message:388713 mqlen:0 task:2 :000000a5 cpu:43.973443 message:1519461 mqlen:0 task:214 :000000a6 cpu:7.06611 message:389491 mqlen:0 task:2 :000000a7 cpu:8.796759 message:390699 mqlen:0 task:2 :000000a8 cpu:7.080335 message:390685 mqlen:0 task:2 :000000a9 cpu:7.658139 message:388847 mqlen:0 task:2 :000000aa cpu:6.768521 message:388847 mqlen:0 task:2 :000000ab cpu:6.700812 message:388847 mqlen:0 task:2 :000000ac cpu:6.736969 message:389149 mqlen:0 task:2 :000000ad cpu:6.434617 message:388853 mqlen:0 task:2 :000000ae cpu:8.392101 message:388959 mqlen:0 task:2 :000000af cpu:8.278774 message:390817 mqlen:0 task:2 :000000b0 cpu:6.286713 message:388847 mqlen:0 task:2 :000000b1 cpu:7.558762 message:388847 mqlen:0 task:2 :000000b2 cpu:7.145271 message:388847 mqlen:0 task:2 :000000b3 cpu:6.928899 message:388847 mqlen:0 task:2 :000000b4 cpu:7.490952 message:388847 mqlen:0 task:2 :000000b5 cpu:8.253865 message:388847 mqlen:0 task:2 :000000b6 cpu:36.884606 message:1517611 mqlen:0 task:215 :000000b7 cpu:34.386959 message:1517545 mqlen:0 task:213 :000000b8 cpu:34.020223 message:1517552 mqlen:0 task:214 :000000b9 cpu:35.876008 message:1518634 mqlen:0 task:213 :000000ba cpu:37.942516 message:1517704 mqlen:0 task:213 :000000bb cpu:34.14391 message:1517893 mqlen:0 task:213 :000000bc cpu:33.56013 message:1520057 mqlen:0 task:214 :000000bd cpu:29.114817 message:1517557 mqlen:0 task:215 :000000be cpu:31.716974 message:1517552 mqlen:0 task:214 :000000bf cpu:32.721376 message:1517578 mqlen:0 task:216 :000000c0 cpu:34.1634 message:1517554 mqlen:0 task:214 :000000c1 cpu:31.86743 message:1517540 mqlen:0 task:213 :000000c2 cpu:32.422077 message:1517540 mqlen:0 task:213 :000000c3 cpu:35.682844 message:1518503 mqlen:0 task:213 :000000c4 cpu:35.932121 message:1519706 mqlen:0 task:213 :000000c5 cpu:0.187225 message:12530 mqlen:0 task:1 :000000c6 cpu:0.324219 message:15786 mqlen:0 task:0 :000000c7 cpu:0.226837 message:12798 mqlen:0 task:0 :000000c8 cpu:0.201828 message:12797 mqlen:0 task:1 :000000c9 cpu:0.201077 message:12531 mqlen:0 task:0 :000000ca cpu:0.208303 message:12537 mqlen:0 task:0 :000000cb cpu:0.219387 message:12523 mqlen:0 task:0 :000000cc cpu:0.199731 message:12526 mqlen:0 task:0 :000000cd cpu:0.157803 message:9718 mqlen:0 task:0 :000000ce cpu:0.171028 message:9159 mqlen:0 task:0 :000000cf cpu:0.113268 message:6822 mqlen:0 task:0 :000000d0 cpu:0.087807 message:5052 mqlen:0 task:0 :000000d1 cpu:0.004208 message:71 mqlen:0 task:1

  2. telnet下task agentservice输入如下,有玩家登录挂住了,chatLib.reqChnlMsg是向聊天服请求信息,聊天服是独立的cluster节点,经排查能正常处理业务 thread: 0x7f4f9f37a7b8 session: 1498651 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' (...tail calls...) depends/svrAddressMgr.lua:106: in function 'svrAddressMgr.getSvr' (...tail calls...) server/chat/chatLib.lua:27: in function <server/chat/chatLib.lua:26> [C]: in function 'pcall' server/chat/chatLib.lua:26: in function 'chatLib.call' server/chat/chatLib.lua:77: in function 'chatLib.reqChnlMsg' server/chat/playerchatmodule.lua:56: in function 'playerchatmodule.init_service' server/player/player.lua:386: in function 'playerbuffmodule.init_service' server/player/agentCenter.lua:229: in function <server/player/agentCenter.lua:220> [C]: in function 'xpcall' skynet/lualib/skynet/queue.lua:34: in function <skynet/lualib/skynet/queue.lua:24> (...tail calls...) [C]: in function 'xpcall' depends/serviceCenterBase.lua:108: in function 'serviceCenterBase.dispatchcmd' server/player/agentservice.lua:83: in upvalue 'f' skynet/lualib/skynet.lua:293: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fa124fe58 session: 1504452 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:293: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fa17b25c8 session: 1504453 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:293: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fa196b698 session: 1504455 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:293: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3625d8 session: 22 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3833a8 session: 90 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3865b8 session: 88 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac390598 session: 366 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac390e28 session: 134 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3912e8 session: 354 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac392d58 session: 102 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac393538 session: 142 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac394d38 session: 148 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac397398 session: 34 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac39bab8 session: 370 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac39c738 session: 372 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3a4cf8 session: 352 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3a64f8 session: 360 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3aa008 session: 356 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3ad7b8 session: 78 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3afe78 session: 368 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3b0428 session: 130 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3b20f8 session: 362 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3b2d88 session: 1504454 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:403: in upvalue 'suspend_sleep' skynet/lualib/skynet.lua:410: in function 'skynet.sleep' depends/scheduler.lua:216: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3b61e8 session: 80 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3b70c8 session: 24 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3bad38 session: 32 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3c4478 session: 358 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3c63d8 session: 374 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3c7cc8 session: 36 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3d32f8 session: 38 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3d3e88 session: 40 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3d8328 session: 42 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3dac58 session: 92 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3e1398 session: 124 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3e3b38 session: 44 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3e5ba8 session: 46 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3ef228 session: 48 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3eff28 session: 76 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3f7ac8 session: 58 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3f84b8 session: 50 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3f86b8 session: 52 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3f8928 session: 54 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3f8ae8 session: 56 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3f9e68 session: 132 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3fa128 session: 30 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac3fa3e8 session: 140 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac40d608 session: 60 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac426488 session: 16 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac42e1d8 session: 26 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac42ee68 session: 100 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4348b8 session: 364 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac439058 session: 20 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac446f58 session: 62 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac453cc8 session: 120 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4543f8 session: 18 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac456868 session: 118 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac456a88 session: 340 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac457028 session: 342 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac457d58 session: 344 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac458348 session: 74 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4588f8 session: 346 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac458c28 session: 128 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac458ed8 session: 348 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac459878 session: 98 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac45a718 session: 116 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac45c708 session: 106 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac45d5c8 session: 350 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac45e1a8 session: 114 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac45e3c8 session: 72 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac45f5d8 session: 94 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac466dd8 session: 96 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac467388 session: 86 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4676c8 session: 104 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac468018 session: 112 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac468678 session: 138 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4696f8 session: 10 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac469a88 session: 110 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac46b748 session: 12 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac46bad8 session: 126 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac46e068 session: 376 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac46ec48 session: 378 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac46f8c8 session: 380 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac46fc48 session: 28 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac471ac8 session: 108 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4725d8 session: 14 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac473898 session: 64 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4753f8 session: 82 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4797a8 session: 84 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac47ccc8 session: 66 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac47df38 session: 68 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac47ec78 session: 70 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac481b48 session: 382 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac482778 session: 384 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4833f8 session: 386 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac484078 session: 388 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac484cf8 session: 390 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac489818 session: 402 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac48a498 session: 404 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac48b118 session: 406 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac48bd98 session: 408 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac48ca18 session: 410 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac48d698 session: 412 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac48e2f8 session: 414 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac48ef88 session: 416 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac48fc38 session: 418 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4908b8 session: 420 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac490fd8 session: 422 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac49e488 session: 144 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac49f2b8 session: 146 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4aa9e8 session: 122 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4af8a8 session: 136 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4b0928 session: 200 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4b1048 session: 202 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4b1cc8 session: 204 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4b1f88 session: 270 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4b2be8 session: 272 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4b4868 session: 150 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4b54f8 session: 152 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4b6188 session: 154 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4b6e18 session: 156 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4b7aa8 session: 158 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4b8738 session: 160 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4b93c8 session: 162 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4ba058 session: 164 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4bace8 session: 166 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4bb978 session: 168 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4bc608 session: 170 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4bd298 session: 172 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4bdef8 session: 174 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4bebb8 session: 176 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4bf848 session: 178 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c04d8 session: 180 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c1168 session: 182 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c1df8 session: 184 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c2a88 session: 186 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c3718 session: 188 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c43a8 session: 190 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c5008 session: 192 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c5cc8 session: 194 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c6958 session: 196 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c75e8 session: 198 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c8298 session: 232 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c89b8 session: 234 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4c9288 session: 236 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4ca808 session: 206 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4cb498 session: 208 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4cc128 session: 210 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4ccdb8 session: 212 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4cda48 session: 214 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4ce6d8 session: 216 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4cf368 session: 218 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4cfff8 session: 220 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4d0c88 session: 222 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4d1918 session: 224 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4d25a8 session: 226 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4d3238 session: 228 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4d3ec8 session: 230 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4d4b48 session: 248 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4d5248 session: 250 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4d5b38 session: 252 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4d7108 session: 238 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4d7d98 session: 240 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4d8a28 session: 242 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4d96b8 session: 244 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4da348 session: 246 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4dafc8 session: 256 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4db6e8 session: 258 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4dbfb8 session: 260 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4dd588 session: 254 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4de118 session: 262 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4e0b18 session: 264 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4e0df8 session: 266 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4e1548 session: 268 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4e2398 session: 392 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4e3018 session: 394 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4e3c98 session: 396 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4e4918 session: 398 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4e5038 session: 400 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4e8308 session: 274 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4e8f98 session: 276 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4e9c28 session: 278 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4ea8b8 session: 280 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4eb548 session: 282 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4ec1d8 session: 284 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4ece68 session: 286 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4edaf8 session: 288 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4ee758 session: 290 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4ef418 session: 292 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f00a8 session: 294 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f0d08 session: 296 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f19c8 session: 298 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f2658 session: 300 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f32b8 session: 302 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f3f78 session: 304 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f4c08 session: 306 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f5898 session: 308 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f6528 session: 310 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f71b8 session: 312 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f7e48 session: 314 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f8ad8 session: 316 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4f9768 session: 318 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4fa3f8 session: 320 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4fb088 session: 322 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4fbd18 session: 324 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4fc9a8 session: 326 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4fd608 session: 328 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4fe2c8 session: 330 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4fef58 session: 332 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac4ffbe8 session: 334 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac500878 session: 336 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac501508 session: 338 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac502f58 session: 424 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fac504fc8 session: 426 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' skynet/lualib/skynet/sharedata.lua:16: in upvalue 'func' skynet/lualib/skynet.lua:775: in upvalue 'f' skynet/lualib/skynet.lua:266: in function <skynet/lualib/skynet.lua:265> thread: 0x7f4fad494b38 session: 1497395 stack traceback: [C]: in function 'coroutine.yield' skynet/lualib/skynet.lua:601: in upvalue 'yield_call' skynet/lualib/skynet.lua:621: in function 'skynet.call' (...tail calls...) depends/svrAddressMgr.lua:106: in function 'svrAddressMgr.getSvr' (...tail calls...) server/chat/chatLib.lua:27: in function <server/chat/chatLib.lua:26> [C]: in function 'pcall' server/chat/chatLib.lua:26: in function 'chatLib.call' server/chat/chatLib.lua:77: in function 'chatLib.reqChnlMsg' server/chat/playerchatmodule.lua:56: in function 'playerchatmodule.init_service' server/player/player.lua:386: in function 'playerbuffmodule.init_service' server/player/playermessage.lua:121: in function <server/player/playermessage.lua:116> [C]: in function 'xpcall' server/player/playermessage.lua:158: in local 'f' server/player/agentCenter.lua:597: in function 'agentCenter.handle_request' server/player/agentCenter.lua:82: in upvalue 'f' skynet/lualib/skynet.lua:293: in function <skynet/lualib/skynet.lua:265>

  3. ps -ef | grep skynet输出如下 root 32349 30791 0 11:09 pts/1 00:00:00 grep --color=auto skyent [root@ip-10-13-164-36 testService]# ps -ef | grep skynet root 11316 1 1 Mar29 ? 01:52:07 /home/slgz_aws/login/skynet/skynet bootstrap/config_login root 11372 1 1 Mar29 ? 01:53:06 /home/slgz_aws/world/skynet/skynet bootstrap/config_world root 11411 1 1 Mar29 ? 01:48:48 /home/slgz_aws/chat/skynet/skynet bootstrap/config_chat root 11819 1 3 Mar29 ? 03:11:35 /home/slgz_aws/server1/skynet/skynet bootstrap/config_game root 32351 30791 0 11:09 pts/1 00:00:00 grep --color=auto skynet

  4. pstack 查看死循环的游戏服进程, 出问题的是thread_worker, 死循环了 [root@ip-10-13-164-36 testService]# pstack 11819 Thread 12 (Thread 0x7f4fcaf1d700 (LWP 11820)):

    0 0x00007f4fcbc799fd in nanosleep () from /usr/lib64/libc.so.6

    1 0x00007f4fcbc79894 in sleep () from /usr/lib64/libc.so.6

    2 0x0000000000409bf0 in thread_monitor (p=0x1c5e9e0) at skynet-src/skynet_start.c:107

    3 0x00007f4fcc697ea5 in start_thread () from /usr/lib64/libpthread.so.0

    4 0x00007f4fcbcb2b0d in clone () from /usr/lib64/libc.so.6

    Thread 11 (Thread 0x7f4fca71c700 (LWP 11821)):

    0 0x00007f4fcbc799fd in nanosleep () from /usr/lib64/libc.so.6

    1 0x00007f4fcbcaa2d4 in usleep () from /usr/lib64/libc.so.6

    2 0x0000000000409ad9 in thread_timer (p=0x1c5e9e0) at skynet-src/skynet_start.c:138

    3 0x00007f4fcc697ea5 in start_thread () from /usr/lib64/libpthread.so.0

    4 0x00007f4fcbcb2b0d in clone () from /usr/lib64/libc.so.6

    Thread 10 (Thread 0x7f4fc9f1b700 (LWP 11822)):

    0 0x00007f4fcbcb30e3 in epoll_wait () from /usr/lib64/libc.so.6

    1 0x000000000040d07c in sp_wait (max=64, e=0x7f4fcb123050, efd=) at skynet-src/socket_epoll.h:58

    2 socket_server_poll (ss=0x7f4fcb123010, result=result@entry=0x7f4fc9f1aed0, more=more@entry=0x7f4fc9f1aecc) at skynet-src/socket_server.c:1716

    3 0x000000000040ae6b in skynet_socket_poll () at skynet-src/skynet_socket.c:84

    4 0x0000000000409a57 in thread_socket (p=0x1c5e9e0) at skynet-src/skynet_start.c:69

    5 0x00007f4fcc697ea5 in start_thread () from /usr/lib64/libpthread.so.0

    6 0x00007f4fcbcb2b0d in clone () from /usr/lib64/libc.so.6

    Thread 9 (Thread 0x7f4fc971a700 (LWP 11823)):

    0 0x00007f4fcc69ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0

    1 0x0000000000409a1c in thread_worker (p=) at skynet-src/skynet_start.c:171

    2 0x00007f4fcc697ea5 in start_thread () from /usr/lib64/libpthread.so.0

    3 0x00007f4fcbcb2b0d in clone () from /usr/lib64/libc.so.6

    Thread 8 (Thread 0x7f4fc8f19700 (LWP 11824)):

    0 0x00007f4fcc69ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0

    1 0x0000000000409a1c in thread_worker (p=) at skynet-src/skynet_start.c:171

    2 0x00007f4fcc697ea5 in start_thread () from /usr/lib64/libpthread.so.0

    3 0x00007f4fcbcb2b0d in clone () from /usr/lib64/libc.so.6

    Thread 7 (Thread 0x7f4fc0718700 (LWP 11825)):

    0 0x00007f4fcc69ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0

    1 0x0000000000409a1c in thread_worker (p=) at skynet-src/skynet_start.c:171

    2 0x00007f4fcc697ea5 in start_thread () from /usr/lib64/libpthread.so.0

    3 0x00007f4fcbcb2b0d in clone () from /usr/lib64/libc.so.6

    Thread 6 (Thread 0x7f4fc3fff700 (LWP 11826)):

    0 0x00007f4fcc69ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0

    1 0x0000000000409a1c in thread_worker (p=) at skynet-src/skynet_start.c:171

    2 0x00007f4fcc697ea5 in start_thread () from /usr/lib64/libpthread.so.0

    3 0x00007f4fcbcb2b0d in clone () from /usr/lib64/libc.so.6

    Thread 5 (Thread 0x7f4fc37fe700 (LWP 11827)):

    0 0x00007f4fcc69ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0

    1 0x0000000000409a1c in thread_worker (p=) at skynet-src/skynet_start.c:171

    2 0x00007f4fcc697ea5 in start_thread () from /usr/lib64/libpthread.so.0

    3 0x00007f4fcbcb2b0d in clone () from /usr/lib64/libc.so.6

    Thread 4 (Thread 0x7f4fc2ffd700 (LWP 11828)):

    0 0x0000000000407783 in skynet_handle_grab (handle=handle@entry=12) at skynet-src/skynet_handle.c:153

    1 0x0000000000409173 in skynet_context_message_dispatch (sm=sm@entry=0x1c5eb50, q=q@entry=0x7f4fb8234a10, weight=weight@entry=0) at skynet-src/skynet_server.c:308

    2 0x000000000040999d in thread_worker (p=) at skynet-src/skynet_start.c:164

    3 0x00007f4fcc697ea5 in start_thread () from /usr/lib64/libpthread.so.0

    4 0x00007f4fcbcb2b0d in clone () from /usr/lib64/libc.so.6

    Thread 3 (Thread 0x7f4fc27fc700 (LWP 11829)):

    0 0x00007f4fcc69ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0

    1 0x0000000000409a1c in thread_worker (p=) at skynet-src/skynet_start.c:171

    2 0x00007f4fcc697ea5 in start_thread () from /usr/lib64/libpthread.so.0

    3 0x00007f4fcbcb2b0d in clone () from /usr/lib64/libc.so.6

    Thread 2 (Thread 0x7f4fc1ffb700 (LWP 11830)):

    0 0x00007f4fcc69ba35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0

    1 0x0000000000409a1c in thread_worker (p=) at skynet-src/skynet_start.c:171

    2 0x00007f4fcc697ea5 in start_thread () from /usr/lib64/libpthread.so.0

    3 0x00007f4fcbcb2b0d in clone () from /usr/lib64/libc.so.6

    Thread 1 (Thread 0x7f4fccabe740 (LWP 11819)):

    0 0x00007f4fcc699017 in pthread_join () from /usr/lib64/libpthread.so.0

    1 0x000000000040a0b9 in start (thread=8) at skynet-src/skynet_start.c:228

    2 skynet_start (config=config@entry=0x7ffca3358d80) at skynet-src/skynet_start.c:345

    3 0x0000000000407185 in main (argc=, argv=) at skynet-src/skynet_main.c:166

    [root@ip-10-13-164-36 testService]# ^C [root@ip-10-13-164-36 testService]# ^C [root@ip-10-13-164-36 testService]#

  5. top查看进程信息,游戏服死循环,如下:pid=11819 cup使用97% top - 11:15:59 up 5 days, 9:12, 4 users, load average: 0.96, 0.94, 0.95 Tasks: 137 total, 2 running, 134 sleeping, 0 stopped, 0 zombie %Cpu(s): 26.2 us, 0.8 sy, 0.0 ni, 72.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.2 st KiB Mem : 16260872 total, 12014828 free, 3089836 used, 1156208 buff/cache KiB Swap: 8388604 total, 8388604 free, 0 used. 12862844 avail Mem

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    11819 root 20 0 1656436 1.0g 6376 t 97.7 6.4 197:27.03 /home/slgz_aws/server1/skynet/skynet bootstrap/config_game
    427 root 20 0 246340 32780 10048 R 8.3 0.2 0:00.25 /usr/bin/gdb --quiet -nx /proc/11819/exe 11819
    11316 root 20 0 647604 145208 5260 S 1.7 0.9 112:14.38 /home/slgz_aws/login/skynet/skynet bootstrap/config_login
    11372 root 20 0 488976 143992 1180 S 1.7 0.9 113:13.47 /home/slgz_aws/world/skynet/skynet bootstrap/config_world
    11411 root 20 0 645544 137588 5244 S 1.7 0.8 108:54.37 /home/slgz_aws/chat/skynet/skynet bootstrap/config_chat
    399 root 20 0 162132 2332 1592 R 0.3 0.0 0:00.02 top
    1 root 20 0 191180 4060 2600 S 0.0 0.0 0:07.96 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
    2 root 20 0 0 0 0 S 0.0 0.0 0:00.05 [kthreadd]
    4 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kworker/0:0H]
    5 root 20 0 0 0 0 S 0.0 0.0 0:01.95 [kworker/u30:0]
    6 root 20 0 0 0 0 S 0.0 0.0 0:00.09 [ksoftirqd/0]
    7 root rt 0 0 0 0 S 0.0 0.0 0:00.25 [migration/0]
    8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [rcu_bh]
    9 root 20 0 0 0 0 S 0.0 0.0 1:37.17 [rcu_sched]
    10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [lru-add-drain]
    11 root rt 0 0 0 0 S 0.0 0.0 0:01.95 [watchdog/0]
    12 root rt 0 0 0 0 S 0.0 0.0 0:01.56 [watchdog/1]
    13 root rt 0 0 0 0 S 0.0 0.0 0:00.20 [migration/1]
    14 root 20 0 0 0 0 S 0.0 0.0 0:00.07 [ksoftirqd/1]
    16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kworker/1:0H]
    17 root rt 0 0 0 0 S 0.0 0.0 0:01.52 [watchdog/2]
    18 root rt 0 0 0 0 S 0.0 0.0 0:00.28 [migration/2]
    19 root 20 0 0 0 0 S 0.0 0.0 0:00.07 [ksoftirqd/2]
    21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kworker/2:0H]
    22 root rt 0 0 0 0 S 0.0 0.0 0:01.52 [watchdog/3]
    23 root rt 0 0 0 0 S 0.0 0.0 0:00.26 [migration/3]
    24 root 20 0 0 0 0 S 0.0 0.0 0:00.08 [ksoftirqd/3]
    26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kworker/3:0H]
    28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [kdevtmpfs]
    29 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [netns]
    30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [xenwatch]
    31 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [xenbus]
    33 root 20 0 0 0 0 S 0.0 0.0 0:00.23 [khungtaskd]
    34 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [writeback]
    35 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kintegrityd]
    36 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [bioset]
    37 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [bioset]
    38 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [bioset]
    39 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kblockd]
    41 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [md]
    42 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [edac-poller]
    43 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [watchdogd]
    50 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [kswapd0]
    51 root 25 5 0 0 0 S 0.0 0.0 0:00.00 [ksmd]
    52 root 39 19 0 0 0 S 0.0 0.0 0:01.39 [khugepaged]
    53 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [crypto]
    61 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kthrotld]
    62 root 20 0 0 0 0 S 0.0 0.0 0:00.01 [kworker/u30:1]
    63 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kmpath_rdacd]
    64 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kaluad]
    65 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kpsmoused]
    [root@ip-10-13-164-36 server1]#

  6. free查看内存使用情况,内存足够

[root@ip-10-13-164-36 server1]# free -h total used free shared buff/cache available Mem: 15G 2.9G 11G 24M 1.1G 12G Swap: 8.0G 0B 8.0G

gels123 commented 2 years ago

外网又炸锅了,求大神帮忙看看哈

gels123 commented 2 years ago

登录服正常,聊天服正常,游戏服连聊天服不行,死循环了。

gels123 commented 2 years ago

我印象中,skynet中是有提供这方面的支持的(消息队列的包数量就是其中一个)。 你可以利用这一点,做出自己的应对或者记录,另外你也不要只看特定service的,可以统一加上。

嗯,做了一些记录

gels123 commented 2 years ago

补充一下,list信息如下: list :00000004 snlua cdummy :00000006 snlua datacenterd :00000007 snlua service_mgr :00000008 snlua service_provider :00000009 snlua service_cell ltls_holder :0000000b snlua sharedatad :0000000c snlua clusterd :0000000d snlua configd :0000000e snlua gate :0000000f snlua dbservice :00000010 snlua redisService master 4 :00000011 snlua redisService sub 1 :00000012 snlua redisService sub 2 :00000013 snlua redisService sub 3 :00000014 snlua redisService sub 4 :00000015 snlua redisService master 4 :00000016 snlua redisService sub 1 :00000017 snlua redisService sub 2 :00000018 snlua redisService sub 3 :00000019 snlua redisService sub 4 :0000001a snlua interactionhubd :0000001b snlua clustersender world_1 ip-10-13-164-36.us-west-2.compute.internal796 127.0.0.1 9016 :0000001c snlua logService 1 :0000001d snlua multicastd :0000001e snlua debug_console 10014 :0000001f snlua protod :00000020 snlua alertService 1 :00000021 snlua cacheservice 1 :00000022 snlua dbslave global 0 :00000023 snlua dbslave global 1 :00000024 snlua dbslave global 2 :00000025 snlua dbslave global 3 :00000026 snlua dbslave global 4 :00000027 snlua dbslave global 5 :00000028 snlua dbslave global 6 :00000029 snlua dbslave global 7 :0000002a snlua dbslave global 8 :0000002b snlua dbslave global 9 :0000002c snlua dbslave global 10 :0000002d snlua dbslave global 11 :0000002e snlua dbslave global 12 :0000002f snlua dbslave global 13 :00000030 snlua dbslave global 14 :00000031 snlua dbslave global 15 :00000032 snlua dbslave web :00000033 snlua dbslave player 0 0 :00000034 snlua dbslave player 1 0 :00000035 snlua dbslave player 2 0 :00000036 snlua dbslave player 3 0 :00000037 snlua dbslave player 4 0 :00000038 snlua dbslave player 5 0 :00000039 snlua dbslave player 6 0 :0000003a snlua dbslave player 7 0 :0000003b snlua dbslave player 8 0 :0000003c snlua dbslave player 9 0 :0000003d snlua dbslave player 3 1 :0000003e snlua dbslave player 0 1 :0000003f snlua dbslave player 5 1 :00000040 snlua dbslave player 2 1 :00000041 snlua dbslave player 9 1 :00000042 snlua dbslave player 6 1 :00000043 snlua dbslave player 4 1 :00000044 snlua dbslave player 1 1 :00000045 snlua dbslave player 7 1 :00000046 snlua dbslave player 8 1 :00000047 snlua dbslave player 3 2 :00000048 snlua dbslave player 2 2 :00000049 snlua dbslave player 5 2 :0000004a snlua dbslave player 9 2 :0000004b snlua dbslave player 4 2 :0000004c snlua dbslave player 0 2 :0000004d snlua dbslave player 1 2 :0000004e snlua dbslave player 8 2 :0000004f snlua dbslave player 6 2 :00000050 snlua dbslave player 2 3 :00000051 snlua dbslave player 7 2 :00000052 snlua dbslave player 1 3 :00000053 snlua dbslave player 3 3 :00000054 snlua dbslave player 5 3 :00000055 snlua dbslave player 9 3 :00000056 snlua dbslave player 4 3 :00000057 snlua dbslave player 7 3 :00000058 snlua dbslave player 1 4 :00000059 snlua dbslave player 0 3 :0000005a snlua dbslave player 8 3 :0000005b snlua dbslave player 3 4 :0000005c snlua dbslave player 2 4 :0000005d snlua dbslave player 5 4 :0000005e snlua dbslave player 6 3 :0000005f snlua dbslave player 4 4 :00000060 snlua dbslave player 1 5 :00000061 snlua dbslave player 9 4 :00000062 snlua dbslave player 7 4 :00000063 snlua dbslave player 3 5 :00000064 snlua dbslave player 2 5 :00000065 snlua dbslave player 8 4 :00000066 snlua dbslave player 0 4 :00000067 snlua dbslave player 6 4 :00000068 snlua dbslave player 4 5 :00000069 snlua dbslave player 7 5 :0000006a snlua dbslave player 9 5 :0000006b snlua dbslave player 1 6 :0000006c snlua dbslave player 5 5 :0000006d snlua dbslave player 8 5 :0000006e snlua dbslave player 6 5 :0000006f snlua dbslave player 3 6 :00000070 snlua dbslave player 9 6 :00000071 snlua dbslave player 2 6 :00000072 snlua dbslave player 0 5 :00000073 snlua dbslave player 4 6 :00000074 snlua dbslave player 5 6 :00000075 snlua dbslave player 1 7 :00000076 snlua dbslave player 8 6 :00000077 snlua dbslave player 7 6 :00000078 snlua dbslave player 6 6 :00000079 snlua dbslave player 9 7 :0000007a snlua dbslave player 2 7 :0000007b snlua dbslave player 4 7 :0000007c snlua dbslave player 3 7 :0000007d snlua dbslave player 5 7 :0000007e snlua dbslave player 0 6 :0000007f snlua dbslave player 8 7 :00000080 snlua dbslave player 7 7 :00000081 snlua dbslave player 6 7 :00000082 snlua dbslave player 0 7 :00000083 snlua gateservice :00000084 snlua publicService 1 :00000085 snlua mailService 1 :00000086 snlua battleService 1 :00000087 snlua elasticSearchService :00000088 snlua searchService 1 1 :00000089 snlua worldMapService 1 :0000008a snlua queueService 1 :0000008b snlua arenaService 1 :0000008c snlua guildservice :0000008d snlua rankService 1 :0000008e snlua gmweb :0000008f snlua gmweb agent :00000090 snlua gmweb agent :00000091 snlua gmweb agent :00000092 snlua gmweb agent :00000093 snlua gmweb agent :00000094 snlua gmweb agent :00000095 snlua gmweb agent :00000096 snlua gmweb agent :00000097 snlua gmweb agent :00000098 snlua gmweb agent :00000099 snlua gmweb agent :0000009a snlua gmweb agent :0000009b snlua gmweb agent :0000009c snlua gmweb agent :0000009d snlua gmweb agent :0000009e snlua gmweb agent :0000009f snlua gmserver :000000a0 snlua gateBattleService 1 RUDP :000000a1 snlua battleFrameService 1 :000000a2 snlua battleFrameService 2 :000000a3 snlua battleFrameService 3 :000000a4 snlua battleFrameService 4 :000000a5 snlua agentservice 1 1 131 :000000a6 snlua dbslave gamelog 0 :000000a7 snlua dbslave gamelog 1 :000000a8 snlua dbslave gamelog 2 :000000a9 snlua dbslave gamelog 3 :000000aa snlua dbslave gamelog 4 :000000ab snlua dbslave gamelog 5 :000000ac snlua dbslave gamelog 6 :000000ad snlua dbslave gamelog 7 :000000ae snlua dbslave gamelog 8 :000000af snlua dbslave gamelog 9 :000000b0 snlua dbslave gamelog 10 :000000b1 snlua dbslave gamelog 11 :000000b2 snlua dbslave gamelog 12 :000000b3 snlua dbslave gamelog 13 :000000b4 snlua dbslave gamelog 14 :000000b5 snlua dbslave gamelog 15 :000000b6 snlua agentservice 1 2 131 :000000b7 snlua agentservice 1 3 131 :000000b8 snlua agentservice 1 4 131 :000000b9 snlua agentservice 1 5 131 :000000ba snlua agentservice 1 6 131 :000000bb snlua agentservice 1 7 131 :000000bc snlua agentservice 1 8 131 :000000bd snlua agentservice 1 9 131 :000000be snlua agentservice 1 10 131 :000000bf snlua agentservice 1 11 131 :000000c0 snlua agentservice 1 12 131 :000000c1 snlua agentservice 1 13 131 :000000c2 snlua agentservice 1 14 131 :000000c3 snlua agentservice 1 15 131 :000000c4 snlua agentservice 1 16 131 :000000c5 snlua clustersender login_1 ip-10-13-164-36.us-west-2.compute.internal796 10.13.164.36 9011 :000000c6 snlua clusteragent 12 14 130 :000000c7 snlua clusterproxy chat_1 .chatSvr@1@1 :000000c8 snlua clustersender chat_1 ip-10-13-164-36.us-west-2.compute.internal796 10.13.164.36 9012 :000000c9 snlua clusterproxy world_1 .activitySvr@1@11 :000000ca snlua clusterproxy world_1 .activitySvr@1@10

JieTrancender commented 2 years ago

建议把登录时候聊天可能卡住的地方先异步出去,不要卡住登录。不能登录和不能聊天差别很大。

JieTrancender commented 2 years ago

如果说是死循环,看看日志有没有endless,如果有的话向该服务发送0信号,signal addr 0

cloudwu commented 2 years ago
  1. markdown 应该用 ```` 引用文本, 如
    log
  2. 建议花五分钟先学习一下 markdown , 再考虑解决其他问题。
  3. 既然是 clusterd 无反应,可以用 signal c 0 试试中断。
  4. 查问题前,应保证代码更新到最新,避免浪费更多人的时间。我怀疑是 https://github.com/cloudwu/skynet/issues/1346
gels123 commented 2 years ago

不要卡住登录。不能登录和不能聊天差别很大。

有的,有每60秒检测endless loop的逻辑,有TIMEOUT就发送 signal addr 0,后面按云风大大的先更新到1.5.0版本,再验证一下~

gels123 commented 2 years ago

感谢大家!!

yuanfengyun commented 2 years ago

用strace跑一下,看看是在调用哪些函数。应该就能知道程序在干啥了。

terry8210 commented 2 years ago

好几天没报问题上来了哦,估计更新版本后解决了BUG了~

gels123 commented 2 years ago

用strace跑一下,看看是在调用哪些函数。应该就能知道程序在干啥了。

感谢哈~~,目前外服已经更新到skynet1.5.0版本了,暂时没复现问题。