machinezone / IXWebSocket

websocket and http client and server library, with TLS support and very few dependencies
BSD 3-Clause "New" or "Revised" License
539 stars 173 forks source link

Receiving message rate drops dramatically after a while #458

Closed cvoica closed 1 year ago

cvoica commented 1 year ago

While using the library I observe the below behavior: the client application, which is only counting the messages and displays the below output every second, is able to receive messages at a high rate for several seconds. This period is sometimes longer than a few seconds seen below. Then the rate drops by a lot, couple of hundreds KB/s.

In the server I have a timeout and I disconnect a blocked reader after 20s. The disconnection from the server side occurs in at 15:19:43. As you can see below the client keeps displaying that it gets message. This means that it reads from memory which I did investigated and got to suspect the handling you have on the _chunks list.

I was then thinking that it might benefit me and your time to look into this together.

24-Apr-2023 15:18:38  : Connection established
24-Apr-2023 15:18:38  : Subscription messages sent
24-Apr-2023 15:18:39  : Message rate:       0 msg/sec ... 0 KB/s bytes:0 cnt:0
24-Apr-2023 15:18:40  : Message rate:  114236 msg/sec ... 55023 KB/s bytes:56343892 cnt:114351
24-Apr-2023 15:18:41  : Message rate:  117532 msg/sec ... 56977 KB/s bytes:58345086 cnt:232001
24-Apr-2023 15:18:42  : Message rate:  121814 msg/sec ... 62665 KB/s bytes:64169120 cnt:353937
24-Apr-2023 15:18:43  : Message rate:  116371 msg/sec ... 56426 KB/s bytes:57780433 cnt:470425
24-Apr-2023 15:18:44  : Message rate:  116131 msg/sec ... 55265 KB/s bytes:56592014 cnt:586673
24-Apr-2023 15:18:45  : Message rate:  117856 msg/sec ... 55106 KB/s bytes:56428935 cnt:705001
24-Apr-2023 15:18:46  : Message rate:  119643 msg/sec ... 55836 KB/s bytes:57176161 cnt:824764
24-Apr-2023 15:18:47  : Message rate:  118653 msg/sec ... 56286 KB/s bytes:57637744 cnt:943536
24-Apr-2023 15:18:48  : Message rate:   79072 msg/sec ... 41194 KB/s bytes:42183523 cnt:1022688
24-Apr-2023 15:18:49  : Message rate:    2230 msg/sec ... 1078 KB/s bytes:1104174 cnt:1024921
24-Apr-2023 15:18:50  : Message rate:    2480 msg/sec ... 1455 KB/s bytes:1490256 cnt:1027404
24-Apr-2023 15:18:51  : Message rate:    2881 msg/sec ... 1780 KB/s bytes:1822929 cnt:1030288
24-Apr-2023 15:18:52  : Message rate:    3495 msg/sec ... 2166 KB/s bytes:2218347 cnt:1033787
24-Apr-2023 15:18:53  : Message rate:    4815 msg/sec ... 2431 KB/s bytes:2489379 cnt:1038607
24-Apr-2023 15:18:54  : Message rate:   11545 msg/sec ... 5308 KB/s bytes:5435661 cnt:1050164
24-Apr-2023 15:18:55  : Message rate:    4121 msg/sec ... 1689 KB/s bytes:1730458 cnt:1054290
24-Apr-2023 15:18:56  : Message rate:    5936 msg/sec ... 2723 KB/s bytes:2788847 cnt:1060232
24-Apr-2023 15:18:57  : Message rate:   10736 msg/sec ... 5603 KB/s bytes:5738238 cnt:1070979
24-Apr-2023 15:18:58  : Message rate:   11197 msg/sec ... 5364 KB/s bytes:5493704 cnt:1082188
24-Apr-2023 15:18:59  : Message rate:    5854 msg/sec ... 2804 KB/s bytes:2872296 cnt:1088048
24-Apr-2023 15:19:00  : Message rate:   10219 msg/sec ... 4667 KB/s bytes:4780004 cnt:1098278
24-Apr-2023 15:19:01  : Message rate:    4428 msg/sec ... 2237 KB/s bytes:2291048 cnt:1102711
24-Apr-2023 15:19:02  : Message rate:    8396 msg/sec ... 4276 KB/s bytes:4379606 cnt:1111116
24-Apr-2023 15:19:03  : Message rate:    7404 msg/sec ... 3838 KB/s bytes:3930576 cnt:1118528
24-Apr-2023 15:19:04  : Message rate:    7060 msg/sec ... 3571 KB/s bytes:3656712 cnt:1125596
24-Apr-2023 15:19:05  : Message rate:    8460 msg/sec ... 3981 KB/s bytes:4077077 cnt:1134065
24-Apr-2023 15:19:06  : Message rate:    5463 msg/sec ... 2915 KB/s bytes:2985611 cnt:1139534
24-Apr-2023 15:19:07  : Message rate:   10898 msg/sec ... 5150 KB/s bytes:5274174 cnt:1150443
24-Apr-2023 15:19:08  : Message rate:    6652 msg/sec ... 3098 KB/s bytes:3173229 cnt:1157102
24-Apr-2023 15:19:09  : Message rate:    9913 msg/sec ... 4542 KB/s bytes:4651586 cnt:1167025
24-Apr-2023 15:19:10  : Message rate:    5802 msg/sec ... 2546 KB/s bytes:2607683 cnt:1172833
24-Apr-2023 15:19:11  : Message rate:   10882 msg/sec ... 5291 KB/s bytes:5418430 cnt:1183726
24-Apr-2023 15:19:12  : Message rate:    7551 msg/sec ... 3242 KB/s bytes:3320200 cnt:1191285
24-Apr-2023 15:19:13  : Message rate:    8644 msg/sec ... 4497 KB/s bytes:4605168 cnt:1199938
24-Apr-2023 15:19:14  : Message rate:    8396 msg/sec ... 3766 KB/s bytes:3856406 cnt:1208343
24-Apr-2023 15:19:15  : Message rate:    8065 msg/sec ... 3843 KB/s bytes:3935485 cnt:1216417
24-Apr-2023 15:19:16  : Message rate:   10787 msg/sec ... 5822 KB/s bytes:5961935 cnt:1227215
24-Apr-2023 15:19:17  : Message rate:    5236 msg/sec ... 2776 KB/s bytes:2843394 cnt:1232457
24-Apr-2023 15:19:18  : Message rate:   10010 msg/sec ... 5560 KB/s bytes:5693614 cnt:1242478
24-Apr-2023 15:19:19  : Message rate:    5368 msg/sec ... 2456 KB/s bytes:2515512 cnt:1247852
24-Apr-2023 15:19:22  : Message rate:    3081 msg/sec ... 4438 KB/s bytes:4545234 cnt:1256539
24-Apr-2023 15:19:23  : Message rate:      65 msg/sec ... 28 KB/s bytes:28691 cnt:1256605
24-Apr-2023 15:19:24  : Message rate:      67 msg/sec ... 37 KB/s bytes:38143 cnt:1256673
24-Apr-2023 15:19:25  : Message rate:      65 msg/sec ... 22 KB/s bytes:22725 cnt:1256739
24-Apr-2023 15:19:26  : Message rate:      67 msg/sec ... 24 KB/s bytes:25335 cnt:1256807
24-Apr-2023 15:19:27  : Message rate:      67 msg/sec ... 35 KB/s bytes:36104 cnt:1256875
24-Apr-2023 15:19:28  : Message rate:      66 msg/sec ... 27 KB/s bytes:28314 cnt:1256942
24-Apr-2023 15:19:29  : Message rate:      67 msg/sec ... 21 KB/s bytes:22165 cnt:1257010
24-Apr-2023 15:19:30  : Message rate:      67 msg/sec ... 23 KB/s bytes:23827 cnt:1257078
24-Apr-2023 15:19:31  : Message rate:      66 msg/sec ... 28 KB/s bytes:29593 cnt:1257145
24-Apr-2023 15:19:32  : Message rate:      67 msg/sec ... 31 KB/s bytes:32030 cnt:1257213
24-Apr-2023 15:19:33  : Message rate:      67 msg/sec ... 31 KB/s bytes:32758 cnt:1257281
24-Apr-2023 15:19:34  : Message rate:      66 msg/sec ... 23 KB/s bytes:23606 cnt:1257348
24-Apr-2023 15:19:35  : Message rate:      67 msg/sec ... 26 KB/s bytes:26963 cnt:1257416
24-Apr-2023 15:19:36  : Message rate:      67 msg/sec ... 28 KB/s bytes:28879 cnt:1257484
24-Apr-2023 15:19:37  : Message rate:      67 msg/sec ... 27 KB/s bytes:28224 cnt:1257552
24-Apr-2023 15:19:38  : Message rate:      67 msg/sec ... 23 KB/s bytes:23718 cnt:1257620
24-Apr-2023 15:19:39  : Message rate:      66 msg/sec ... 26 KB/s bytes:26781 cnt:1257687
24-Apr-2023 15:19:40  : Message rate:      67 msg/sec ... 26 KB/s bytes:27584 cnt:1257755
24-Apr-2023 15:19:41  : Message rate:      67 msg/sec ... 26 KB/s bytes:27268 cnt:1257823
24-Apr-2023 15:19:42  : Message rate:      67 msg/sec ... 36 KB/s bytes:36908 cnt:1257891
24-Apr-2023 15:19:43  : Message rate:      66 msg/sec ... 52 KB/s bytes:53295 cnt:1257958
24-Apr-2023 15:19:44  : Message rate:      67 msg/sec ... 25 KB/s bytes:26022 cnt:1258026
24-Apr-2023 15:19:45  : Message rate:      67 msg/sec ... 23 KB/s bytes:24317 cnt:1258094
24-Apr-2023 15:19:46  : Message rate:      66 msg/sec ... 32 KB/s bytes:33382 cnt:1258161
24-Apr-2023 15:19:47  : Message rate:      67 msg/sec ... 24 KB/s bytes:24670 cnt:1258229
24-Apr-2023 15:19:48  : Message rate:      67 msg/sec ... 41 KB/s bytes:42268 cnt:1258297
24-Apr-2023 15:19:49  : Message rate:      67 msg/sec ... 39 KB/s bytes:40169 cnt:1258365
24-Apr-2023 15:19:50  : Message rate:      67 msg/sec ... 31 KB/s bytes:32163 cnt:1258433
24-Apr-2023 15:19:51  : Message rate:      67 msg/sec ... 35 KB/s bytes:36483 cnt:1258501
24-Apr-2023 15:19:52  : Message rate:      66 msg/sec ... 39 KB/s bytes:40880 cnt:1258568
24-Apr-2023 15:19:53  : Message rate:      68 msg/sec ... 64 KB/s bytes:65667 cnt:1258637
24-Apr-2023 15:19:54  : Message rate:      66 msg/sec ... 45 KB/s bytes:46221 cnt:1258704
24-Apr-2023 15:19:55  : Message rate:      67 msg/sec ... 32 KB/s bytes:33504 cnt:1258772
24-Apr-2023 15:19:56  : Message rate:      67 msg/sec ... 45 KB/s bytes:46790 cnt:1258840
24-Apr-2023 15:19:57  : Message rate:      67 msg/sec ... 25 KB/s bytes:26352 cnt:1258908
24-Apr-2023 15:19:58  : Message rate:      67 msg/sec ... 34 KB/s bytes:35804 cnt:1258976
24-Apr-2023 15:19:59  : Message rate:      66 msg/sec ... 28 KB/s bytes:29184 cnt:1259043
24-Apr-2023 15:20:00  : Message rate:      66 msg/sec ... 32 KB/s bytes:33044 cnt:1259111
24-Apr-2023 15:20:01  : Message rate:      66 msg/sec ... 19 KB/s bytes:20458 cnt:1259178
24-Apr-2023 15:20:02  : Message rate:      66 msg/sec ... 25 KB/s bytes:26246 cnt:1259246
24-Apr-2023 15:20:03  : Message rate:      68 msg/sec ... 50 KB/s bytes:51901 cnt:1259315
24-Apr-2023 15:20:05  : Message rate:      67 msg/sec ... 39 KB/s bytes:40032 cnt:1259383
24-Apr-2023 15:20:06  : Message rate:      67 msg/sec ... 76 KB/s bytes:77911 cnt:1259451
24-Apr-2023 15:20:07  : Message rate:      68 msg/sec ... 69 KB/s bytes:71647 cnt:1259520
24-Apr-2023 15:20:08  : Message rate:      67 msg/sec ... 58 KB/s bytes:59442 cnt:1259589
24-Apr-2023 15:20:09  : Message rate:      67 msg/sec ... 46 KB/s bytes:47563 cnt:1259657
24-Apr-2023 15:20:10  : Message rate:      66 msg/sec ... 39 KB/s bytes:40469 cnt:1259724
24-Apr-2023 15:20:11  : Message rate:      68 msg/sec ... 30 KB/s bytes:31245 cnt:1259793
24-Apr-2023 15:20:12  : Message rate:      67 msg/sec ... 54 KB/s bytes:55484 cnt:1259861
24-Apr-2023 15:20:13  : Message rate:      67 msg/sec ... 54 KB/s bytes:56310 cnt:1259929
24-Apr-2023 15:20:14  : Message rate:      67 msg/sec ... 37 KB/s bytes:38580 cnt:1259997
24-Apr-2023 15:20:15  : Message rate:      66 msg/sec ... 30 KB/s bytes:31126 cnt:1260064
24-Apr-2023 15:20:16  : Message rate:      66 msg/sec ... 40 KB/s bytes:41771 cnt:1260131
24-Apr-2023 15:20:17  : Message rate:      66 msg/sec ... 40 KB/s bytes:41470 cnt:1260198
24-Apr-2023 15:20:18  : Message rate:      67 msg/sec ... 45 KB/s bytes:46473 cnt:1260266
24-Apr-2023 15:20:19  : Message rate:      67 msg/sec ... 75 KB/s bytes:77276 cnt:1260334
24-Apr-2023 15:20:20  : Message rate:      66 msg/sec ... 30 KB/s bytes:31711 cnt:1260401
24-Apr-2023 15:20:21  : Message rate:      68 msg/sec ... 51 KB/s bytes:52327 cnt:1260470
24-Apr-2023 15:20:22  : Message rate:      68 msg/sec ... 73 KB/s bytes:75633 cnt:1260539
24-Apr-2023 15:20:23  : Message rate:      66 msg/sec ... 56 KB/s bytes:57958 cnt:1260607
24-Apr-2023 15:20:24  : Message rate:      66 msg/sec ... 33 KB/s bytes:34729 cnt:1260674
24-Apr-2023 15:20:25  : Message rate:      67 msg/sec ... 37 KB/s bytes:38064 cnt:1260742
24-Apr-2023 15:20:26  : Message rate:      69 msg/sec ... 32 KB/s bytes:33520 cnt:1260812
24-Apr-2023 15:20:27  : Message rate:      68 msg/sec ... 47 KB/s bytes:48720 cnt:1260881
24-Apr-2023 15:20:28  : Message rate:      66 msg/sec ... 58 KB/s bytes:59569 cnt:1260948
24-Apr-2023 15:20:29  : Message rate:      67 msg/sec ... 47 KB/s bytes:48525 cnt:1261016
24-Apr-2023 15:20:30  : Message rate:      68 msg/sec ... 56 KB/s bytes:57617 cnt:1261085
24-Apr-2023 15:20:31  : Message rate:      67 msg/sec ... 49 KB/s bytes:50719 cnt:1261153
24-Apr-2023 15:20:32  : Message rate:      67 msg/sec ... 46 KB/s bytes:47860 cnt:1261221
24-Apr-2023 15:20:33  : Message rate:      67 msg/sec ... 35 KB/s bytes:36646 cnt:1261289
24-Apr-2023 15:20:34  : Message rate:      67 msg/sec ... 30 KB/s bytes:30803 cnt:1261357
24-Apr-2023 15:20:35  : Message rate:      67 msg/sec ... 25 KB/s bytes:25764 cnt:1261425
24-Apr-2023 15:20:36  : Message rate:      67 msg/sec ... 23 KB/s bytes:23799 cnt:1261493
24-Apr-2023 15:20:37  : Message rate:      69 msg/sec ... 26 KB/s bytes:26648 cnt:1261563
24-Apr-2023 15:20:38  : Message rate:      68 msg/sec ... 66 KB/s bytes:67788 cnt:1261632
24-Apr-2023 15:20:39  : Message rate:      66 msg/sec ... 39 KB/s bytes:40823 cnt:1261700
24-Apr-2023 15:20:40  : Message rate:      67 msg/sec ... 33 KB/s bytes:34155 cnt:1261768
24-Apr-2023 15:20:41  : Message rate:      68 msg/sec ... 44 KB/s bytes:45501 cnt:1261837
24-Apr-2023 15:20:42  : Message rate:      68 msg/sec ... 47 KB/s bytes:48255 cnt:1261906
24-Apr-2023 15:20:43  : Message rate:      68 msg/sec ... 32 KB/s bytes:33589 cnt:1261975
24-Apr-2023 15:20:44  : Message rate:      68 msg/sec ... 20 KB/s bytes:21384 cnt:1262044
24-Apr-2023 15:20:45  : Message rate:      67 msg/sec ... 23 KB/s bytes:24427 cnt:1262112
24-Apr-2023 15:20:46  : Message rate:      67 msg/sec ... 17 KB/s bytes:17448 cnt:1262180
24-Apr-2023 15:20:47  : Message rate:      67 msg/sec ... 16 KB/s bytes:16921 cnt:1262248
24-Apr-2023 15:20:48  : Message rate:      67 msg/sec ... 21 KB/s bytes:22083 cnt:1262316
24-Apr-2023 15:20:49  : Message rate:      67 msg/sec ... 20 KB/s bytes:21441 cnt:1262384
24-Apr-2023 15:20:50  : Message rate:      68 msg/sec ... 24 KB/s bytes:24919 cnt:1262453
24-Apr-2023 15:20:51  : Message rate:      68 msg/sec ... 38 KB/s bytes:39370 cnt:1262522
24-Apr-2023 15:20:52  : Message rate:      67 msg/sec ... 34 KB/s bytes:35395 cnt:1262591
24-Apr-2023 15:20:53  : Message rate:      68 msg/sec ... 43 KB/s bytes:45015 cnt:1262660
24-Apr-2023 15:20:54  : Message rate:      68 msg/sec ... 37 KB/s bytes:38555 cnt:1262729
24-Apr-2023 15:20:55  : Message rate:      68 msg/sec ... 39 KB/s bytes:40589 cnt:1262798
24-Apr-2023 15:20:56  : Message rate:      68 msg/sec ... 30 KB/s bytes:31335 cnt:1262867
24-Apr-2023 15:20:57  : Message rate:      67 msg/sec ... 40 KB/s bytes:41501 cnt:1262935
24-Apr-2023 15:20:58  : Message rate:      68 msg/sec ... 38 KB/s bytes:39752 cnt:1263004
24-Apr-2023 15:20:59  : Message rate:      68 msg/sec ... 30 KB/s bytes:30812 cnt:1263073
24-Apr-2023 15:21:00  : Message rate:      68 msg/sec ... 28 KB/s bytes:29113 cnt:1263142
24-Apr-2023 15:21:01  : Message rate:      67 msg/sec ... 20 KB/s bytes:21089 cnt:1263210
24-Apr-2023 15:21:02  : Message rate:      68 msg/sec ... 34 KB/s bytes:35577 cnt:1263279
24-Apr-2023 15:21:03  : Message rate:      67 msg/sec ... 40 KB/s bytes:41370 cnt:1263347
24-Apr-2023 15:21:04  : Message rate:      68 msg/sec ... 37 KB/s bytes:38376 cnt:1263416
24-Apr-2023 15:21:05  : Message rate:      68 msg/sec ... 29 KB/s bytes:30399 cnt:1263485
24-Apr-2023 15:21:06  : Message rate:      69 msg/sec ... 21 KB/s bytes:22087 cnt:1263555
24-Apr-2023 15:21:07  : Message rate:      69 msg/sec ... 30 KB/s bytes:31528 cnt:1263625
24-Apr-2023 15:21:08  : Message rate:      68 msg/sec ... 35 KB/s bytes:36650 cnt:1263694
24-Apr-2023 15:21:09  : Message rate:      68 msg/sec ... 27 KB/s bytes:28286 cnt:1263763
24-Apr-2023 15:21:10  : Message rate:      68 msg/sec ... 42 KB/s bytes:43134 cnt:1263832
24-Apr-2023 15:21:11  : Message rate:      68 msg/sec ... 36 KB/s bytes:37113 cnt:1263901
24-Apr-2023 15:21:12  : Message rate:      68 msg/sec ... 60 KB/s bytes:62238 cnt:1263970
24-Apr-2023 15:21:13  : Message rate:      68 msg/sec ... 36 KB/s bytes:37555 cnt:1264039
24-Apr-2023 15:21:14  : Message rate:      68 msg/sec ... 26 KB/s bytes:26830 cnt:1264108
24-Apr-2023 15:21:15  : Message rate:      68 msg/sec ... 30 KB/s bytes:31384 cnt:1264177
24-Apr-2023 15:21:16  : Message rate:      67 msg/sec ... 33 KB/s bytes:34055 cnt:1264245
24-Apr-2023 15:21:17  : Message rate:      68 msg/sec ... 42 KB/s bytes:43756 cnt:1264314
24-Apr-2023 15:21:18  : Message rate:      68 msg/sec ... 49 KB/s bytes:50216 cnt:1264383
24-Apr-2023 15:21:19  : Message rate:      68 msg/sec ... 48 KB/s bytes:49901 cnt:1264452
24-Apr-2023 15:21:20  : Message rate:      67 msg/sec ... 59 KB/s bytes:60528 cnt:1264520
24-Apr-2023 15:21:21  : Message rate:      69 msg/sec ... 44 KB/s bytes:45534 cnt:1264590
24-Apr-2023 15:21:22  : Message rate:      68 msg/sec ... 41 KB/s bytes:42477 cnt:1264659
24-Apr-2023 15:21:23  : Message rate:      68 msg/sec ... 42 KB/s bytes:43665 cnt:1264728
24-Apr-2023 15:21:24  : Message rate:      67 msg/sec ... 36 KB/s bytes:37520 cnt:1264796
24-Apr-2023 15:21:25  : Message rate:      68 msg/sec ... 65 KB/s bytes:66589 cnt:1264865
24-Apr-2023 15:21:26  : Message rate:      69 msg/sec ... 67 KB/s bytes:68620 cnt:1264935
24-Apr-2023 15:21:27  : Message rate:      68 msg/sec ... 23 KB/s bytes:24029 cnt:1265004
24-Apr-2023 15:21:28  : Message rate:      67 msg/sec ... 41 KB/s bytes:42977 cnt:1265072
24-Apr-2023 15:21:29  : Message rate:      69 msg/sec ... 37 KB/s bytes:38571 cnt:1265142
24-Apr-2023 15:21:30  : Message rate:      69 msg/sec ... 28 KB/s bytes:29556 cnt:1265212
24-Apr-2023 15:21:31  : Message rate:      68 msg/sec ... 34 KB/s bytes:34841 cnt:1265281
24-Apr-2023 15:21:32  : Message rate:      68 msg/sec ... 42 KB/s bytes:43333 cnt:1265351
24-Apr-2023 15:21:33  : Message rate:      67 msg/sec ... 44 KB/s bytes:45327 cnt:1265419
24-Apr-2023 15:21:34  : Message rate:      69 msg/sec ... 48 KB/s bytes:49715 cnt:1265489
24-Apr-2023 15:21:35  : Message rate:      68 msg/sec ... 24 KB/s bytes:25150 cnt:1265558
24-Apr-2023 15:21:36  : Message rate:      70 msg/sec ... 29 KB/s bytes:29810 cnt:1265629
24-Apr-2023 15:21:37  : Message rate:      69 msg/sec ... 41 KB/s bytes:42532 cnt:1265699
24-Apr-2023 15:21:38  : Message rate:      69 msg/sec ... 42 KB/s bytes:44015 cnt:1265769
24-Apr-2023 15:21:39  : Message rate:      68 msg/sec ... 39 KB/s bytes:40453 cnt:1265838
24-Apr-2023 15:21:40  : Message rate:      68 msg/sec ... 32 KB/s bytes:33172 cnt:1265907
24-Apr-2023 15:21:41  : Message rate:      68 msg/sec ... 37 KB/s bytes:38496 cnt:1265976
24-Apr-2023 15:21:42  : Message rate:      68 msg/sec ... 24 KB/s bytes:24618 cnt:1266045
24-Apr-2023 15:21:43  : Message rate:      69 msg/sec ... 16 KB/s bytes:16581 cnt:1266115
24-Apr-2023 15:21:44  : Message rate:      67 msg/sec ... 19 KB/s bytes:19522 cnt:1266183
24-Apr-2023 15:21:45  : Message rate:      69 msg/sec ... 40 KB/s bytes:41727 cnt:1266253
24-Apr-2023 15:21:46  : Message rate:      68 msg/sec ... 28 KB/s bytes:29693 cnt:1266322
24-Apr-2023 15:21:47  : Message rate:      69 msg/sec ... 21 KB/s bytes:21738 cnt:1266392
24-Apr-2023 15:21:48  : Message rate:      69 msg/sec ... 24 KB/s bytes:25539 cnt:1266462
24-Apr-2023 15:21:49  : Message rate:      70 msg/sec ... 24 KB/s bytes:24947 cnt:1266533
24-Apr-2023 15:21:50  : Message rate:      68 msg/sec ... 27 KB/s bytes:27864 cnt:1266602
24-Apr-2023 15:21:51  : Message rate:      69 msg/sec ... 38 KB/s bytes:39750 cnt:1266672
24-Apr-2023 15:21:52  : Message rate:      68 msg/sec ... 30 KB/s bytes:31488 cnt:1266741
24-Apr-2023 15:21:53  : Message rate:      69 msg/sec ... 27 KB/s bytes:28379 cnt:1266811
24-Apr-2023 15:21:54  : Message rate:      69 msg/sec ... 30 KB/s bytes:30780 cnt:1266881
24-Apr-2023 15:21:55  : Message rate:      69 msg/sec ... 35 KB/s bytes:36834 cnt:1266951
24-Apr-2023 15:21:56  : Message rate:      69 msg/sec ... 47 KB/s bytes:48881 cnt:1267021
24-Apr-2023 15:21:57  : Message rate:      68 msg/sec ... 32 KB/s bytes:32810 cnt:1267090
24-Apr-2023 15:21:58  : Message rate:      69 msg/sec ... 28 KB/s bytes:29432 cnt:1267160
24-Apr-2023 15:21:59  : Message rate:      70 msg/sec ... 21 KB/s bytes:21547 cnt:1267231
24-Apr-2023 15:22:00  : Message rate:      69 msg/sec ... 17 KB/s bytes:17649 cnt:1267301
24-Apr-2023 15:22:01  : Message rate:      70 msg/sec ... 22 KB/s bytes:23470 cnt:1267372
24-Apr-2023 15:22:02  : Message rate:      68 msg/sec ... 34 KB/s bytes:34941 cnt:1267441
24-Apr-2023 15:22:03  : Message rate:      68 msg/sec ... 28 KB/s bytes:29049 cnt:1267511
24-Apr-2023 15:22:04  : Message rate:      69 msg/sec ... 29 KB/s bytes:30440 cnt:1267581
24-Apr-2023 15:22:05  : Message rate:      70 msg/sec ... 33 KB/s bytes:34796 cnt:1267652
24-Apr-2023 15:22:06  : Message rate:      70 msg/sec ... 30 KB/s bytes:30972 cnt:1267723
24-Apr-2023 15:22:07  : Message rate:      69 msg/sec ... 37 KB/s bytes:38361 cnt:1267793
24-Apr-2023 15:22:08  : Message rate:      69 msg/sec ... 43 KB/s bytes:44131 cnt:1267863
24-Apr-2023 15:22:09  : Message rate:      68 msg/sec ... 48 KB/s bytes:49400 cnt:1267932
24-Apr-2023 15:22:11  : Message rate:      69 msg/sec ... 31 KB/s bytes:32618 cnt:1268002
24-Apr-2023 15:22:12  : Message rate:      69 msg/sec ... 27 KB/s bytes:28044 cnt:1268072
24-Apr-2023 15:22:13  : Message rate:      69 msg/sec ... 34 KB/s bytes:35721 cnt:1268142
24-Apr-2023 15:22:14  : Message rate:      68 msg/sec ... 44 KB/s bytes:45789 cnt:1268211
24-Apr-2023 15:22:15  : Message rate:      69 msg/sec ... 38 KB/s bytes:39191 cnt:1268281
24-Apr-2023 15:22:16  : Message rate:      70 msg/sec ... 26 KB/s bytes:27319 cnt:1268352
24-Apr-2023 15:22:17  : Message rate:      68 msg/sec ... 31 KB/s bytes:32235 cnt:1268421
24-Apr-2023 15:22:18  : Message rate:      69 msg/sec ... 30 KB/s bytes:31371 cnt:1268491
24-Apr-2023 15:22:19  : Message rate:      69 msg/sec ... 23 KB/s bytes:24294 cnt:1268561
24-Apr-2023 15:22:20  : Message rate:      68 msg/sec ... 22 KB/s bytes:23031 cnt:1268630
24-Apr-2023 15:22:21  : Message rate:      69 msg/sec ... 30 KB/s bytes:31094 cnt:1268700
24-Apr-2023 15:22:22  : Message rate:      70 msg/sec ... 29 KB/s bytes:30435 cnt:1268771
24-Apr-2023 15:22:23  : Message rate:      68 msg/sec ... 32 KB/s bytes:33136 cnt:1268840
24-Apr-2023 15:22:24  : Message rate:      69 msg/sec ... 33 KB/s bytes:33878 cnt:1268910
24-Apr-2023 15:22:25  : Message rate:      69 msg/sec ... 32 KB/s bytes:33181 cnt:1268980
24-Apr-2023 15:22:26  : Message rate:      68 msg/sec ... 43 KB/s bytes:44992 cnt:1269049
24-Apr-2023 15:22:27  : Message rate:      70 msg/sec ... 34 KB/s bytes:35277 cnt:1269120
24-Apr-2023 15:22:28  : Message rate:      69 msg/sec ... 35 KB/s bytes:35917 cnt:1269190
24-Apr-2023 15:22:29  : Message rate:      68 msg/sec ... 40 KB/s bytes:41212 cnt:1269259
24-Apr-2023 15:22:30  : Message rate:      70 msg/sec ... 35 KB/s bytes:36109 cnt:1269330
24-Apr-2023 15:22:31  : Message rate:      68 msg/sec ... 33 KB/s bytes:34121 cnt:1269399
24-Apr-2023 15:22:32  : Message rate:      69 msg/sec ... 38 KB/s bytes:39620 cnt:1269469
24-Apr-2023 15:22:33  : Message rate:      68 msg/sec ... 35 KB/s bytes:36589 cnt:1269538
24-Apr-2023 15:22:34  : Message rate:      68 msg/sec ... 41 KB/s bytes:42788 cnt:1269607
24-Apr-2023 15:22:35  : Message rate:      71 msg/sec ... 31 KB/s bytes:32045 cnt:1269679
24-Apr-2023 15:22:36  : Message rate:      70 msg/sec ... 44 KB/s bytes:45895 cnt:1269750
24-Apr-2023 15:22:37  : Message rate:      70 msg/sec ... 38 KB/s bytes:38977 cnt:1269821
24-Apr-2023 15:22:38  : Message rate:      69 msg/sec ... 43 KB/s bytes:44382 cnt:1269891
24-Apr-2023 15:22:39  : Message rate:      68 msg/sec ... 45 KB/s bytes:46892 cnt:1269960
24-Apr-2023 15:22:40  : Message rate:      69 msg/sec ... 28 KB/s bytes:28717 cnt:1270030
24-Apr-2023 15:22:41  : Message rate:      69 msg/sec ... 33 KB/s bytes:34311 cnt:1270100
24-Apr-2023 15:22:42  : Message rate:      69 msg/sec ... 52 KB/s bytes:53266 cnt:1270170
24-Apr-2023 15:22:43  : Message rate:      69 msg/sec ... 52 KB/s bytes:53526 cnt:1270240
24-Apr-2023 15:22:44  : Message rate:      69 msg/sec ... 30 KB/s bytes:31662 cnt:1270310
24-Apr-2023 15:22:45  : Message rate:      70 msg/sec ... 29 KB/s bytes:30024 cnt:1270381
24-Apr-2023 15:22:46  : Message rate:      70 msg/sec ... 30 KB/s bytes:31584 cnt:1270452
24-Apr-2023 15:22:47  : Message rate:      70 msg/sec ... 24 KB/s bytes:24776 cnt:1270523
24-Apr-2023 15:22:48  : Message rate:      69 msg/sec ... 32 KB/s bytes:32979 cnt:1270593
24-Apr-2023 15:22:49  : Message rate:      69 msg/sec ... 34 KB/s bytes:35400 cnt:1270663
24-Apr-2023 15:22:50  : Message rate:      69 msg/sec ... 27 KB/s bytes:28109 cnt:1270733
24-Apr-2023 15:22:51  : Message rate:      70 msg/sec ... 19 KB/s bytes:20161 cnt:1270804
24-Apr-2023 15:22:52  : Message rate:      70 msg/sec ... 19 KB/s bytes:19644 cnt:1270875
24-Apr-2023 15:22:53  : Message rate:      67 msg/sec ... 21 KB/s bytes:21839 cnt:1270943
24-Apr-2023 15:22:54  : Message rate:      70 msg/sec ... 19 KB/s bytes:20195 cnt:1271014
24-Apr-2023 15:22:55  : Message rate:      70 msg/sec ... 35 KB/s bytes:36016 cnt:1271085
24-Apr-2023 15:22:56  : Message rate:      69 msg/sec ... 25 KB/s bytes:25911 cnt:1271155
24-Apr-2023 15:22:57  : Message rate:      70 msg/sec ... 35 KB/s bytes:36440 cnt:1271226
24-Apr-2023 15:22:58  : Message rate:      70 msg/sec ... 66 KB/s bytes:68132 cnt:1271297
24-Apr-2023 15:22:59  : Message rate:      69 msg/sec ... 50 KB/s bytes:51234 cnt:1271367
24-Apr-2023 15:23:00  : Message rate:      70 msg/sec ... 49 KB/s bytes:50438 cnt:1271438
24-Apr-2023 15:23:01  : Message rate:      70 msg/sec ... 43 KB/s bytes:44901 cnt:1271509
24-Apr-2023 15:23:02  : Message rate:      70 msg/sec ... 46 KB/s bytes:47985 cnt:1271580
24-Apr-2023 15:23:03  : Message rate:      70 msg/sec ... 33 KB/s bytes:34454 cnt:1271651
24-Apr-2023 15:23:04  : Message rate:      71 msg/sec ... 68 KB/s bytes:69887 cnt:1271723
24-Apr-2023 15:23:05  : Message rate:      71 msg/sec ... 40 KB/s bytes:41297 cnt:1271795
24-Apr-2023 15:23:06  : Message rate:      71 msg/sec ... 32 KB/s bytes:33208 cnt:1271867
24-Apr-2023 15:23:07  : Message rate:      70 msg/sec ... 54 KB/s bytes:55378 cnt:1271938
24-Apr-2023 15:23:08  : Message rate:      70 msg/sec ... 40 KB/s bytes:41229 cnt:1272009
24-Apr-2023 15:23:09  : Message rate:      70 msg/sec ... 41 KB/s bytes:42715 cnt:1272080
24-Apr-2023 15:23:10  : Message rate:      71 msg/sec ... 33 KB/s bytes:34738 cnt:1272152
24-Apr-2023 15:23:11  : Message rate:      68 msg/sec ... 40 KB/s bytes:41765 cnt:1272221
24-Apr-2023 15:23:12  : Message rate:      70 msg/sec ... 88 KB/s bytes:90587 cnt:1272292
24-Apr-2023 15:23:13  : Message rate:      70 msg/sec ... 53 KB/s bytes:54430 cnt:1272363
24-Apr-2023 15:23:14  : Message rate:      69 msg/sec ... 48 KB/s bytes:49283 cnt:1272433
24-Apr-2023 15:23:15  : Message rate:      70 msg/sec ... 53 KB/s bytes:54847 cnt:1272504
cvoica commented 1 year ago

I debugged the application and noticed that the _rxbuf is big when the rates start to drop. The _rxbuf.erase(_rxbuf.begin(), _rxbuf.begin() + ws.header_size + (size_t) ws.N); is then slowing down the application.

(gdb) print _rxbuf.size()
$2 = 40256441
(gdb) python
>import time
>s=time.time()
>gdb.execute("continue 3000")
>print(time.time()-s)
>
Breakpoint 1, ix::WebSocketTransport::dispatch(ix::WebSocketTransport::PollResult, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, ix::WebSocketTransport::MessageKind)> const&) (this=this@entry=0x7ffc62228fb8, pollResult=<optimized out>, onMessageCallback=...) at /home/root/IXWebSocket-11.4.3/ixwebsocket/IXWebSocketTransport.cpp:550
550                         emitMessage(_fragmentedMessageKind,
7.368263244628906
(gdb) print _rxbuf.size()
$3 = 38810270
(gdb) where
#0  ix::WebSocketTransport::dispatch(ix::WebSocketTransport::PollResult, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, ix::WebSocketTransport::MessageKind)> const&) (this=this@entry=0x7ffc62228fb8, pollResult=<optimized out>, onMessageCallback=...)
    at /home/root/IXWebSocket-11.4.3/ixwebsocket/IXWebSocketTransport.cpp:550
#1  0x0000559f77ce1e89 in ix::WebSocket::run (this=this@entry=0x7ffc62228fb8) at /home/root/IXWebSocket-11.4.3/ixwebsocket/IXWebSocket.cpp:382
#2  0x0000559f77c92847 in WebInterface::run (this=this@entry=0x7ffc62228fa0) at cpp/src/client/webinterface.cpp:91
#3  0x0000559f77c8e5e5 in main (argc=<optimized out>, argv=<optimized out>) at cpp/src/client/main.cpp:20
cvoica commented 1 year ago

As I see the issue now it's a problem of removing from the beginning of the vector: every small message (e.g. below 1K) is going to trigger a memory move for the rest of the vector. At a certain size this will be problematic. The questions are then if you are willing to replace this mechanism or deflect to _chunks once the _rxbuf contains too many small messages? What would be the disadvantage of not having a _rxbuf growing without a limit?

cvoica commented 1 year ago

it seems to be the same like #444