lynxthecat / cake-autorate

Eliminate the excess latency and jitter terrorizing your 4G, 5G, LTE, Starlink or other variable rate connection!
https://www.bufferbloat.net
GNU General Public License v2.0
241 stars 24 forks source link

Error: line 1549: ewma: unbound variable #192

Closed bairhys closed 1 year ago

bairhys commented 1 year ago

Hey Lynx,

I updated to the latest commit c4f196a and had a couple of crashes since. Attached is the last few lines of the log. Near the end, the log lists ERROR; /root/cake-autorate/cake-autorate.sh: line 1549: ewma: unbound variable. But maybe something else is going on since there are a lot of DATA lines near the top of the attached log, then no DATA lines for the rest.

LOAD; 2023-06-13-09:49:15; 1686649755.643599; 1686649755.643255; 76; 59; 50000; 5000
DATA; 2023-06-13-09:49:15; 1686649755.650894; 1686649755.650260; 76; 59; 0; 1; 1686649755.621951; 94.140.14.15; 527; 33180; 36000; 1974; 2820; 30240; -4675; 7000; 15034; 11675; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:15; 1686649755.655340; 1686649755.654720; 76; 59; 0; 1; 1686649755.621951; 94.140.14.15; 527; 33182; 36000; 2054; 2818; 30240; -4663; 7000; 14713; 11663; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:15; 1686649755.667715; 1686649755.667092; 76; 59; 0; 1; 1686649755.660957; 94.140.14.140; 527; 33039; 35000; 2399; 1961; 30240; -3883; -3000; 7391; 883; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:15; 1686649755.727124; 1686649755.726362; 76; 59; 0; 1; 1686649755.718637; 94.140.14.141; 527; 33002; 33000; 3148; -2; 30240; -3069; 7000; 11758; 10069; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:15; 1686649755.775829; 1686649755.775203; 76; 59; 0; 1; 1686649755.768371; 9.9.9.9; 528; 16347; 18000; 2457; 1653; 30240; 1051; 21000; 9517; 19949; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:15; 1686649755.806341; 1686649755.805634; 76; 59; 0; 1; 1686649755.798517; 9.9.9.10; 528; 16662; 18000; 1290; 1338; 30240; 1049; 1000; 13093; -49; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-13-09:49:15; 1686649755.844460; 1686649755.844117; 9; 8; 50000; 5000
DATA; 2023-06-13-09:49:15; 1686649755.853559; 1686649755.852937; 9; 8; 0; 0; 1686649755.798517; 9.9.9.10; 528; 16663; 18000; 1294; 1337; 30240; 1004; 1000; 11848; -4; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:15; 1686649755.858003; 1686649755.857326; 9; 8; 0; 0; 1686649755.798517; 9.9.9.10; 528; 16664; 18000; 1297; 1336; 30240; 1000; 1000; 10722; 0; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:15; 1686649755.875894; 1686649755.875276; 9; 8; 0; 0; 1686649755.869001; 9.9.9.11; 528; 14220; 17000; 1817; 2780; 30240; 3139; 23000; 13140; 19861; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:15; 1686649755.936623; 1686649755.935982; 9; 8; 0; 0; 1686649755.929680; 94.140.14.15; 528; 33183; 35000; 2031; 1817; 30240; -4643; 15000; 15181; 19643; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:15; 1686649755.978509; 1686649755.977397; 9; 8; 0; 0; 1686649755.971223; 94.140.14.140; 528; 33041; 36000; 2452; 2959; 30240; -3874; 5000; 7531; 8874; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.037574; 1686649756.036954; 9; 8; 0; 0; 1686649756.030751; 94.140.14.141; 528; 33004; 36000; 3133; 2996; 30240; -3051; 14000; 12260; 17051; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-13-09:49:16; 1686649756.045221; 1686649756.044876; 9; 13; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.051622; 1686649756.050984; 9; 13; 0; 0; 1686649756.030751; 94.140.14.141; 528; 33006; 36000; 3119; 2994; 30240; -3033; 14000; 12713; 17033; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.056046; 1686649756.055426; 9; 13; 0; 0; 1686649756.030751; 94.140.14.141; 528; 33008; 36000; 3106; 2992; 30240; -3015; 14000; 13121; 17015; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.065217; 1686649756.064596; 9; 13; 0; 0; 1686649756.058286; 9.9.9.9; 529; 16348; 18000; 2380; 1652; 30240; 1059; 10000; 9462; 8941; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.125552; 1686649756.124933; 9; 13; 0; 0; 1686649756.118807; 9.9.9.10; 529; 16665; 18000; 1300; 1335; 30240; 1019; 20000; 11506; 18981; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.175590; 1686649756.174972; 9; 13; 0; 0; 1686649756.168673; 9.9.9.11; 529; 14221; 16000; 1813; 1779; 30240; 3157; 22000; 13681; 18843; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.237543; 1686649756.236919; 9; 13; 0; 0; 1686649756.230639; 94.140.14.15; 529; 33185; 36000; 2105; 2815; 30240; -4625; 13000; 15413; 17625; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-13-09:49:16; 1686649756.246100; 1686649756.245751; 18; 13; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.252523; 1686649756.251890; 18; 13; 0; 0; 1686649756.230639; 94.140.14.15; 529; 33187; 36000; 2172; 2813; 30240; -4607; 13000; 15621; 17607; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.256949; 1686649756.256331; 18; 13; 0; 0; 1686649756.230639; 94.140.14.15; 529; 33189; 36000; 2232; 2811; 30240; -4589; 13000; 15807; 17589; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.278504; 1686649756.277776; 18; 13; 0; 0; 1686649756.271659; 94.140.14.140; 529; 33043; 36000; 2499; 2957; 30240; -3866; 4000; 7562; 7866; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.335734; 1686649756.335114; 18; 13; 0; 0; 1686649756.328930; 94.140.14.141; 529; 33008; 34000; 2905; 992; 30240; -2998; 13000; 13394; 15998; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.365662; 1686649756.365041; 18; 13; 0; 0; 1686649756.358976; 9.9.9.9; 530; 16349; 18000; 2310; 1651; 30240; 1066; 9000; 9316; 7934; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.424620; 1686649756.424001; 18; 13; 0; 0; 1686649756.417868; 9.9.9.10; 530; 16665; 17000; 1208; 335; 30240; 1035; 18000; 12024; 16965; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-13-09:49:16; 1686649756.447011; 1686649756.446662; 12; 10; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.453098; 1686649756.452463; 12; 10; 0; 0; 1686649756.417868; 9.9.9.10; 530; 16665; 17000; 1125; 335; 30240; 1051; 18000; 12491; 16949; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.457518; 1686649756.456880; 12; 10; 0; 0; 1686649756.417868; 9.9.9.10; 530; 16665; 17000; 1049; 335; 30240; 1067; 18000; 12912; 16933; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.478163; 1686649756.477436; 12; 10; 0; 0; 1686649756.471323; 9.9.9.11; 530; 14225; 19000; 2094; 4775; 30240; 3173; 20000; 13979; 16827; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.536452; 1686649756.535830; 12; 10; 0; 0; 1686649756.529451; 94.140.14.15; 530; 33190; 35000; 2191; 1810; 30240; -4572; 12000; 15879; 16572; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.578439; 1686649756.577703; 12; 10; 0; 0; 1686649756.571121; 94.140.14.140; 530; 33045; 36000; 2542; 2955; 30240; -3859; 3000; 7495; 6859; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.628092; 1686649756.627401; 12; 10; 0; 0; 1686649756.620945; 94.140.14.141; 530; 33009; 35000; 2818; 1991; 30240; -2992; 3000; 12690; 5992; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-13-09:49:16; 1686649756.647898; 1686649756.647495; 61; 98; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.655267; 1686649756.654586; 61; 98; 0; 1; 1686649756.620945; 94.140.14.141; 530; 33010; 35000; 2739; 1990; 30240; -2986; 3000; 12053; 5986; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.659892; 1686649756.659250; 61; 98; 0; 1; 1686649756.620945; 94.140.14.141; 530; 33011; 35000; 2667; 1989; 30240; -2980; 3000; 11476; 5980; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.664260; 1686649756.663636; 61; 98; 0; 1; 1686649756.654129; 9.9.9.9; 531; 16351; 19000; 2342; 2649; 30240; 1066; 2000; 8519; 934; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.722515; 1686649756.721829; 61; 98; 0; 1; 1686649756.714024; 9.9.9.10; 531; 16667; 19000; 1170; 2333; 30240; 1077; 12000; 12723; 10923; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.775885; 1686649756.775266; 61; 98; 0; 1; 1686649756.769129; 9.9.9.11; 531; 14227; 17000; 2158; 2773; 30240; 3188; 19000; 14153; 15812; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.831869; 1686649756.831246; 61; 98; 0; 1; 1686649756.824735; 94.140.14.15; 531; 33191; 35000; 2154; 1809; 30240; -4561; 6000; 15373; 10561; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-13-09:49:16; 1686649756.848716; 1686649756.848362; 143; 28; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.855842; 1686649756.855199; 143; 28; 0; 0; 1686649756.824735; 94.140.14.15; 531; 33192; 35000; 2121; 1808; 30240; -4550; 6000; 14914; 10550; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.860316; 1686649756.859695; 143; 28; 0; 0; 1686649756.824735; 94.140.14.15; 531; 33193; 35000; 2091; 1807; 30240; -4539; 6000; 14498; 10539; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.878220; 1686649756.877451; 143; 28; 0; 0; 1686649756.871115; 94.140.14.140; 531; 33047; 36000; 2581; 2953; 30240; -3854; 1000; 7244; 4854; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.937611; 1686649756.936963; 143; 28; 0; 0; 1686649756.930502; 94.140.14.141; 531; 33012; 35000; 2602; 1988; 30240; -2966; 11000; 11712; 13966; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:16; 1686649756.965706; 1686649756.965060; 143; 28; 0; 0; 1686649756.958826; 9.9.9.9; 532; 16351; 17000; 2181; 649; 30240; 1071; 7000; 8272; 5929; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.025430; 1686649757.024808; 143; 28; 0; 0; 1686649757.018671; 9.9.9.10; 532; 16668; 18000; 1185; 1332; 30240; 1091; 16000; 12930; 14909; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-13-09:49:17; 1686649757.050480; 1686649757.049828; 9; 23; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.056614; 1686649757.055976; 9; 23; 0; 0; 1686649757.018671; 9.9.9.10; 532; 16669; 18000; 1198; 1331; 30240; 1105; 16000; 13116; 14895; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.061118; 1686649757.060501; 9; 23; 0; 0; 1686649757.018671; 9.9.9.10; 532; 16670; 18000; 1210; 1330; 30240; 1119; 16000; 13283; 14881; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.070402; 1686649757.069782; 9; 23; 0; 0; 1686649757.063462; 9.9.9.11; 532; 14228; 16000; 2121; 1772; 30240; 3197; 13000; 13739; 9803; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.128278; 1686649757.127564; 9; 23; 0; 0; 1686649757.121402; 94.140.14.15; 532; 33196; 37000; 2253; 3804; 30240; -4535; -1000; 13456; 3535; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.187928; 1686649757.187243; 9; 23; 0; 0; 1686649757.180263; 94.140.14.140; 532; 33048; 35000; 2521; 1952; 30240; -3840; 10000; 7870; 13840; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.226885; 1686649757.226263; 9; 23; 0; 0; 1686649757.220101; 94.140.14.141; 532; 33013; 35000; 2543; 1987; 30240; -2963; 0000; 10880; 2963; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-13-09:49:17; 1686649757.250337; 1686649757.249989; 188; 13; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.256676; 1686649757.256035; 188; 13; 0; 0; 1686649757.220101; 94.140.14.141; 532; 33014; 35000; 2490; 1986; 30240; -2960; 0000; 10127; 2960; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.261180; 1686649757.260558; 188; 13; 0; 0; 1686649757.220101; 94.140.14.141; 532; 33015; 35000; 2442; 1985; 30240; -2957; 0000; 9445; 2957; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.274500; 1686649757.273878; 188; 13; 0; 0; 1686649757.267767; 9.9.9.9; 533; 16351; 17000; 2035; 649; 30240; 1084; 15000; 8808; 13916; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.326501; 1686649757.325879; 188; 13; 0; 0; 1686649757.319638; 9.9.9.10; 533; 16672; 19000; 1316; 2328; 30240; 1132; 15000; 13338; 13868; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.375434; 1686649757.374766; 188; 13; 0; 0; 1686649757.368240; 9.9.9.11; 533; 14229; 16000; 2087; 1771; 30240; 3209; 16000; 13648; 12791; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-13-09:49:17; 1686649757.451274; 1686649757.450926; 43; 40; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.457802; 1686649757.457161; 43; 40; 0; 0; 1686649757.368240; 9.9.9.11; 533; 14230; 16000; 2056; 1770; 30240; 3221; 16000; 13565; 12779; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.462308; 1686649757.461689; 43; 40; 0; 0; 1686649757.368240; 9.9.9.11; 533; 14231; 16000; 2028; 1769; 30240; 3233; 16000; 13489; 12767; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-13-09:49:17; 1686649757.652068; 1686649757.651717; 12; 55; 50000; 5000
DEBUG; 2023-06-13-09:49:17; 1686649757.654265; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:17; 1686649757.655455; load check is: (( 12 kbps > 10 kbps for download && 40 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:17; 1686649757.657217; load above connection stall threshold so resuming normal operation.
DEBUG; 2023-06-13-09:49:17; 1686649757.658730; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:17; 1686649757.659670; load check is: (( 12 kbps > 10 kbps for download && 55 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:17; 1686649757.660579; load above connection stall threshold so resuming normal operation.
DATA; 2023-06-13-09:49:17; 1686649757.664886; 1686649757.664266; 12; 55; 0; 1; 1686649757.368240; 9.9.9.11; 533; 14232; 16000; 2003; 1768; 30240; 3245; 16000; 13419; 12755; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.669342; 1686649757.668720; 12; 55; 0; 1; 1686649757.368240; 9.9.9.11; 533; 14233; 16000; 1980; 1767; 30240; 3257; 16000; 13354; 12743; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-13-09:49:17; 1686649757.852976; 1686649757.852622; 17; 12; 50000; 5000
DEBUG; 2023-06-13-09:49:17; 1686649757.855550; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:17; 1686649757.856503; load check is: (( 17 kbps > 10 kbps for download && 55 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:17; 1686649757.857373; load above connection stall threshold so resuming normal operation.
DEBUG; 2023-06-13-09:49:17; 1686649757.859385; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:17; 1686649757.860326; load check is: (( 17 kbps > 10 kbps for download && 12 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:17; 1686649757.861214; load above connection stall threshold so resuming normal operation.
DATA; 2023-06-13-09:49:17; 1686649757.865563; 1686649757.864943; 17; 12; 0; 0; 1686649757.368240; 9.9.9.11; 533; 14234; 16000; 1959; 1766; 30240; 3269; 16000; 13294; 12731; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-13-09:49:17; 1686649757.870011; 1686649757.869390; 17; 12; 0; 0; 1686649757.368240; 9.9.9.11; 533; 14235; 16000; 1940; 1765; 30240; 3281; 16000; 13239; 12719; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-13-09:49:18; 1686649758.053889; 1686649758.053544; 8; 15; 50000; 5000
DEBUG; 2023-06-13-09:49:18; 1686649758.055932; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:18; 1686649758.056820; load check is: (( 8 kbps > 10 kbps for download && 12 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:18; 1686649758.058717; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:18; 1686649758.061653; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:18; 1686649758.062736; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:18; 1686649758.066914; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:18; 1686649758.254766; 1686649758.254417; 17; 18; 50000; 5000
DEBUG; 2023-06-13-09:49:18; 1686649758.257023; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:18; 1686649758.258000; load check is: (( 17 kbps > 10 kbps for download && 15 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:18; 1686649758.258869; load above connection stall threshold so resuming normal operation.
DEBUG; 2023-06-13-09:49:18; 1686649758.260287; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:18; 1686649758.261236; load check is: (( 17 kbps > 10 kbps for download && 18 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:18; 1686649758.262419; load above connection stall threshold so resuming normal operation.
DEBUG; 2023-06-13-09:49:18; 1686649758.265161; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:18; 1686649758.267528; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:18; 1686649758.455609; 1686649758.455262; 21; 12; 50000; 5000
DEBUG; 2023-06-13-09:49:18; 1686649758.457656; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:18; 1686649758.458690; load check is: (( 21 kbps > 10 kbps for download && 18 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:18; 1686649758.459572; load above connection stall threshold so resuming normal operation.
DEBUG; 2023-06-13-09:49:18; 1686649758.461055; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:18; 1686649758.461982; load check is: (( 21 kbps > 10 kbps for download && 12 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:18; 1686649758.462858; load above connection stall threshold so resuming normal operation.
DEBUG; 2023-06-13-09:49:18; 1686649758.464776; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:18; 1686649758.467927; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:18; 1686649758.656458; 1686649758.656109; 6; 12; 50000; 5000
DEBUG; 2023-06-13-09:49:18; 1686649758.658557; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:18; 1686649758.659445; load check is: (( 6 kbps > 10 kbps for download && 12 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:18; 1686649758.660421; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:18; 1686649758.662891; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:18; 1686649758.663902; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:18; 1686649758.665816; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:18; 1686649758.857311; 1686649758.856968; 16; 8; 50000; 5000
DEBUG; 2023-06-13-09:49:18; 1686649758.859439; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:18; 1686649758.860324; load check is: (( 16 kbps > 10 kbps for download && 12 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:18; 1686649758.861178; load above connection stall threshold so resuming normal operation.
DEBUG; 2023-06-13-09:49:18; 1686649758.862627; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:18; 1686649758.863482; load check is: (( 16 kbps > 10 kbps for download && 8 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:18; 1686649758.864460; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:18; 1686649758.866262; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:18; 1686649758.867442; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:18; 1686649758.870227; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:18; 1686649758.925115; Starting: change_state_maintain_pingers with PID: 14438
DEBUG; 2023-06-13-09:49:18; 1686649758.926176; Changing maintain_pingers state from: RUNNING to: PAUSED
LOAD; 2023-06-13-09:49:19; 1686649759.058246; 1686649759.057888; 14; 12; 50000; 5000
DEBUG; 2023-06-13-09:49:19; 1686649759.060377; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:19; 1686649759.061253; load check is: (( 14 kbps > 10 kbps for download && 8 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:19; 1686649759.062242; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:19; 1686649759.063847; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:19; 1686649759.064859; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:19; 1686649759.066800; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:19; 1686649759.069200; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:19; 1686649759.259094; 1686649759.258745; 0; 8; 50000; 5000
DEBUG; 2023-06-13-09:49:19; 1686649759.261439; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:19; 1686649759.262409; load check is: (( 0 kbps > 10 kbps for download && 12 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:19; 1686649759.263432; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:19; 1686649759.265916; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:19; 1686649759.266997; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:19; 1686649759.269676; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:19; 1686649759.459882; 1686649759.459539; 4; 49; 50000; 5000
DEBUG; 2023-06-13-09:49:19; 1686649759.461959; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:19; 1686649759.462923; load check is: (( 4 kbps > 10 kbps for download && 8 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:19; 1686649759.463941; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:19; 1686649759.466916; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:19; 1686649759.468189; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:19; 1686649759.470278; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:19; 1686649759.660749; 1686649759.660399; 28; 55; 50000; 5000
DEBUG; 2023-06-13-09:49:19; 1686649759.662792; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:19; 1686649759.663689; load check is: (( 28 kbps > 10 kbps for download && 49 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:19; 1686649759.664540; load above connection stall threshold so resuming normal operation.
DEBUG; 2023-06-13-09:49:19; 1686649759.665974; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:19; 1686649759.666828; load check is: (( 28 kbps > 10 kbps for download && 55 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:19; 1686649759.667681; load above connection stall threshold so resuming normal operation.
DEBUG; 2023-06-13-09:49:19; 1686649759.669678; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:19; 1686649759.671835; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:19; 1686649759.861607; 1686649759.861261; 0; 8; 50000; 5000
DEBUG; 2023-06-13-09:49:19; 1686649759.863662; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:19; 1686649759.864773; load check is: (( 0 kbps > 10 kbps for download && 55 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:19; 1686649759.865801; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:19; 1686649759.868387; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:19; 1686649759.869384; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:19; 1686649759.871364; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:19; 1686649759.924915; Starting: change_state_maintain_pingers with PID: 14438
DEBUG; 2023-06-13-09:49:19; 1686649759.926013; Changing maintain_pingers state from: PAUSED to: RUNNING
DEBUG; 2023-06-13-09:49:19; 1686649759.927433; no ping response from reflector: 9.9.9.9 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:19; 1686649759.928563; reflector=9.9.9.9, sum_reflector_offences=1 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:19; 1686649759.929851; no ping response from reflector: 9.9.9.10 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:19; 1686649759.930843; reflector=9.9.9.10, sum_reflector_offences=1 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:19; 1686649759.932191; no ping response from reflector: 9.9.9.11 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:19; 1686649759.933170; reflector=9.9.9.11, sum_reflector_offences=1 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:19; 1686649759.934484; no ping response from reflector: 94.140.14.15 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:19; 1686649759.935468; reflector=94.140.14.15, sum_reflector_offences=1 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:19; 1686649759.936778; no ping response from reflector: 94.140.14.140 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:19; 1686649759.937752; reflector=94.140.14.140, sum_reflector_offences=1 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:19; 1686649759.939145; no ping response from reflector: 94.140.14.141 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:19; 1686649759.940126; reflector=94.140.14.141, sum_reflector_offences=1 and reflector_misbehaving_detection_thr=3
LOAD; 2023-06-13-09:49:20; 1686649760.062463; 1686649760.062117; 0; 14; 50000; 5000
DEBUG; 2023-06-13-09:49:20; 1686649760.064504; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:20; 1686649760.065393; load check is: (( 0 kbps > 10 kbps for download && 8 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:20; 1686649760.066375; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:20; 1686649760.068979; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:20; 1686649760.070003; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:20; 1686649760.071924; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:20; 1686649760.263325; 1686649760.262980; 0; 11; 50000; 5000
DEBUG; 2023-06-13-09:49:20; 1686649760.265372; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:20; 1686649760.266254; load check is: (( 0 kbps > 10 kbps for download && 14 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:20; 1686649760.267229; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:20; 1686649760.269822; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:20; 1686649760.270889; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:20; 1686649760.272842; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:20; 1686649760.464168; 1686649760.463823; 0; 12; 50000; 5000
DEBUG; 2023-06-13-09:49:20; 1686649760.466363; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:20; 1686649760.467415; load check is: (( 0 kbps > 10 kbps for download && 11 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:20; 1686649760.468670; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:20; 1686649760.471372; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:20; 1686649760.472364; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:20; 1686649760.474316; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:20; 1686649760.665032; 1686649760.664682; 3; 8; 50000; 5000
DEBUG; 2023-06-13-09:49:20; 1686649760.667065; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:20; 1686649760.668054; load check is: (( 3 kbps > 10 kbps for download && 12 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:20; 1686649760.669062; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:20; 1686649760.671850; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:20; 1686649760.673036; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:20; 1686649760.674956; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:20; 1686649760.865897; 1686649760.865550; 0; 8; 50000; 5000
DEBUG; 2023-06-13-09:49:20; 1686649760.868047; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:20; 1686649760.868949; load check is: (( 0 kbps > 10 kbps for download && 8 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:20; 1686649760.869929; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:20; 1686649760.872391; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:20; 1686649760.873407; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:20; 1686649760.875454; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:20; 1686649760.942003; Starting: change_state_maintain_pingers with PID: 14438
DEBUG; 2023-06-13-09:49:20; 1686649760.944091; Changing maintain_pingers state from: RUNNING to: PAUSED
LOAD; 2023-06-13-09:49:21; 1686649761.066764; 1686649761.066422; 0; 8; 50000; 5000
DEBUG; 2023-06-13-09:49:21; 1686649761.068862; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:21; 1686649761.069759; load check is: (( 0 kbps > 10 kbps for download && 8 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:21; 1686649761.070731; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:21; 1686649761.073187; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:21; 1686649761.074199; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:21; 1686649761.076970; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:21; 1686649761.268722; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:21; 1686649761.269645; load check is: (( 0 kbps > 10 kbps for download && 8 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:21; 1686649761.270616; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:21; 1686649761.273067; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:21; 1686649761.274066; Changing main state from: STALL to: RUNNING
LOAD; 2023-06-13-09:49:21; 1686649761.267683; 1686649761.267337; 0; 48; 50000; 5000
DEBUG; 2023-06-13-09:49:21; 1686649761.276032; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:21; 1686649761.468646; Warning: no reflector response within: 0.250000 seconds. Checking loads.
LOAD; 2023-06-13-09:49:21; 1686649761.468620; 1686649761.468275; 0; 85; 50000; 5000
DEBUG; 2023-06-13-09:49:21; 1686649761.470578; load check is: (( 0 kbps > 10 kbps for download && 48 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:21; 1686649761.472222; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:21; 1686649761.475482; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:21; 1686649761.477096; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:21; 1686649761.479685; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:21; 1686649761.669455; Warning: no reflector response within: 0.250000 seconds. Checking loads.
LOAD; 2023-06-13-09:49:21; 1686649761.669430; 1686649761.669103; 5; 51; 50000; 5000
DEBUG; 2023-06-13-09:49:21; 1686649761.671347; load check is: (( 5 kbps > 10 kbps for download && 85 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:21; 1686649761.672990; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:21; 1686649761.675976; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:21; 1686649761.677591; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:21; 1686649761.680397; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:21; 1686649761.870253; Warning: no reflector response within: 0.250000 seconds. Checking loads.
LOAD; 2023-06-13-09:49:21; 1686649761.870225; 1686649761.869889; 0; 8; 50000; 5000
DEBUG; 2023-06-13-09:49:21; 1686649761.872401; load check is: (( 0 kbps > 10 kbps for download && 51 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:21; 1686649761.874044; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:21; 1686649761.877063; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:21; 1686649761.878771; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:21; 1686649761.881275; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:21; 1686649761.930372; Starting: change_state_maintain_pingers with PID: 14438
DEBUG; 2023-06-13-09:49:21; 1686649761.931524; Changing maintain_pingers state from: PAUSED to: RUNNING
DEBUG; 2023-06-13-09:49:21; 1686649761.933311; no ping response from reflector: 9.9.9.9 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:21; 1686649761.934535; reflector=9.9.9.9, sum_reflector_offences=2 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:21; 1686649761.936149; no ping response from reflector: 9.9.9.10 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:21; 1686649761.937223; reflector=9.9.9.10, sum_reflector_offences=2 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:21; 1686649761.938600; no ping response from reflector: 9.9.9.11 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:21; 1686649761.939814; reflector=9.9.9.11, sum_reflector_offences=2 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:21; 1686649761.941181; no ping response from reflector: 94.140.14.15 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:21; 1686649761.942167; reflector=94.140.14.15, sum_reflector_offences=2 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:21; 1686649761.943483; no ping response from reflector: 94.140.14.140 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:21; 1686649761.944466; reflector=94.140.14.140, sum_reflector_offences=2 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:21; 1686649761.945783; no ping response from reflector: 94.140.14.141 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:21; 1686649761.946758; reflector=94.140.14.141, sum_reflector_offences=2 and reflector_misbehaving_detection_thr=3
LOAD; 2023-06-13-09:49:22; 1686649762.071085; 1686649762.070757; 0; 48; 50000; 5000
DEBUG; 2023-06-13-09:49:22; 1686649762.071107; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:22; 1686649762.072019; load check is: (( 0 kbps > 10 kbps for download && 8 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:22; 1686649762.073035; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:22; 1686649762.077507; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:22; 1686649762.078667; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:22; 1686649762.080661; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:22; 1686649762.271970; Warning: no reflector response within: 0.250000 seconds. Checking loads.
LOAD; 2023-06-13-09:49:22; 1686649762.271946; 1686649762.271617; 6; 10; 50000; 5000
DEBUG; 2023-06-13-09:49:22; 1686649762.272864; load check is: (( 6 kbps > 10 kbps for download && 48 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:22; 1686649762.273893; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:22; 1686649762.278614; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:22; 1686649762.279710; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:22; 1686649762.283318; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:22; 1686649762.472849; Warning: no reflector response within: 0.250000 seconds. Checking loads.
LOAD; 2023-06-13-09:49:22; 1686649762.472823; 1686649762.472483; 0; 18; 50000; 5000
DEBUG; 2023-06-13-09:49:22; 1686649762.473746; load check is: (( 0 kbps > 10 kbps for download && 10 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:22; 1686649762.474777; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:22; 1686649762.480188; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:22; 1686649762.481282; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:22; 1686649762.484879; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:22; 1686649762.673713; Warning: no reflector response within: 0.250000 seconds. Checking loads.
LOAD; 2023-06-13-09:49:22; 1686649762.673686; 1686649762.673351; 5; 34; 50000; 5000
DEBUG; 2023-06-13-09:49:22; 1686649762.674652; load check is: (( 5 kbps > 10 kbps for download && 18 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:22; 1686649762.675601; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:22; 1686649762.679991; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:22; 1686649762.681093; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:22; 1686649762.684657; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:22; 1686649762.874612; 1686649762.874269; 3; 31; 50000; 5000
DEBUG; 2023-06-13-09:49:22; 1686649762.876652; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:22; 1686649762.877539; load check is: (( 3 kbps > 10 kbps for download && 34 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:22; 1686649762.878753; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:22; 1686649762.881197; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:22; 1686649762.882277; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:22; 1686649762.884291; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:22; 1686649762.932667; Starting: change_state_maintain_pingers with PID: 14438
DEBUG; 2023-06-13-09:49:22; 1686649762.934722; Changing maintain_pingers state from: RUNNING to: PAUSED
LOAD; 2023-06-13-09:49:23; 1686649763.075748; 1686649763.075395; 5; 48; 50000; 5000
DEBUG; 2023-06-13-09:49:23; 1686649763.078093; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:23; 1686649763.079063; load check is: (( 5 kbps > 10 kbps for download && 31 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:23; 1686649763.080103; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:23; 1686649763.082587; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:23; 1686649763.083676; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:23; 1686649763.086443; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:23; 1686649763.276330; 1686649763.275986; 0; 24; 50000; 5000
DEBUG; 2023-06-13-09:49:23; 1686649763.278473; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:23; 1686649763.279520; load check is: (( 0 kbps > 10 kbps for download && 48 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:23; 1686649763.280852; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:23; 1686649763.283511; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:23; 1686649763.284603; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:23; 1686649763.286986; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:23; 1686649763.477205; 1686649763.476861; 4; 63; 50000; 5000
DEBUG; 2023-06-13-09:49:23; 1686649763.479753; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:23; 1686649763.480715; load check is: (( 4 kbps > 10 kbps for download && 24 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:23; 1686649763.481734; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:23; 1686649763.484240; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:23; 1686649763.485318; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:23; 1686649763.487877; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:23; 1686649763.678100; 1686649763.677700; 9; 68; 50000; 5000
DEBUG; 2023-06-13-09:49:23; 1686649763.680167; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:23; 1686649763.681061; load check is: (( 9 kbps > 10 kbps for download && 63 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:23; 1686649763.682047; Changing main state from: RUNNING to: STALL
DEBUG; 2023-06-13-09:49:23; 1686649763.684522; Connection stall ended. Resuming normal operation.
DEBUG; 2023-06-13-09:49:23; 1686649763.685533; Changing main state from: STALL to: RUNNING
DEBUG; 2023-06-13-09:49:23; 1686649763.687452; processed response from [9.9.9.11] that is > 500ms old. Skipping.
LOAD; 2023-06-13-09:49:23; 1686649763.878988; 1686649763.878640; 33; 21; 50000; 5000
DEBUG; 2023-06-13-09:49:23; 1686649763.881044; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:23; 1686649763.881924; load check is: (( 33 kbps > 10 kbps for download && 68 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:23; 1686649763.882778; load above connection stall threshold so resuming normal operation.
DEBUG; 2023-06-13-09:49:23; 1686649763.884224; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:23; 1686649763.885086; load check is: (( 33 kbps > 10 kbps for download && 21 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:23; 1686649763.885935; load above connection stall threshold so resuming normal operation.
DEBUG; 2023-06-13-09:49:23; 1686649763.887785; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:23; 1686649763.889722; processed response from [9.9.9.11] that is > 500ms old. Skipping.
DEBUG; 2023-06-13-09:49:23; 1686649763.930716; Starting: change_state_maintain_pingers with PID: 14438
DEBUG; 2023-06-13-09:49:23; 1686649763.931830; Changing maintain_pingers state from: PAUSED to: RUNNING
DEBUG; 2023-06-13-09:49:23; 1686649763.933264; no ping response from reflector: 9.9.9.9 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:23; 1686649763.934208; reflector=9.9.9.9, sum_reflector_offences=3 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:23; 1686649763.935095; Warning: reflector: 9.9.9.9 seems to be misbehaving.
DEBUG; 2023-06-13-09:49:23; 1686649763.936106; Starting: replace_pinger_reflector with PID: 14438
DEBUG; 2023-06-13-09:49:23; 1686649763.936951; replacing reflector: 9.9.9.9 with 94.140.15.15.
DEBUG; 2023-06-13-09:49:23; 1686649763.937806; Starting: kill_pinger with PID: 14438
DEBUG; 2023-06-13-09:49:23; 1686649763.952735; Starting: start_pinger with PID: 14438
DEBUG; 2023-06-13-09:49:23; 1686649763.954105; Resetting reflector offences associated with reflector: 94.140.15.15.
DEBUG; 2023-06-13-09:49:23; 1686649763.960073; no ping response from reflector: 9.9.9.10 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:23; 1686649763.962145; reflector=9.9.9.10, sum_reflector_offences=3 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:23; 1686649763.964025; Warning: reflector: 9.9.9.10 seems to be misbehaving.
DEBUG; 2023-06-13-09:49:23; 1686649763.965818; Warning: skipping replacement of reflector: 9.9.9.10 given prior replacement within this reflector health check cycle.
DEBUG; 2023-06-13-09:49:23; 1686649763.968453; no ping response from reflector: 9.9.9.11 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:23; 1686649763.970322; reflector=9.9.9.11, sum_reflector_offences=3 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:23; 1686649763.972360; Warning: reflector: 9.9.9.11 seems to be misbehaving.
DEBUG; 2023-06-13-09:49:23; 1686649763.974152; Warning: skipping replacement of reflector: 9.9.9.11 given prior replacement within this reflector health check cycle.
DEBUG; 2023-06-13-09:49:23; 1686649763.976559; no ping response from reflector: 94.140.14.15 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:23; 1686649763.978677; reflector=94.140.14.15, sum_reflector_offences=3 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:23; 1686649763.980584; Warning: reflector: 94.140.14.15 seems to be misbehaving.
DEBUG; 2023-06-13-09:49:23; 1686649763.982436; Warning: skipping replacement of reflector: 94.140.14.15 given prior replacement within this reflector health check cycle.
DEBUG; 2023-06-13-09:49:23; 1686649763.984722; no ping response from reflector: 94.140.14.140 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:23; 1686649763.986614; reflector=94.140.14.140, sum_reflector_offences=3 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:23; 1686649763.988631; Warning: reflector: 94.140.14.140 seems to be misbehaving.
DEBUG; 2023-06-13-09:49:23; 1686649763.990465; Warning: skipping replacement of reflector: 94.140.14.140 given prior replacement within this reflector health check cycle.
DEBUG; 2023-06-13-09:49:23; 1686649763.992789; no ping response from reflector: 94.140.14.141 within reflector_response_deadline: 1.0s
DEBUG; 2023-06-13-09:49:23; 1686649763.994706; reflector=94.140.14.141, sum_reflector_offences=3 and reflector_misbehaving_detection_thr=3
DEBUG; 2023-06-13-09:49:23; 1686649763.996581; Warning: reflector: 94.140.14.141 seems to be misbehaving.
DEBUG; 2023-06-13-09:49:23; 1686649763.998499; Warning: skipping replacement of reflector: 94.140.14.141 given prior replacement within this reflector health check cycle.
DEBUG; 2023-06-13-09:49:24; 1686649764.047582; Read in new reflectors: 94.140.15.15 9.9.9.10 9.9.9.11 94.140.14.15 94.140.14.140 94.140.14.141
DEBUG; 2023-06-13-09:49:24; 1686649764.056164; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:24; 1686649764.057197; load check is: (( 33 kbps > 10 kbps for download && 21 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:24; 1686649764.058172; load above connection stall threshold so resuming normal operation.
DEBUG; 2023-06-13-09:49:24; 1686649764.061276; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:24; 1686649764.062172; load check is: (( 33 kbps > 10 kbps for download && 21 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:24; 1686649764.063028; load above connection stall threshold so resuming normal operation.
LOAD; 2023-06-13-09:49:24; 1686649764.080226; 1686649764.079869; 2; 9; 50000; 5000
DEBUG; 2023-06-13-09:49:24; 1686649764.080584; Warning: no reflector response within: 0.250000 seconds. Checking loads.
DEBUG; 2023-06-13-09:49:24; 1686649764.081465; load check is: (( 2 kbps > 10 kbps for download && 21 kbps > 10 kbps for upload ))
DEBUG; 2023-06-13-09:49:24; 1686649764.082497; Changing main state from: RUNNING to: STALL
ERROR; 2023-06-13-09:49:24; 1686649764.084640; /root/cake-autorate/cake-autorate.sh: line 1549: ewma: unbound variable
DEBUG; 2023-06-13-09:49:24; 1686649764.093499; Starting: cleanup_and_killall with PID: 14390
INFO; 2023-06-13-09:49:24; 1686649764.094545; Stopping cake-autorate with PID: 14390 and config: /root/cake-autorate/cake-autorate_config.primary.sh
INFO; 2023-06-13-09:49:24; 1686649764.095510; Killing all background processes and cleaning up temporary files.
DEBUG; 2023-06-13-09:49:24; 1686649764.279646; Terminating monitor_achieved_rates.
DEBUG; 2023-06-13-09:49:24; 1686649764.934317; Starting: change_state_maintain_pingers with PID: 14438
DEBUG; 2023-06-13-09:49:24; 1686649764.935880; Changing maintain_pingers state from: RUNNING to: PAUSED
DEBUG; 2023-06-13-09:49:25; 1686649765.108531; Starting: kill_maintain_log_file with PID: 14416
DEBUG; 2023-06-13-09:49:25; 1686649765.108758; Starting: kill_maintain_pingers with PID: 14438
DEBUG; 2023-06-13-09:49:25; 1686649765.109543; Starting: flush_log_fd with PID: 14416
DEBUG; 2023-06-13-09:49:25; 1686649765.109524; Terminating maintain_pingers.
lynxthecat commented 1 year ago

Thanks for this.

Does 'logread | grep ERROR' reveal any older errors?

For some reason the present ewma passed to the function 'ewma_iteration' is not set - see this line:

https://github.com/lynxthecat/cake-autorate/blob/c4f196a2a436c9d14ad8f1bf336d09400e0f43b6/cake-autorate.sh#L1549

You are using 'tsping' right? If so, this crash is caused by one of the four 'ewma_iteration' calls inside parse_tsping():

https://github.com/lynxthecat/cake-autorate/blob/c4f196a2a436c9d14ad8f1bf336d09400e0f43b6/cake-autorate.sh#LL671C84-L671C84

Otherwise it is caused by one of the four corresponding 'ewma_iteration' calls inside parse_fping() or parse_ping().

Please can you add this log_msg line before the read here:

https://github.com/lynxthecat/cake-autorate/blob/c4f196a2a436c9d14ad8f1bf336d09400e0f43b6/cake-autorate.sh#LL573C81-L573C98

log_msg "DEBUG" "${command[@]:1}"
read -r timestamp reflector seq _ _ _ _ _ dl_owd_ms ul_owd_ms checksum <<< "${command[@]:1}"

And please can you add log_msg lines before the 'ewma_iteration' calls as follows:

log_msg "DEBUG" "reflector is ${reflector}; ewma_iteration ${dl_owd_us} ${dl_alpha} dl_owd_baselines_us[${reflector}]"
ewma_iteration "${dl_owd_us}" "${dl_alpha}" "dl_owd_baselines_us[${reflector}]"
log_msg "DEBUG"  "reflector is ${reflector}; ewma_iteration ${ul_owd_us} ${ul_alpha} ul_owd_baselines_us[${reflector}]"
ewma_iteration "${ul_owd_us}" "${ul_alpha}" "ul_owd_baselines_us[${reflector}]"

and

log_msg "DEBUG" "reflector is ${reflector}; ewma_iteration ${dl_owd_delta_us} ${alpha_delta_ewma} dl_owd_delta_ewmas_us[${reflector}]"
ewma_iteration "${dl_owd_delta_us}" "${alpha_delta_ewma}" "dl_owd_delta_ewmas_us[${reflector}]"
log_msg "DEBUG" "reflector is ${reflector}; ${ul_owd_delta_us} ${alpha_delta_ewma} ul_owd_delta_ewmas_us[${reflector}]"
ewma_iteration "${ul_owd_delta_us}" "${alpha_delta_ewma}" "ul_owd_delta_ewmas_us[${reflector}]"

And verify that these are indeed getting properly printed in the log file? If you could post a snippet log sample below to verify that would be great.

Then we need to wait for the crash to reoccur and hopefully the log lines will reveal the cause.

Also, can you please upload your config file?

bairhys commented 1 year ago

Hey Lynx, thanks for looking into this

root@OpenWrt:~# logread | grep ERROR
Mon Jun 12 01:49:55 2023 user.notice cake-autorate.primary: ERROR: 1686534595.499049 /root/cake-autorate/cake-autorate.sh: line 1549: ewma: unbound variable
Mon Jun 12 03:08:59 2023 user.notice cake-autorate.primary: ERROR: 1686539339.376265 /root/cake-autorate/cake-autorate.sh: line 1549: ewma: unbound variable
Mon Jun 12 22:22:35 2023 user.notice cake-autorate.primary: ERROR: 1686608555.573483 /root/cake-autorate/cake-autorate.sh: line 1549: ewma: unbound variable
Tue Jun 13 09:49:24 2023 user.notice cake-autorate.primary: ERROR: 1686649764.084640 /root/cake-autorate/cake-autorate.sh: line 1549: ewma: unbound variable

Here is my modified cake-autorate.sh file changes:

  1. https://gist.github.com/bairhys/b07fed659e08e4e3ec909599f1009981#file-cake-autorate-sh-L573-L574

  2. https://gist.github.com/bairhys/b07fed659e08e4e3ec909599f1009981#file-cake-autorate-sh-L672-L675

  3. https://gist.github.com/bairhys/b07fed659e08e4e3ec909599f1009981#file-cake-autorate-sh-L692-L695

Looks like the changes are added to the log, here is a sample of the log while running cake-autorate.sh manually: https://gist.github.com/bairhys/6c748b4b2e62abd0bec88901391b1bcd

It is running as a service now so I'll get back to you if it crashes

My config

root@OpenWrt:~# cat cake-autorate/cake-autorate_config.primary.sh
#!/bin/bash

# *** STANDARD CONFIGURATION OPTIONS ***

### For multihomed setups, it is the responsibility of the user to ensure that the probes
### sent by this instance of cake-autorate actually travel through these interfaces.
### See ping_extra_args and ping_prefix_string

output_processing_stats=1       # enable (1) or disable (0) output monitoring lines showing processing stats
output_load_stats=1             # enable (1) or disable (0) output monitoring lines showing achieved loads
output_reflector_stats=1        # enable (1) or disable (0) output monitoring lines showing reflector stats
output_cake_changes=1           # enable (1) or disable (0) output monitoring lines showing cake bandwidth changes
debug=1                         # enable (1) or disable (0) out of debug lines

#log_to_file=1
# log file path defaults to /var/log/
# or, if set below, then ${log_file_path_override}
#log_file_path_override="/var/log/cake-autorate.primary.log"

dl_if=ifb4wan # download interface
ul_if=wan     # upload interface

# Set either of the below to 0 to adjust one direction only
# or alternatively set both to 0 to simply use cake-autorate to monitor a connection
adjust_dl_shaper_rate=1 # enable (1) or disable (0) actually changing the dl shaper rate
adjust_ul_shaper_rate=1 # enable (1) or disable (0) actually changing the ul shaper rate

min_dl_shaper_rate_kbps=30000  # minimum bandwidth for download (Kbit/s)
base_dl_shaper_rate_kbps=50000 # steady state bandwidth for download (Kbit/s)
max_dl_shaper_rate_kbps=100000  # maximum bandwidth for download (Kbit/s)

min_ul_shaper_rate_kbps=4000  # minimum bandwidth for upload (Kbit/s)
base_ul_shaper_rate_kbps=5000 # steady state bandwidth for upload (KBit/s)
max_ul_shaper_rate_kbps=15000  # maximum bandwidth for upload (Kbit/s)

# *** OVERRIDES ***

randomize_reflectors=0

pinger_binary=tsping

reflectors=(
"9.9.9.9" "9.9.9.10" "9.9.9.11" # Quad9
"94.140.14.15" "94.140.14.140" "94.140.14.141" "94.140.15.15" "94.140.15.16" # AdGuard
"64.6.65.6" "156.154.70.1" "156.154.70.2" "156.154.70.3" "156.154.70.4" "156.154.70.5" "156.154.71.1" "156.154.71.2" "156.154.71.3" "156.154.71.4" "156.154.71.5" # Neustar
"208.67.220.2" "208.67.220.123" "208.67.220.220" "208.67.222.2" "208.67.222.123" # OpenDNS
"185.228.168.9" "185.228.168.10" # CleanBrowsing
)

dl_delay_thr_ms=30 # (milliseconds)
ul_delay_thr_ms=30 # (milliseconds)

### See cake-autorate_defaults.sh for additional configuration options
### that can be set in this configuration file to override the defaults.
### Place any such overrides below this line.

# *** DO NOT EDIT BELOW THIS LINE ***

config_file_check="cake-autorate"
lynxthecat commented 1 year ago

Oops, the reflector response lines are not getting printed properly (first change).

Can you replace the character @ with the character * in this line:

log_msg "DEBUG" "${command[@]:1}"

and restart? And paste new log samples?

bairhys commented 1 year ago

Done. Is this ok?

root@OpenWrt:~/cake-autorate# ./cake-autorate.sh
DEBUG; 2023-06-14-13:06:02; 1686747962.538439; Starting: rotate_log_file with PID: 29445
DEBUG; 2023-06-14-13:06:02; 1686747962.546279; Starting: print_headers with PID: 29445
DATA_HEADER; LOG_DATETIME; LOG_TIMESTAMP; PROC_TIME_US; DL_ACHIEVED_RATE_KBPS; UL_ACHIEVED_RATE_KBPS; DL_LOAD_PERCENT; UL_LOAD_PERCENT; RTT_TIMESTAMP; REFLECTOR; SEQUENCE; DL_OWD_BASELINE; DL_OWD_US; DL_OWD_DELTA_EWMA_US; DL_OWD_DELTA_US; DL_ADJ_DELAY_THR; UL_OWD_BASELINE; UL_OWD_US; UL_OWD_DELTA_EWMA_US; UL_OWD_DELTA_US; UL_ADJ_DELAY_THR; SUM_DL_DELAYS; SUM_UL_DELAYS; DL_LOAD_CONDITION; UL_LOAD_CONDITION; CAKE_DL_RATE_KBPS; CAKE_UL_RATE_KBPS
LOAD_HEADER; LOG_DATETIME; LOG_TIMESTAMP; PROC_TIME_US; DL_ACHIEVED_RATE_KBPS; UL_ACHIEVED_RATE_KBPS; CAKE_DL_RATE_KBPS; CAKE_UL_RATE_KBPS
REFLECTOR_HEADER; LOG_DATETIME; LOG_TIMESTAMP; PROC_TIME_US; REFLECTOR; MIN_SUM_OWD_BASELINES_US; SUM_OWD_BASELINES_US; SUM_OWD_BASELINES_DELTA_US; SUM_OWD_BASELINES_DELTA_THR_US; MIN_DL_DELTA_EWMA_US; DL_DELTA_EWMA_US; DL_DELTA_EWMA_DELTA_US; DL_DELTA_EWMA_DELTA_THR; MIN_UL_DELTA_EWMA_US; UL_DELTA_EWMA_US; UL_DELTA_EWMA_DELTA_US; UL_DELTA_EWMA_DELTA_THR
SYSLOG; 2023-06-14-13:06:02; 1686747962.556893; Starting cake-autorate with PID: 29445 and config: /root/cake-autorate/cake-autorate_config.primary.sh
DEBUG; 2023-06-14-13:06:02; 1686747962.567730; Starting: maintain_log_file with PID: 29458
DEBUG; 2023-06-14-13:06:02; 1686747962.568164; CAKE-autorate version: 2.0.0
DEBUG; 2023-06-14-13:06:02; 1686747962.570465; config_path: /root/cake-autorate/cake-autorate_config.primary.sh
DEBUG; 2023-06-14-13:06:02; 1686747962.572548; run_path: /var/run/cake-autorate/primary
DEBUG; 2023-06-14-13:06:02; 1686747962.574445; log_file_path: /var/log/cake-autorate.primary.log
DEBUG; 2023-06-14-13:06:02; 1686747962.576449; pinger_binary:tsping
DEBUG; 2023-06-14-13:06:02; 1686747962.578027; download interface: ifb4wan (30000 / 50000 / 100000)
DEBUG; 2023-06-14-13:06:02; 1686747962.579595; upload interface: wan (4000 / 5000 / 15000)
DEBUG; 2023-06-14-13:06:02; 1686747962.581033; rx_bytes_path: /sys/class/net/ifb4wan/statistics/tx_bytes
DEBUG; 2023-06-14-13:06:02; 1686747962.582594; tx_bytes_path: /sys/class/net/wan/statistics/tx_bytes
DEBUG; 2023-06-14-13:06:02; 1686747962.584192; Starting: verify_ifs_up with PID: 29445
SHAPER; 2023-06-14-13:06:02; 1686747962.601508; tc qdisc change root dev ifb4wan cake bandwidth 50000Kbit
SHAPER; 2023-06-14-13:06:02; 1686747962.607721; tc qdisc change root dev wan cake bandwidth 5000Kbit
DEBUG; 2023-06-14-13:06:02; 1686747962.624294; Starting: monitor_achieved_rates with PID: 29469
DEBUG; 2023-06-14-13:06:02; 1686747962.626644; Starting: maintain_pingers with PID: 29470
LOAD; 2023-06-14-13:06:02; 1686747962.630496; 1686747962.630144; 0; 0; 50000; 5000
INFO; 2023-06-14-13:06:02; 1686747962.641152; Started cake-autorate with PID: 29445 and config: /root/cake-autorate/cake-autorate_config.primary.sh
DEBUG; 2023-06-14-13:06:02; 1686747962.655995; Starting: parse_tsping with PID: 29474
DEBUG; 2023-06-14-13:06:02; 1686747962.655623; Starting pingers.
DEBUG; 2023-06-14-13:06:02; 1686747962.658419; Starting: start_pinger with PID: 29470
DEBUG; 2023-06-14-13:06:02; 1686747962.660690; Starting: change_state_maintain_pingers with PID: 29470
DEBUG; 2023-06-14-13:06:02; 1686747962.662947; Changing maintain_pingers state from: START to: RUNNING
DEBUG; 2023-06-14-13:06:02; 1686747962.698210; 1686747962.696974 9.9.9.9 0 47162668 47162680 47162680 47162696 28 16 12 1686747962.696974
DEBUG; 2023-06-14-13:06:02; 1686747962.701078; reflector is 9.9.9.9; ewma_iteration 16000 900000 dl_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:02; 1686747962.703147; reflector is 9.9.9.9; ewma_iteration 12000 900000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:02; 1686747962.705205; reflector is 9.9.9.9; ewma_iteration -8400 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:02; 1686747962.707190; reflector is 9.9.9.9; -8800 95000 ul_owd_delta_ewmas_us[9.9.9.9]
DATA; 2023-06-14-13:06:02; 1686747962.713977; 1686747962.713237; 0; 0; 0; 0; 1686747962.696974; 9.9.9.9; 0; 24400; 16000; -798; -8400; 30240; 20800; 12000; -836; -8800; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:02; 1686747962.743371; 1686747962.742071 9.9.9.10 0 47162719 47162726 47162726 47162742 23 16 7 1686747962.742071
DEBUG; 2023-06-14-13:06:02; 1686747962.746828; reflector is 9.9.9.10; ewma_iteration 16000 900000 dl_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:02; 1686747962.749591; reflector is 9.9.9.10; ewma_iteration 7000 900000 ul_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:02; 1686747962.752535; reflector is 9.9.9.10; ewma_iteration -8400 95000 dl_owd_delta_ewmas_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:02; 1686747962.755139; reflector is 9.9.9.10; -9300 95000 ul_owd_delta_ewmas_us[9.9.9.10]
DATA; 2023-06-14-13:06:02; 1686747962.761636; 1686747962.760896; 0; 0; 0; 0; 1686747962.742071; 9.9.9.10; 0; 24400; 16000; -798; -8400; 30240; 16300; 7000; -883; -9300; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:02; 1686747962.793334; 1686747962.792175 9.9.9.11 0 47162769 47162776 47162776 47162792 23 16 7 1686747962.792175
DEBUG; 2023-06-14-13:06:02; 1686747962.796210; reflector is 9.9.9.11; ewma_iteration 16000 900000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:02; 1686747962.798603; reflector is 9.9.9.11; ewma_iteration 7000 900000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:02; 1686747962.801096; reflector is 9.9.9.11; ewma_iteration -8400 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:02; 1686747962.803428; reflector is 9.9.9.11; -9300 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DATA; 2023-06-14-13:06:02; 1686747962.809568; 1686747962.808825; 0; 0; 0; 0; 1686747962.792175; 9.9.9.11; 0; 24400; 16000; -798; -8400; 30240; 16300; 7000; -883; -9300; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-14-13:06:02; 1686747962.829821; 1686747962.829469; 411; 121; 50000; 5000
DEBUG; 2023-06-14-13:06:02; 1686747962.831876; reflector is 9.9.9.11; ewma_iteration 16000 900000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:02; 1686747962.833962; reflector is 9.9.9.11; ewma_iteration 7000 900000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:02; 1686747962.836212; reflector is 9.9.9.11; ewma_iteration -840 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:02; 1686747962.838502; reflector is 9.9.9.11; -930 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DATA; 2023-06-14-13:06:02; 1686747962.844229; 1686747962.843469; 411; 121; 0; 2; 1686747962.792175; 9.9.9.11; 0; 16840; 16000; -801; -840; 30240; 7930; 7000; -887; -930; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:02; 1686747962.847375; reflector is 9.9.9.11; ewma_iteration 16000 900000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:02; 1686747962.849821; reflector is 9.9.9.11; ewma_iteration 7000 900000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:02; 1686747962.852325; reflector is 9.9.9.11; ewma_iteration -84 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:02; 1686747962.854661; reflector is 9.9.9.11; -93 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DATA; 2023-06-14-13:06:02; 1686747962.860893; 1686747962.860108; 411; 121; 0; 2; 1686747962.792175; 9.9.9.11; 0; 16084; 16000; -732; -84; 30240; 7093; 7000; -811; -93; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:02; 1686747962.864286; 1686747962.863153 94.140.14.15 0 47162819 47162823 47162823 47162863 44 40 4 1686747962.863153
DEBUG; 2023-06-14-13:06:02; 1686747962.867192; reflector is 94.140.14.15; ewma_iteration 40000 900000 dl_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:02; 1686747962.869724; reflector is 94.140.14.15; ewma_iteration 4000 900000 ul_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:02; 1686747962.872278; reflector is 94.140.14.15; ewma_iteration -6000 95000 dl_owd_delta_ewmas_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:02; 1686747962.874610; reflector is 94.140.14.15; -9600 95000 ul_owd_delta_ewmas_us[94.140.14.15]
DATA; 2023-06-14-13:06:02; 1686747962.880969; 1686747962.880222; 411; 121; 0; 2; 1686747962.863153; 94.140.14.15; 0; 46000; 40000; -570; -6000; 30240; 13600; 4000; -912; -9600; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:02; 1686747962.914067; 1686747962.912882 94.140.14.140 0 47162869 47162873 47162873 47162912 43 39 4 1686747962.912882
DEBUG; 2023-06-14-13:06:02; 1686747962.916989; reflector is 94.140.14.140; ewma_iteration 39000 900000 dl_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:02; 1686747962.919449; reflector is 94.140.14.140; ewma_iteration 4000 900000 ul_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:02; 1686747962.922021; reflector is 94.140.14.140; ewma_iteration -6100 95000 dl_owd_delta_ewmas_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:02; 1686747962.924375; reflector is 94.140.14.140; -9600 95000 ul_owd_delta_ewmas_us[94.140.14.140]
DATA; 2023-06-14-13:06:02; 1686747962.935376; 1686747962.934023; 411; 121; 0; 2; 1686747962.912882; 94.140.14.140; 0; 45100; 39000; -579; -6100; 30240; 13600; 4000; -912; -9600; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:02; 1686747962.961889; 1686747962.960641 94.140.14.141 0 47162919 47162918 47162918 47162960 41 42 -1 1686747962.960641
DEBUG; 2023-06-14-13:06:02; 1686747962.965421; reflector is 94.140.14.141; ewma_iteration 42000 900000 dl_owd_baselines_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:02; 1686747962.968973; reflector is 94.140.14.141; ewma_iteration -1000 900000 ul_owd_baselines_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:02; 1686747962.972869; reflector is 94.140.14.141; ewma_iteration -5800 95000 dl_owd_delta_ewmas_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:02; 1686747962.976320; reflector is 94.140.14.141; -10100 95000 ul_owd_delta_ewmas_us[94.140.14.141]
DATA; 2023-06-14-13:06:02; 1686747962.984290; 1686747962.983527; 411; 121; 0; 2; 1686747962.960641; 94.140.14.141; 0; 47800; 42000; -551; -5800; 30240; 9100; -1000; -959; -10100; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:02; 1686747962.993082; 1686747962.991502 9.9.9.9 1 47162970 47162975 47162975 47162991 21 16 5 1686747962.991502
DEBUG; 2023-06-14-13:06:02; 1686747962.997502; reflector is 9.9.9.9; ewma_iteration 16000 900000 dl_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.000939; reflector is 9.9.9.9; ewma_iteration 5000 900000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.004933; reflector is 9.9.9.9; ewma_iteration -840 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.008451; reflector is 9.9.9.9; -1580 95000 ul_owd_delta_ewmas_us[9.9.9.9]
DATA; 2023-06-14-13:06:03; 1686747963.016263; 1686747963.015519; 411; 121; 0; 2; 1686747962.991502; 9.9.9.9; 1; 16840; 16000; -801; -840; 30240; 6580; 5000; -906; -1580; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-14-13:06:03; 1686747963.030882; 1686747963.030530; 272; 104; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.035220; reflector is 9.9.9.9; ewma_iteration 16000 900000 dl_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.039033; reflector is 9.9.9.9; ewma_iteration 5000 900000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.043587; reflector is 9.9.9.9; ewma_iteration -84 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.047136; reflector is 9.9.9.9; -158 95000 ul_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.052763; reflector is 9.9.9.9; ewma_iteration 16000 900000 dl_owd_baselines_us[9.9.9.9]
DATA; 2023-06-14-13:06:03; 1686747963.055460; 1686747963.054166; 272; 104; 0; 2; 1686747962.991502; 9.9.9.9; 1; 16084; 16000; -732; -84; 30240; 5158; 5000; -834; -158; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.055218; reflector is 9.9.9.9; ewma_iteration 5000 900000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.059108; reflector is 9.9.9.9; ewma_iteration -8 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.062877; reflector is 9.9.9.9; -15 95000 ul_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.067796; 1686747963.041775 9.9.9.10 1 47163020 47163026 47163026 47163041 21 15 6 1686747963.041775
DATA; 2023-06-14-13:06:03; 1686747963.071478; 1686747963.070364; 272; 104; 0; 2; 1686747962.991502; 9.9.9.9; 1; 16008; 16000; -663; -8; 30240; 5015; 5000; -756; -15; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.071023; reflector is 9.9.9.10; ewma_iteration 15000 900000 dl_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:03; 1686747963.074937; reflector is 9.9.9.10; ewma_iteration 6000 900000 ul_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:03; 1686747963.078781; reflector is 9.9.9.10; ewma_iteration -940 95000 dl_owd_delta_ewmas_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:03; 1686747963.082391; reflector is 9.9.9.10; -1030 95000 ul_owd_delta_ewmas_us[9.9.9.10]
DATA; 2023-06-14-13:06:03; 1686747963.090193; 1686747963.089427; 272; 104; 0; 2; 1686747963.041775; 9.9.9.10; 1; 15940; 15000; -811; -940; 30240; 7030; 6000; -896; -1030; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.106220; 1686747963.104763 9.9.9.11 1 47163070 47163086 47163086 47163104 34 18 16 1686747963.104763
DEBUG; 2023-06-14-13:06:03; 1686747963.110640; reflector is 9.9.9.11; ewma_iteration 18000 1000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.114693; reflector is 9.9.9.11; ewma_iteration 16000 1000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.119546; reflector is 9.9.9.11; ewma_iteration 1915 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.123819; reflector is 9.9.9.11; 8899 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DATA; 2023-06-14-13:06:03; 1686747963.132033; 1686747963.131273; 272; 104; 0; 2; 1686747963.104763; 9.9.9.11; 1; 16085; 18000; -480; 1915; 30240; 7101; 16000; 111; 8899; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.164495; 1686747963.163164 94.140.14.15 1 47163121 47163123 47163123 47163163 42 40 2 1686747963.163164
DEBUG; 2023-06-14-13:06:03; 1686747963.169176; reflector is 94.140.14.15; ewma_iteration 40000 900000 dl_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:03; 1686747963.172717; reflector is 94.140.14.15; ewma_iteration 2000 900000 ul_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:03; 1686747963.176697; reflector is 94.140.14.15; ewma_iteration -600 95000 dl_owd_delta_ewmas_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:03; 1686747963.180990; reflector is 94.140.14.15; -1160 95000 ul_owd_delta_ewmas_us[94.140.14.15]
DATA; 2023-06-14-13:06:03; 1686747963.187635; 1686747963.186873; 272; 104; 0; 2; 1686747963.163164; 94.140.14.15; 1; 40600; 40000; -572; -600; 30240; 3160; 2000; -935; -1160; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.210294; 1686747963.208644 94.140.14.140 1 47163171 47163168 47163168 47163208 37 40 -3 1686747963.208644
DEBUG; 2023-06-14-13:06:03; 1686747963.213428; reflector is 94.140.14.140; ewma_iteration 40000 900000 dl_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.215941; reflector is 94.140.14.140; ewma_iteration -3000 900000 ul_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.218673; reflector is 94.140.14.140; ewma_iteration -510 95000 dl_owd_delta_ewmas_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.221191; reflector is 94.140.14.140; -1660 95000 ul_owd_delta_ewmas_us[94.140.14.140]
DATA; 2023-06-14-13:06:03; 1686747963.227747; 1686747963.226983; 272; 104; 0; 2; 1686747963.208644; 94.140.14.140; 1; 40510; 40000; -572; -510; 30240; -1340; -3000; -983; -1660; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-14-13:06:03; 1686747963.231584; 1686747963.231234; 351; 102; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.233818; reflector is 94.140.14.140; ewma_iteration 40000 900000 dl_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.236445; reflector is 94.140.14.140; ewma_iteration -3000 900000 ul_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.239163; reflector is 94.140.14.140; ewma_iteration -51 95000 dl_owd_delta_ewmas_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.241721; reflector is 94.140.14.140; -166 95000 ul_owd_delta_ewmas_us[94.140.14.140]
DATA; 2023-06-14-13:06:03; 1686747963.248873; 1686747963.247564; 351; 102; 0; 2; 1686747963.208644; 94.140.14.140; 1; 40051; 40000; -522; -51; 30240; -2834; -3000; -905; -166; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.253694; reflector is 94.140.14.140; ewma_iteration 40000 900000 dl_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.257012; reflector is 94.140.14.140; ewma_iteration -3000 900000 ul_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.259749; reflector is 94.140.14.140; ewma_iteration -5 95000 dl_owd_delta_ewmas_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.262054; reflector is 94.140.14.140; -17 95000 ul_owd_delta_ewmas_us[94.140.14.140]
DATA; 2023-06-14-13:06:03; 1686747963.268252; 1686747963.267402; 351; 102; 0; 2; 1686747963.208644; 94.140.14.140; 1; 40005; 40000; -472; -5; 30240; -2983; -3000; -820; -17; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.269387; 1686747963.256336 94.140.14.141 1 47163221 47163218 47163218 47163256 35 38 -3 1686747963.256336
DEBUG; 2023-06-14-13:06:03; 1686747963.272185; reflector is 94.140.14.141; ewma_iteration 38000 900000 dl_owd_baselines_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:03; 1686747963.274064; reflector is 94.140.14.141; ewma_iteration -3000 900000 ul_owd_baselines_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:03; 1686747963.276094; reflector is 94.140.14.141; ewma_iteration -980 95000 dl_owd_delta_ewmas_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:03; 1686747963.278420; reflector is 94.140.14.141; -1210 95000 ul_owd_delta_ewmas_us[94.140.14.141]
DATA; 2023-06-14-13:06:03; 1686747963.284076; 1686747963.283315; 351; 102; 0; 2; 1686747963.256336; 94.140.14.141; 1; 38980; 38000; -591; -980; 30240; -1790; -3000; -982; -1210; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.306443; 1686747963.305234 9.9.9.9 2 47163271 47163290 47163290 47163305 34 15 19 1686747963.305234
DEBUG; 2023-06-14-13:06:03; 1686747963.309026; reflector is 9.9.9.9; ewma_iteration 15000 900000 dl_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.310856; reflector is 9.9.9.9; ewma_iteration 19000 1000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.312839; reflector is 9.9.9.9; ewma_iteration -100 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.314681; reflector is 9.9.9.9; 13972 95000 ul_owd_delta_ewmas_us[9.9.9.9]
DATA; 2023-06-14-13:06:03; 1686747963.320592; 1686747963.319829; 351; 102; 0; 2; 1686747963.305234; 9.9.9.9; 2; 15100; 15000; -609; -100; 30240; 5028; 19000; 643; 13972; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.353017; 1686747963.351849 9.9.9.10 2 47163321 47163335 47163335 47163351 30 16 14 1686747963.351849
DEBUG; 2023-06-14-13:06:03; 1686747963.355479; reflector is 9.9.9.10; ewma_iteration 16000 1000 dl_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:03; 1686747963.357308; reflector is 9.9.9.10; ewma_iteration 14000 1000 ul_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:03; 1686747963.359440; reflector is 9.9.9.10; ewma_iteration 60 95000 dl_owd_delta_ewmas_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:03; 1686747963.361288; reflector is 9.9.9.10; 6964 95000 ul_owd_delta_ewmas_us[9.9.9.10]
DATA; 2023-06-14-13:06:03; 1686747963.367142; 1686747963.366111; 351; 102; 0; 2; 1686747963.351849; 9.9.9.10; 2; 15940; 16000; -728; 60; 30240; 7036; 14000; -149; 6964; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.404052; 1686747963.401607 9.9.9.11 2 47163372 47163386 47163386 47163401 29 15 14 1686747963.401607
DEBUG; 2023-06-14-13:06:03; 1686747963.407594; reflector is 9.9.9.11; ewma_iteration 15000 900000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.410244; reflector is 9.9.9.11; ewma_iteration 14000 1000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.413010; reflector is 9.9.9.11; ewma_iteration -108 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.415645; reflector is 9.9.9.11; 6893 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DATA; 2023-06-14-13:06:03; 1686747963.422802; 1686747963.421764; 351; 102; 0; 2; 1686747963.401607; 9.9.9.11; 2; 15108; 15000; -444; -108; 30240; 7107; 14000; 755; 6893; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-14-13:06:03; 1686747963.432670; 1686747963.432315; 326; 140; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.435006; reflector is 9.9.9.11; ewma_iteration 15000 900000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.439252; reflector is 9.9.9.11; ewma_iteration 14000 1000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.441836; reflector is 9.9.9.11; ewma_iteration -10 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.444466; reflector is 9.9.9.11; 6887 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.449126; reflector is 9.9.9.11; ewma_iteration 15000 900000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.451065; reflector is 9.9.9.11; ewma_iteration 14000 1000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.453145; reflector is 9.9.9.11; ewma_iteration -1 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DATA; 2023-06-14-13:06:03; 1686747963.453127; 1686747963.451496; 326; 140; 0; 2; 1686747963.401607; 9.9.9.11; 2; 15010; 15000; -402; -10; 30240; 7113; 14000; 1337; 6887; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.454977; reflector is 9.9.9.11; 6881 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.460350; 1686747963.457193 94.140.14.15 2 47163422 47163418 47163418 47163457 35 39 -4 1686747963.457193
DEBUG; 2023-06-14-13:06:03; 1686747963.463133; reflector is 94.140.14.15; ewma_iteration 39000 900000 dl_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:03; 1686747963.465016; reflector is 94.140.14.15; ewma_iteration -4000 900000 ul_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:03; 1686747963.467097; reflector is 94.140.14.15; ewma_iteration -160 95000 dl_owd_delta_ewmas_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:03; 1686747963.469062; reflector is 94.140.14.15; -716 95000 ul_owd_delta_ewmas_us[94.140.14.15]
DATA; 2023-06-14-13:06:03; 1686747963.468795; 1686747963.467173; 326; 140; 0; 2; 1686747963.401607; 9.9.9.11; 2; 15001; 15000; -363; -1; 30240; 7119; 14000; 1863; 6881; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DATA; 2023-06-14-13:06:03; 1686747963.477885; 1686747963.477092; 326; 140; 0; 2; 1686747963.457193; 94.140.14.15; 2; 39160; 39000; -532; -160; 30240; -3284; -4000; -914; -716; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.523165; 1686747963.521866 94.140.14.140 2 47163472 47163483 47163483 47163521 49 38 11 1686747963.521866
DEBUG; 2023-06-14-13:06:03; 1686747963.525470; reflector is 94.140.14.140; ewma_iteration 38000 900000 dl_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.527209; reflector is 94.140.14.140; ewma_iteration 11000 1000 ul_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.530384; reflector is 94.140.14.140; ewma_iteration -200 95000 dl_owd_delta_ewmas_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.532657; reflector is 94.140.14.140; 13969 95000 ul_owd_delta_ewmas_us[94.140.14.140]
DATA; 2023-06-14-13:06:03; 1686747963.539424; 1686747963.538584; 326; 140; 0; 2; 1686747963.521866; 94.140.14.140; 2; 38200; 38000; -446; -200; 30240; -2969; 11000; 584; 13969; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.569021; 1686747963.566892 94.140.14.141 2 47163523 47163528 47163528 47163566 43 38 5 1686747963.566892
DEBUG; 2023-06-14-13:06:03; 1686747963.573268; reflector is 94.140.14.141; ewma_iteration 38000 900000 dl_owd_baselines_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:03; 1686747963.576630; reflector is 94.140.14.141; ewma_iteration 5000 1000 ul_owd_baselines_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:03; 1686747963.579829; reflector is 94.140.14.141; ewma_iteration -98 95000 dl_owd_delta_ewmas_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:03; 1686747963.585070; reflector is 94.140.14.141; 6783 95000 ul_owd_delta_ewmas_us[94.140.14.141]
DATA; 2023-06-14-13:06:03; 1686747963.592760; 1686747963.591969; 326; 140; 0; 2; 1686747963.566892; 94.140.14.141; 2; 38098; 38000; -544; -98; 30240; -1783; 5000; -244; 6783; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.613586; 1686747963.612294 9.9.9.9 3 47163573 47163595 47163595 47163612 39 17 22 1686747963.612294
DEBUG; 2023-06-14-13:06:03; 1686747963.616270; reflector is 9.9.9.9; ewma_iteration 17000 1000 dl_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.618563; reflector is 9.9.9.9; ewma_iteration 22000 1000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.621134; reflector is 9.9.9.9; ewma_iteration 1899 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.624014; reflector is 9.9.9.9; 16956 95000 ul_owd_delta_ewmas_us[9.9.9.9]
LOAD; 2023-06-14-13:06:03; 1686747963.634267; 1686747963.633917; 354; 155; 50000; 5000
DATA; 2023-06-14-13:06:03; 1686747963.632009; 1686747963.631155; 326; 140; 0; 2; 1686747963.612294; 9.9.9.9; 3; 15101; 17000; -370; 1899; 30240; 5044; 22000; 2192; 16956; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.638710; reflector is 9.9.9.9; ewma_iteration 17000 1000 dl_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.640621; reflector is 9.9.9.9; ewma_iteration 22000 1000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.642628; reflector is 9.9.9.9; ewma_iteration 1898 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.644476; reflector is 9.9.9.9; 16940 95000 ul_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.648385; reflector is 9.9.9.9; ewma_iteration 17000 1000 dl_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.650387; reflector is 9.9.9.9; ewma_iteration 22000 1000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.653100; reflector is 9.9.9.9; ewma_iteration 1897 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DATA; 2023-06-14-13:06:03; 1686747963.652710; 1686747963.651683; 354; 155; 0; 3; 1686747963.612294; 9.9.9.9; 3; 15102; 17000; -154; 1898; 30240; 5060; 22000; 3593; 16940; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.663968; reflector is 9.9.9.9; 16924 95000 ul_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.666983; 1686747963.650971 9.9.9.10 3 47163623 47163636 47163636 47163650 27 14 13 1686747963.650971
DATA; 2023-06-14-13:06:03; 1686747963.670870; 1686747963.670095; 354; 155; 0; 3; 1686747963.612294; 9.9.9.9; 3; 15103; 17000; 40; 1897; 30240; 5076; 22000; 4859; 16924; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.678908; reflector is 9.9.9.10; ewma_iteration 14000 900000 dl_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:03; 1686747963.680504; reflector is 9.9.9.10; ewma_iteration 13000 1000 ul_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:03; 1686747963.682233; reflector is 9.9.9.10; ewma_iteration -194 95000 dl_owd_delta_ewmas_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:03; 1686747963.683870; reflector is 9.9.9.10; 5959 95000 ul_owd_delta_ewmas_us[9.9.9.10]
DATA; 2023-06-14-13:06:03; 1686747963.689961; 1686747963.689168; 354; 155; 0; 3; 1686747963.650971; 9.9.9.10; 3; 14194; 14000; -677; -194; 30240; 7041; 13000; 431; 5959; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.698753; 1686747963.697556 9.9.9.11 3 47163673 47163681 47163681 47163697 24 16 8 1686747963.697556
DEBUG; 2023-06-14-13:06:03; 1686747963.700910; reflector is 9.9.9.11; ewma_iteration 16000 1000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.702495; reflector is 9.9.9.11; ewma_iteration 8000 1000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.704231; reflector is 9.9.9.11; ewma_iteration 999 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:03; 1686747963.705821; reflector is 9.9.9.11; 881 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DATA; 2023-06-14-13:06:03; 1686747963.711284; 1686747963.710494; 354; 155; 0; 3; 1686747963.697556; 9.9.9.11; 3; 15001; 16000; -233; 999; 30240; 7119; 8000; 1769; 881; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.768676; 1686747963.767427 94.140.14.15 3 47163723 47163728 47163728 47163767 44 39 5 1686747963.767427
DEBUG; 2023-06-14-13:06:03; 1686747963.770836; reflector is 94.140.14.15; ewma_iteration 39000 900000 dl_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:03; 1686747963.772427; reflector is 94.140.14.15; ewma_iteration 5000 1000 ul_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:03; 1686747963.774197; reflector is 94.140.14.15; ewma_iteration -16 95000 dl_owd_delta_ewmas_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:03; 1686747963.775816; reflector is 94.140.14.15; 8275 95000 ul_owd_delta_ewmas_us[94.140.14.15]
DATA; 2023-06-14-13:06:03; 1686747963.781267; 1686747963.780506; 354; 155; 0; 3; 1686747963.767427; 94.140.14.15; 3; 39016; 39000; -482; -16; 30240; -3275; 5000; -41; 8275; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.818466; 1686747963.817202 94.140.14.140 3 47163774 47163778 47163778 47163817 43 39 4 1686747963.817202
DEBUG; 2023-06-14-13:06:03; 1686747963.820677; reflector is 94.140.14.140; ewma_iteration 39000 1000 dl_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.822281; reflector is 94.140.14.140; ewma_iteration 4000 1000 ul_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.824004; reflector is 94.140.14.140; ewma_iteration 800 95000 dl_owd_delta_ewmas_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.825686; reflector is 94.140.14.140; 6962 95000 ul_owd_delta_ewmas_us[94.140.14.140]
DATA; 2023-06-14-13:06:03; 1686747963.831645; 1686747963.830906; 354; 155; 0; 3; 1686747963.817202; 94.140.14.140; 3; 38200; 39000; -327; 800; 30240; -2962; 4000; 1189; 6962; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-14-13:06:03; 1686747963.835309; 1686747963.834956; 312; 112; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.838253; reflector is 94.140.14.140; ewma_iteration 39000 1000 dl_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.840905; reflector is 94.140.14.140; ewma_iteration 4000 1000 ul_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.843616; reflector is 94.140.14.140; ewma_iteration 800 95000 dl_owd_delta_ewmas_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.846245; reflector is 94.140.14.140; 6955 95000 ul_owd_delta_ewmas_us[94.140.14.140]
DATA; 2023-06-14-13:06:03; 1686747963.853409; 1686747963.852449; 312; 112; 0; 2; 1686747963.817202; 94.140.14.140; 3; 38200; 39000; -219; 800; 30240; -2955; 4000; 1736; 6955; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.857998; reflector is 94.140.14.140; ewma_iteration 39000 1000 dl_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.860528; reflector is 94.140.14.140; ewma_iteration 4000 1000 ul_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.863434; reflector is 94.140.14.140; ewma_iteration 800 95000 dl_owd_delta_ewmas_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:03; 1686747963.866054; reflector is 94.140.14.140; 6948 95000 ul_owd_delta_ewmas_us[94.140.14.140]
DATA; 2023-06-14-13:06:03; 1686747963.872738; 1686747963.871976; 312; 112; 0; 2; 1686747963.817202; 94.140.14.140; 3; 38200; 39000; -122; 800; 30240; -2948; 4000; 2231; 6948; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.876058; 1686747963.862603 94.140.14.141 3 47163824 47163823 47163823 47163862 38 39 -1 1686747963.862603
DEBUG; 2023-06-14-13:06:03; 1686747963.878556; reflector is 94.140.14.141; ewma_iteration 39000 1000 dl_owd_baselines_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:03; 1686747963.880403; reflector is 94.140.14.141; ewma_iteration -1000 1000 ul_owd_baselines_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:03; 1686747963.882402; reflector is 94.140.14.141; ewma_iteration 902 95000 dl_owd_delta_ewmas_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:03; 1686747963.884228; reflector is 94.140.14.141; 782 95000 ul_owd_delta_ewmas_us[94.140.14.141]
DATA; 2023-06-14-13:06:03; 1686747963.890137; 1686747963.889212; 312; 112; 0; 2; 1686747963.862603; 94.140.14.141; 3; 38098; 39000; -406; 902; 30240; -1782; -1000; -146; 782; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.903721; 1686747963.902565 9.9.9.9 4 47163874 47163885 47163885 47163902 28 17 11 1686747963.902565
DEBUG; 2023-06-14-13:06:03; 1686747963.906127; reflector is 9.9.9.9; ewma_iteration 17000 1000 dl_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.907998; reflector is 9.9.9.9; ewma_iteration 11000 1000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.910027; reflector is 9.9.9.9; ewma_iteration 1896 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:03; 1686747963.911831; reflector is 9.9.9.9; 5919 95000 ul_owd_delta_ewmas_us[9.9.9.9]
DATA; 2023-06-14-13:06:03; 1686747963.917662; 1686747963.916910; 312; 112; 0; 2; 1686747963.902565; 9.9.9.9; 4; 15104; 17000; 216; 1896; 30240; 5081; 11000; 4959; 5919; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:03; 1686747963.947966; 1686747963.946753 9.9.9.10 4 47163924 47163930 47163930 47163946 22 16 6 1686747963.946753
DEBUG; 2023-06-14-13:06:03; 1686747963.950242; reflector is 9.9.9.10; ewma_iteration 16000 1000 dl_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:03; 1686747963.951802; reflector is 9.9.9.10; ewma_iteration 6000 900000 ul_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:03; 1686747963.953569; reflector is 9.9.9.10; ewma_iteration 1805 95000 dl_owd_delta_ewmas_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:03; 1686747963.955320; reflector is 9.9.9.10; -104 95000 ul_owd_delta_ewmas_us[9.9.9.10]
DATA; 2023-06-14-13:06:03; 1686747963.960796; 1686747963.960033; 312; 112; 0; 2; 1686747963.946753; 9.9.9.10; 4; 14195; 16000; -441; 1805; 30240; 6104; 6000; 380; -104; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:04; 1686747964.009254; 1686747964.008067 9.9.9.11 4 47163974 47163991 47163991 47164008 34 17 17 1686747964.008067
DEBUG; 2023-06-14-13:06:04; 1686747964.011461; reflector is 9.9.9.11; ewma_iteration 17000 1000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.013011; reflector is 9.9.9.11; ewma_iteration 17000 1000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.014743; reflector is 9.9.9.11; ewma_iteration 1998 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.016317; reflector is 9.9.9.11; 9872 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DATA; 2023-06-14-13:06:04; 1686747964.021946; 1686747964.020885; 312; 112; 0; 2; 1686747964.008067; 9.9.9.11; 4; 15002; 17000; -21; 1998; 30240; 7128; 17000; 2538; 9872; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-14-13:06:04; 1686747964.036117; 1686747964.035767; 295; 116; 50000; 5000
DEBUG; 2023-06-14-13:06:04; 1686747964.038719; reflector is 9.9.9.11; ewma_iteration 17000 1000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.041685; reflector is 9.9.9.11; ewma_iteration 17000 1000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.044380; reflector is 9.9.9.11; ewma_iteration 1997 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.047058; reflector is 9.9.9.11; 9863 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DATA; 2023-06-14-13:06:04; 1686747964.053737; 1686747964.052976; 295; 116; 0; 2; 1686747964.008067; 9.9.9.11; 4; 15003; 17000; 170; 1997; 30240; 7137; 17000; 3233; 9863; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:04; 1686747964.059028; reflector is 9.9.9.11; ewma_iteration 17000 1000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.061569; reflector is 9.9.9.11; ewma_iteration 17000 1000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.064301; reflector is 9.9.9.11; ewma_iteration 1996 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.066835; reflector is 9.9.9.11; 9854 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DATA; 2023-06-14-13:06:04; 1686747964.074150; 1686747964.073364; 295; 116; 0; 2; 1686747964.008067; 9.9.9.11; 4; 15004; 17000; 343; 1996; 30240; 7146; 17000; 3861; 9854; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:04; 1686747964.078264; 1686747964.067116 94.140.14.15 4 47164024 47164028 47164028 47164067 43 39 4 1686747964.067116
DEBUG; 2023-06-14-13:06:04; 1686747964.081458; reflector is 94.140.14.15; ewma_iteration 39000 900000 dl_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:04; 1686747964.083944; reflector is 94.140.14.15; ewma_iteration 4000 1000 ul_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:04; 1686747964.086883; reflector is 94.140.14.15; ewma_iteration -1 95000 dl_owd_delta_ewmas_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:04; 1686747964.089762; reflector is 94.140.14.15; 7267 95000 ul_owd_delta_ewmas_us[94.140.14.15]
DATA; 2023-06-14-13:06:04; 1686747964.096202; 1686747964.095440; 295; 116; 0; 2; 1686747964.067116; 94.140.14.15; 4; 39001; 39000; -436; -1; 30240; -3267; 4000; 653; 7267; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:04; 1686747964.114722; 1686747964.113548 94.140.14.140 4 47164075 47164073 47164073 47164113 38 40 -2 1686747964.113548
DEBUG; 2023-06-14-13:06:04; 1686747964.117174; reflector is 94.140.14.140; ewma_iteration 40000 1000 dl_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:04; 1686747964.119179; reflector is 94.140.14.140; ewma_iteration -2000 1000 ul_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:04; 1686747964.121176; reflector is 94.140.14.140; ewma_iteration 1799 95000 dl_owd_delta_ewmas_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:04; 1686747964.122995; reflector is 94.140.14.140; 947 95000 ul_owd_delta_ewmas_us[94.140.14.140]
DATA; 2023-06-14-13:06:04; 1686747964.128905; 1686747964.128137; 295; 116; 0; 2; 1686747964.113548; 94.140.14.140; 4; 38201; 40000; 60; 1799; 30240; -2947; -2000; 2109; 947; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:04; 1686747964.165617; 1686747964.164436 94.140.14.141 4 47164125 47164122 47164122 47164164 39 42 -3 1686747964.164436
DEBUG; 2023-06-14-13:06:04; 1686747964.168146; reflector is 94.140.14.141; ewma_iteration 42000 1000 dl_owd_baselines_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:04; 1686747964.169994; reflector is 94.140.14.141; ewma_iteration -3000 900000 ul_owd_baselines_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:04; 1686747964.172010; reflector is 94.140.14.141; ewma_iteration 3899 95000 dl_owd_delta_ewmas_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:04; 1686747964.173825; reflector is 94.140.14.141; -122 95000 ul_owd_delta_ewmas_us[94.140.14.141]
DATA; 2023-06-14-13:06:04; 1686747964.179666; 1686747964.178905; 295; 116; 0; 2; 1686747964.164436; 94.140.14.141; 4; 38101; 42000; 2; 3899; 30240; -2878; -3000; -143; -122; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:04; 1686747964.196076; 1686747964.194913 9.9.9.9 5 47164175 47164180 47164180 47164194 19 14 5 1686747964.194913
DEBUG; 2023-06-14-13:06:04; 1686747964.198569; reflector is 9.9.9.9; ewma_iteration 14000 900000 dl_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.200437; reflector is 9.9.9.9; ewma_iteration 5000 900000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.202449; reflector is 9.9.9.9; ewma_iteration -110 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.204315; reflector is 9.9.9.9; -8 95000 ul_owd_delta_ewmas_us[9.9.9.9]
DATA; 2023-06-14-13:06:04; 1686747964.210152; 1686747964.209385; 295; 116; 0; 2; 1686747964.194913; 9.9.9.9; 5; 14110; 14000; 185; -110; 30240; 5008; 5000; 4487; -8; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
LOAD; 2023-06-14-13:06:04; 1686747964.237001; 1686747964.236646; 283; 102; 50000; 5000
DEBUG; 2023-06-14-13:06:04; 1686747964.239481; reflector is 9.9.9.9; ewma_iteration 14000 900000 dl_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.242102; reflector is 9.9.9.9; ewma_iteration 5000 900000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.244828; reflector is 9.9.9.9; ewma_iteration -11 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.247138; reflector is 9.9.9.9; 0 95000 ul_owd_delta_ewmas_us[9.9.9.9]
DATA; 2023-06-14-13:06:04; 1686747964.253329; 1686747964.252570; 283; 102; 0; 2; 1686747964.194913; 9.9.9.9; 5; 14011; 14000; 166; -11; 30240; 5000; 5000; 4060; 0; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:04; 1686747964.258622; reflector is 9.9.9.9; ewma_iteration 14000 900000 dl_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.262226; reflector is 9.9.9.9; ewma_iteration 5000 1000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.265146; reflector is 9.9.9.9; ewma_iteration -1 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.267631; reflector is 9.9.9.9; 0 95000 ul_owd_delta_ewmas_us[9.9.9.9]
DATA; 2023-06-14-13:06:04; 1686747964.274532; 1686747964.273775; 283; 102; 0; 2; 1686747964.194913; 9.9.9.9; 5; 14001; 14000; 150; -1; 30240; 5000; 5000; 3674; 0; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:04; 1686747964.278249; 1686747964.256688 9.9.9.10 5 47164226 47164240 47164240 47164256 30 16 14 1686747964.256688
DEBUG; 2023-06-14-13:06:04; 1686747964.280657; reflector is 9.9.9.10; ewma_iteration 16000 1000 dl_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:04; 1686747964.282512; reflector is 9.9.9.10; ewma_iteration 14000 1000 ul_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:04; 1686747964.284475; reflector is 9.9.9.10; ewma_iteration 1804 95000 dl_owd_delta_ewmas_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:04; 1686747964.286294; reflector is 9.9.9.10; 7889 95000 ul_owd_delta_ewmas_us[9.9.9.10]
DATA; 2023-06-14-13:06:04; 1686747964.292411; 1686747964.291651; 283; 102; 0; 2; 1686747964.256688; 9.9.9.10; 5; 14196; 16000; -227; 1804; 30240; 6111; 14000; 1093; 7889; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:04; 1686747964.307506; 1686747964.306341 9.9.9.11 5 47164276 47164291 47164291 47164306 30 15 15 1686747964.306341
DEBUG; 2023-06-14-13:06:04; 1686747964.309791; reflector is 9.9.9.11; ewma_iteration 15000 900000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.311357; reflector is 9.9.9.11; ewma_iteration 15000 1000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.313100; reflector is 9.9.9.11; ewma_iteration 0 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.314672; reflector is 9.9.9.11; 7847 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DATA; 2023-06-14-13:06:04; 1686747964.320362; 1686747964.319336; 283; 102; 0; 2; 1686747964.306341; 9.9.9.11; 5; 15000; 15000; 310; 0; 30240; 7153; 15000; 4239; 7847; 32400; 0; 0; dl_idle; ul_idle; 50000; 5000
DEBUG; 2023-06-14-13:06:04; 1686747964.365363; 1686747964.364192 94.140.14.15 5 47164326 47164323 47164323 47164364 38 41 -3 1686747964.364192
DEBUG; 2023-06-14-13:06:04; 1686747964.367529; reflector is 94.140.14.15; ewma_iteration 41000 1000 dl_owd_baselines_us[94.140.14.15]
^CDEBUG; 2023-06-14-13:06:04; 1686747964.370002; reflector is 94.140.14.15; ewma_iteration -3000 1000 ul_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:04; 1686747964.371939; reflector is 94.140.14.15; ewma_iteration 1998 95000 dl_owd_delta_ewmas_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:04; 1686747964.370241; Starting: cleanup_and_killall with PID: 29445
DEBUG; 2023-06-14-13:06:04; 1686747964.373519; reflector is 94.140.14.15; 266 95000 ul_owd_delta_ewmas_us[94.140.14.15]
INFO; 2023-06-14-13:06:04; 1686747964.375582; Stopping cake-autorate with PID: 29445 and config: /root/cake-autorate/cake-autorate_config.primary.sh
INFO; 2023-06-14-13:06:04; 1686747964.378007; Killing all background processes and cleaning up temporary files.
DEBUG; 2023-06-14-13:06:04; 1686747964.414365; 1686747964.413175 94.140.14.140 5 47164376 47164373 47164373 47164413 37 40 -3 1686747964.413175
DEBUG; 2023-06-14-13:06:04; 1686747964.416566; reflector is 94.140.14.140; ewma_iteration 40000 1000 dl_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:04; 1686747964.418335; reflector is 94.140.14.140; ewma_iteration -3000 900000 ul_owd_baselines_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:04; 1686747964.420151; reflector is 94.140.14.140; ewma_iteration 1798 95000 dl_owd_delta_ewmas_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:04; 1686747964.421810; reflector is 94.140.14.140; -6 95000 ul_owd_delta_ewmas_us[94.140.14.140]
DEBUG; 2023-06-14-13:06:04; 1686747964.436781; Terminating monitor_achieved_rates.
DEBUG; 2023-06-14-13:06:04; 1686747964.459401; 1686747964.458192 94.140.14.141 5 47164426 47164418 47164418 47164458 32 40 -8 1686747964.458192
DEBUG; 2023-06-14-13:06:04; 1686747964.461578; reflector is 94.140.14.141; ewma_iteration 40000 1000 dl_owd_baselines_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:04; 1686747964.463208; reflector is 94.140.14.141; ewma_iteration -8000 900000 ul_owd_baselines_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:04; 1686747964.464926; reflector is 94.140.14.141; ewma_iteration 1898 95000 dl_owd_delta_ewmas_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:04; 1686747964.466680; reflector is 94.140.14.141; -513 95000 ul_owd_delta_ewmas_us[94.140.14.141]
DEBUG; 2023-06-14-13:06:04; 1686747964.507903; 1686747964.506687 9.9.9.9 6 47164477 47164490 47164490 47164506 29 16 13 1686747964.506687
DEBUG; 2023-06-14-13:06:04; 1686747964.510092; reflector is 9.9.9.9; ewma_iteration 16000 1000 dl_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.511679; reflector is 9.9.9.9; ewma_iteration 13000 1000 ul_owd_baselines_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.513406; reflector is 9.9.9.9; ewma_iteration 1998 95000 dl_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.514969; reflector is 9.9.9.9; 7992 95000 ul_owd_delta_ewmas_us[9.9.9.9]
DEBUG; 2023-06-14-13:06:04; 1686747964.558634; 1686747964.557407 9.9.9.10 6 47164527 47164540 47164540 47164557 30 17 13 1686747964.557407
DEBUG; 2023-06-14-13:06:04; 1686747964.560810; reflector is 9.9.9.10; ewma_iteration 17000 1000 dl_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:04; 1686747964.562365; reflector is 9.9.9.10; ewma_iteration 13000 1000 ul_owd_baselines_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:04; 1686747964.564081; reflector is 9.9.9.10; ewma_iteration 2802 95000 dl_owd_delta_ewmas_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:04; 1686747964.565653; reflector is 9.9.9.10; 6883 95000 ul_owd_delta_ewmas_us[9.9.9.10]
DEBUG; 2023-06-14-13:06:04; 1686747964.612675; 1686747964.611508 9.9.9.11 6 47164577 47164596 47164596 47164611 34 15 19 1686747964.611508
DEBUG; 2023-06-14-13:06:04; 1686747964.614829; reflector is 9.9.9.11; ewma_iteration 15000 1000 dl_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.616386; reflector is 9.9.9.11; ewma_iteration 19000 1000 ul_owd_baselines_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.618374; reflector is 9.9.9.11; ewma_iteration 0 95000 dl_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.619945; reflector is 9.9.9.11; 11836 95000 ul_owd_delta_ewmas_us[9.9.9.11]
DEBUG; 2023-06-14-13:06:04; 1686747964.664646; 1686747964.663495 94.140.14.15 6 47164627 47164623 47164623 47164663 36 40 -4 1686747964.663495
DEBUG; 2023-06-14-13:06:04; 1686747964.666868; reflector is 94.140.14.15; ewma_iteration 40000 1000 dl_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:04; 1686747964.668586; reflector is 94.140.14.15; ewma_iteration -4000 900000 ul_owd_baselines_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:04; 1686747964.670435; reflector is 94.140.14.15; ewma_iteration 998 95000 dl_owd_delta_ewmas_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:04; 1686747964.672120; reflector is 94.140.14.15; -74 95000 ul_owd_delta_ewmas_us[94.140.14.15]
DEBUG; 2023-06-14-13:06:04; 1686747964.702820; Terminating monitor_achieved_rates.
DEBUG; 2023-06-14-13:06:04; 1686747964.705313; Starting: kill_maintain_pingers with PID: 29470
DEBUG; 2023-06-14-13:06:04; 1686747964.707439; Terminating maintain_pingers.
DEBUG; 2023-06-14-13:06:04; 1686747964.710444; Terminating parse_tsping.
DEBUG; 2023-06-14-13:06:05; 1686747965.388563; Starting: kill_maintain_log_file with PID: 29458
DEBUG; 2023-06-14-13:06:05; 1686747965.389974; Starting: flush_log_fd with PID: 29458
SYSLOG; 2023-06-14-13:06:05; 1686747965.491124; Stopped cake-autorate with PID: 29445 and config: /root/cake-autorate/cake-autorate_config.primary.sh
lynxthecat commented 1 year ago

Yes I think so. All we need now is a crash and the associated log!

To give some context, cake-autorate uses very strict error checking. Any attempt to access an unset variable is caught and an exit forced with appropriate error message. The cause of this error is that for some reason there is an attempt to access an ewma associated with a certain reflector address presented by a line output from 'tsping' that has not yet been set. The question is why. In the crash log you provided there was a recent change of reflectors, but we initialize the ewmas to defaults on such a change here:

https://github.com/lynxthecat/cake-autorate/blob/c4f196a2a436c9d14ad8f1bf336d09400e0f43b6/cake-autorate.sh#L602

Perhaps 'tsping' returns a different reflector IP address from one of the addresses specified as an argument in certain situations?

In any case, hopefully the log will reveal the cause here.

lynxthecat commented 1 year ago

Has crash occurred yet?

bairhys commented 1 year ago

Yep, looks like from my grafana graphs it ran for about 3 hours before crashing, here is the log:

https://gist.github.com/bairhys/ac82a3ab06f22f579b07252d6fc41856

lynxthecat commented 1 year ago

OK I think I worked it out.

Can you try adding a continue after this line:

https://github.com/lynxthecat/cake-autorate/blob/c4f196a2a436c9d14ad8f1bf336d09400e0f43b6/cake-autorate.sh#LL624C1-L624C8

so the continue is inserted with the fi above and ;; below like this:

                    fi
                    continue
                    ;;

Also please can you move and amend the line you added:

log_msg "DEBUG" "${command[*]:1}"

to just above the case in this line:

https://github.com/lynxthecat/cake-autorate/blob/c4f196a2a436c9d14ad8f1bf336d09400e0f43b6/cake-autorate.sh#L570

to give:

        if [[ "${command-}" ]]
                then
                        log_msg "DEBUG" "${command[*]}"
                        case "${command[0]}" in

And then can you post a new log sample to verify the new logging is working? Hopefully this won't be needed, but good to have just in case it still does crash. If it does not crash then all that was needed was the missing 'continue'.

bairhys commented 1 year ago

Ok my new cake-autorate.sh looks like this: https://gist.github.com/bairhys/c344854b5e0647c4816f961b26b15948

And sample log looks like this: https://gist.github.com/bairhys/02bff6dfccb9df3c6c428b4b30d1861b

Thanks for looking into this, I'll keep an eye on it

lynxthecat commented 1 year ago

Looks, good but please change this line:

            log_msg "DEBUG" "${command[*]:1}"

to:

            log_msg "DEBUG" "${command[*]}"

I'm pretty confident we've got this sorted now. The lack of continue meant processing a non-existant reflector response using phony values following a load update to the parser. But with the change above let's keep it running for a while to verify this bug has been quashed.

bairhys commented 1 year ago

Ah I missed that. My cake-autorate.sh looks like this

https://gist.github.com/bairhys/8040107a2337e54e9647101f74ca8646

lynxthecat commented 1 year ago

Looks good. The new log should show the inter process commands that are sent over to the parser for processing.

bairhys commented 1 year ago

My log looks like this now https://gist.github.com/bairhys/ae51effea93a242a5c27caf0b25eee41

bairhys commented 1 year ago

Yeah it now shows the DEBUG; 2023-06-15-12:41:19; 1686832879.758555; REFLECTOR_RESPONSE 1686832879.757406 94.140.15.15 7063 45679718 45679720 45679720 45679757 39 37 2 1686832879.757406 lines

lynxthecat commented 1 year ago

Should be almost 24 hours now. Any news?

bairhys commented 1 year ago

It has been running great, no crashes so I think you have fixed it. All the other times it crashed within a few hours. Thanks for fixing it so quickly.

In a few days, I might re-download the script and add the continue just so it reduces the logs a bit.

lynxthecat commented 1 year ago

I will push the fix to the parsers very shortly. Thanks for raising this issue and helping me work out the cause and fix.

lynxthecat commented 1 year ago

@bairhys this crash condition should be fixed now. Please reopen it if you still see it after having updated.

bairhys commented 1 year ago

Cake Autorate is still running great since. I just used the setup script to update to the latest commit and restarted service.