search

tempesta-tech / tempesta

All-in-one solution for high performance web content delivery and advanced protection against DDoS and web attacks
https://tempesta-tech.com/
GNU General Public License v2.0
623 stars 103 forks source link

Tempesta was failed to was stop and killed after a timeout. #2261

Closed KonsKo closed 1 month ago

KonsKo commented 1 month ago

Scope

Tempesta was failed to stop and was killed after 30 sec timeout at the end of a test.

Testing

Run a test t_regress.test_stress_failovering.TestStressFailovering.test_failovering_sched_ratio on the SUT server

dmesg:

[18728.576325] RIP: 0033:0x7fadd77dc175
[18728.576326] Code: 83 e2 40 75 50 89 f0 f7 d0 a9 00 00 41 00 74 45 80 3d de fe 0e 00 00 74 60 89 da 4c 89 e6 bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 7f 00 00 00 48 8b 55 b8 64 48 2b 14 25 28
[18728.584706]  ahci libahci
[18728.593095] RSP: 002b:00007ffded3f8220 EFLAGS: 00000202 ORIG_RAX: 0000000000000101
[18728.601480]  wmi [last unloaded: tempesta_lib]
[18728.609875] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fadd77dc175
[18728.609876] RDX: 0000000000000000 RSI: 00007ffded3f83f0 RDI: 00000000ffffff9c

[18728.618257] CPU: 102 PID: 856412 Comm: pmdalinux Kdump: loaded Tainted: G           OEL    5.10.35.tfw-fa7cd2d #1
[18728.626646] RBP: 00007ffded3f8290 R08: 0000000000000008 R09: 0000000000000001
[18728.626647] R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffded3f83f0
[18728.635028] Hardware name: Dell Inc. PowerEdge R650/0TCW38, BIOS 1.12.1 09/13/2023
[18728.635031] RIP: 0010:smp_call_function_single+0xe7/0x120
[18728.643415] R13: 0000557aed4e2573 R14: 0000557aed4e2573 R15: 0000000000000000
[18733.283180] INFO: task 9:856413 blocked for more than 120 seconds.
[18733.286732] Code: 65 48 33 0c 25 28 00 00 00 75 46 c9 c3 48 89 e6 48 89 54 24 18 4c 89 44 24 10 e8 94 fe ff ff 8b 54 24 08 83 e2 01 74 0b f3 90 <8b> 54 24 08 83 e2 01 75 f5 eb c6 8b 05 e8 b5 b3 01 85 c0 0f 85 72
[18733.295153]       Tainted: G           OEL    5.10.35.tfw-fa7cd2d #1
[18733.303543] RSP: 0018:ffffb3fae189fce0 EFLAGS: 00000202
[18733.303544] RAX: 0000000000000000 RBX: 000000087f9ca776 RCX: 0000000000000830
[18733.303545] RDX: 0000000000000001 RSI: 00000000000008fb RDI: 0000000000000830
[18733.303545] RBP: ffffb3fae189fd28 R08: ffffffff87c49b30 R09: 0000000000000006
[18733.303546] R10: 000000000000000f R11: ffff896feb29f09c R12: ffff896ec8e93438
[18733.303546] R13: ffff898dbfc11460 R14: ffff896ec8e93460 R15: ffff898dbfc11460
[18733.303547] FS:  00007f6330da3c40(0000) GS:ffff898dc02c0000(0000) knlGS:0000000000000000
[18733.303548] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18733.303549] CR2: 00007fff15fe2688 CR3: 00000003d5c3a004 CR4: 0000000000770ee0
[18733.303550] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18733.311943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[18733.320331] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[18733.320331] PKRU: 55555554
[18733.320332] Call Trace:
[18733.320334]  ? cpu_show_srbds+0x20/0x20
[18733.320336]  ? seq_printf+0x53/0x70
[18733.320338]  ? ktime_get+0x3e/0xa0
[18733.328730] task:9               state:D
[18733.337119]  aperfmperf_snapshot_cpu+0x42/0x50
[18733.337120]  aperfmperf_get_khz+0x40/0x70
[18733.337121]  show_cpuinfo+0x389/0x450
[18733.337123]  seq_read_iter+0x1c5/0x4c0
[18733.337125]  proc_reg_read_iter+0x31/0x80
[18733.345518]  stack:    0 pid:856413 ppid:     1 flags:0x00000002
[18733.353905]  new_sync_read+0x110/0x1a0
[18733.353906]  vfs_read+0xfe/0x190
[18733.353908]  ksys_read+0x67/0xe0
[18733.353910]  __x64_sys_read+0x1a/0x20
[18733.362303] Call Trace:
[18733.370694]  do_syscall_64+0x38/0x90
[18733.370696]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[18733.370697] RIP: 0033:0x7f63318eea61
[18733.370698] Code: 00 48 8b 15 b9 73 0e 00 f7 d8 64 89 02 b8 ff ff ff ff eb bd e8 40 c4 01 00 f3 0f 1e fa 80 3d e5 f5 0e 00 00 74 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 4f c3 66 0f 1f 44 00 00 55 48 89 e5 48 83 ec
[18733.370700] RSP: 002b:00007ffe33830058 EFLAGS: 00000246
[18733.379095]  __schedule+0x45c/0xa40
[18733.387479]  ORIG_RAX: 0000000000000000
[18733.387480] RAX: ffffffffffffffda RBX: 00005654f65ac2f0 RCX: 00007f63318eea61
[18733.387480] RDX: 0000000000000400 RSI: 00005654f65d7910 RDI: 0000000000000006
[18733.387481] RBP: 00007ffe33830090 R08: 0000000000000000 R09: 0000000000000000
[18733.387482] R10: 0000000000000002 R11: 0000000000000246 R12: 00007f63319d5030
[18733.387482] R13: 00007f63319d4ee0 R14: 00005654f65d7d10 R15: 00005654f65ac2f0
[18793.392580] rcu: INFO: rcu_sched self-detected stall on CPU
[18793.392583] rcu:     10-....: (5322 ticks this GP) idle=3d2/1/0x4000000000000000 softirq=200138/200138 fqs=5065 
[18793.392586]  (t=15000 jiffies g=1325945 q=4533)
[18793.392588] NMI backtrace for cpu 10
[18793.392590] CPU: 10 PID: 747 Comm: khungtaskd Kdump: loaded Tainted: G           OEL    5.10.35.tfw-fa7cd2d #1
[18793.392591] Hardware name: Dell Inc. PowerEdge R650/0TCW38, BIOS 1.12.1 09/13/2023
[18793.392592] Call Trace:
[18793.392593]  <IRQ>
[18793.392600]  dump_stack+0x74/0x92
[18793.392602]  nmi_cpu_backtrace.cold+0x32/0x6a
[18793.392607]  ? lapic_can_unplug_cpu+0x80/0x80
[18793.392610]  nmi_trigger_cpumask_backtrace+0xe8/0xf0
[18793.392611]  arch_trigger_cpumask_backtrace+0x19/0x20
[18793.392614]  rcu_dump_cpu_stacks+0xab/0xdd
[18793.392615]  rcu_sched_clock_irq.cold+0x23a/0x411
[18793.392618]  ? account_system_index_time+0x96/0xa0
[18793.392621]  update_process_times+0x5d/0x90
[18793.392624]  tick_sched_handle.isra.0+0x25/0x60
[18793.392625]  tick_sched_timer+0x6f/0x80
[18793.392626]  ? tick_sched_do_timer+0x60/0x60
[18793.392628]  __hrtimer_run_queues+0xfb/0x230
[18793.392629]  hrtimer_interrupt+0x109/0x220
[18793.392630]  __sysvec_apic_timer_interrupt+0x64/0xe0
[18793.392634]  asm_call_irq_on_stack+0xf/0x20
[18793.392635]  </IRQ>
[18793.392638]  sysvec_apic_timer_interrupt+0x81/0x90
[18793.392639]  asm_sysvec_apic_timer_interrupt+0x12/0x20
[18793.392642] RIP: 0010:console_unlock+0x376/0x510
[18793.392644] Code: ff e8 ee f8 ff ff 85 c0 0f 85 4b ff ff ff e8 a1 f8 ff ff 85 c0 0f 85 0e fd ff ff e9 39 ff ff ff e8 7f 29 00 00 48 89 df 57 9d <0f> 1f 44 00 00 8b 85 44 ff ff ff 85 c0 0f 84 3b fd ff ff e8 62 ef
[18793.392645] RSP: 0018:ffffb3face5d3bf8 EFLAGS: 00000246
[18793.392647] RAX: 0000000000000000 RBX: 0000000000000246 RCX: 0000000000000008
[18793.392647] RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000246
[18793.392648] RBP: ffffb3face5d3cb8 R08: 0000000000000000 R09: ffffb3fae4e80020
[18793.392649] R10: ffff896ed058be20 R11: 0000000000000001 R12: ffffffff8988b30c
[18793.392649] R13: ffffffff8988db78 R14: 0000000000000000 R15: 0000000000000000
[18793.392653]  vprintk_emit+0x163/0x260
[18793.392654]  ? console_unlock+0x2a0/0x510
[18793.392656]  ? __schedule+0x45c/0xa40
[18793.392657]  vprintk_default+0x1d/0x20
[18793.392658]  vprintk_func+0x67/0x100
[18793.392659]  printk+0x58/0x6f
[18793.392660]  ? __schedule+0x45c/0xa40
[18793.392662]  show_trace_log_lvl+0x29c/0x2f4
[18793.392663]  ? __schedule+0x45c/0xa40
[18793.392664]  ? __schedule+0x45c/0xa40
[18793.392665]  show_stack+0x3d/0x3f
[18793.392666]  sched_show_task.cold+0xb6/0xc8
[18793.392668]  watchdog.cold+0x91/0xb6
[18793.392670]  kthread+0x114/0x150
[18793.392674]  ? hungtask_pm_notify+0x40/0x40
[18793.392675]  ? kthread_park+0x90/0x90
[18793.392679]  ret_from_fork+0x1f/0x30
[18894.322898]  schedule+0x4f/0xc0
[18894.322900]  schedule_timeout+0x202/0x290
[18894.322902]  __down+0x82/0xd0
[18894.322907]  down+0x47/0x60
[18894.322910]  console_lock+0x1a/0x40
[18894.322912]  console_device+0x18/0x60
[18894.322916]  tty_lookup_driver+0xfb/0x220
[18894.322918]  ? _cond_resched+0x19/0x30
[18894.322919]  tty_open+0x1a7/0x510
[18894.322924]  chrdev_open+0xd3/0x1c0
[18894.322925]  ? cdev_default_release+0x20/0x20
[18894.322928]  do_dentry_open+0x156/0x370
[18894.322929]  vfs_open+0x2d/0x30
[18894.322932]  path_openat+0xb17/0x1120
[18894.322935]  ? __unlock_page_memcg+0x25/0x60
[18894.322936]  ? unlock_page_memcg+0x20/0x30
[18894.322938]  do_filp_open+0x91/0x100
[18894.322940]  ? __check_object_size+0x13f/0x150
[18894.322941]  do_sys_openat2+0x21c/0x2e0
[18894.322942]  do_sys_open+0x46/0x80
[18894.322943]  __x64_sys_openat+0x20/0x30
[18894.322945]  do_syscall_64+0x38/0x90
[18894.322946]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[18894.322949] RIP: 0033:0x7fbbcd279175
[18894.322950] RSP: 002b:00007ffe25e18410 EFLAGS: 00000202 ORIG_RAX: 0000000000000101
[18894.322952] RAX: ffffffffffffffda RBX: 0000000000080101 RCX: 00007fbbcd279175
[18894.322952] RDX: 0000000000080101 RSI: 00007fbbcd6a3908 RDI: 00000000ffffff9c
[18894.322953] RBP: 00007ffe25e18480 R08: 00007fbbcd998380 R09: 0000000000000007
[18894.322954] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fbbcd6a3908
[18894.322954] R13: 0000000000080101 R14: 00007fbbcd6a3908 R15: 00007ffe25e18490
[18894.322964] INFO: task 9:856415 blocked for more than 281 seconds.
[18894.322966]       Tainted: G           OEL    5.10.35.tfw-fa7cd2d #1
[18894.322966] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[18894.322967] task:9               state:D stack:    0 pid:856415 ppid:     1 flags:0x00000002
[18894.322969] Call Trace:
[18894.322974]  __schedule+0x45c/0xa40
[18894.577236]  ? psi_task_change+0x8e/0xd0
[18894.577256]  ? ttwu_do_wakeup+0x1c/0x150
[18894.590161]  ? vm_area_dup+0x25/0xb0
[18894.590183]  ? prep_new_page+0xcd/0x120
[18894.602673]  ? get_page_from_freelist+0x14e4/0x1810
[18894.610220]  ? find_next_bit+0x14/0x20
[18894.616431]  ? update_load_avg+0x7c/0x630
[18894.623176]  ? newidle_balance+0x37c/0x3d0
[18894.629896]  ? dequeue_entity+0xd8/0x3f0
[18894.636135]  ? __update_idle_core+0x9b/0xb0
[18894.642757]  ? __schedule+0x45c/0xa40
[18894.648823]  ? __unlock_page_memcg+0x25/0x60
[18894.655860]  ? schedule+0x4f/0xc0
[18894.661526]  ? do_wait+0x1c3/0x220
[18894.666727]  ? kernel_wait4+0xaf/0x150
[18894.672902]  ? task_stopped_code+0x50/0x50
[18894.679424]  ? __do_sys_wait4+0x85/0x90
[18894.685700]  ? exit_to_user_mode_prepare+0x3d/0x170
[18894.693227]  ? do_user_addr_fault+0x1f2/0x3c0
[18894.699679]  ? irqentry_exit_to_user_mode+0x9/0x20
[18894.707125]  ? __x64_sys_wait4+0x1e/0x20
[18894.713502]  ? asm_exc_page_fault+0x1e/0x30
[18894.823236] [tdb] Close table 'client0.tdb'
[18894.840722] [tdb] Close table 'sessions0.tdb'
[18894.963829] [tdb] Close table 'filter0.tdb'
[18894.968399] [tempesta fw] modules are stopped
[18895.538580] [tempesta fw] exiting...
[18895.655054] [tdb] Shutdown Tempesta DB
[18910.382704] End test:   t_regress.test_stress_failovering.TestStressFailovering.test_failovering_sched_ratio

test logs:

24-10-10 21:55:29.228100 WARNING   |      Stopping TempestaFW on localhost                                                                                                                                                               tf_cfg.py:245
24-10-10 21:55:29.231069 INFO      | Run command '/home/jenkins/workspace/Tests_on_bare_metal/tempesta/scripts/tempesta.sh --stop'  on host localhost with environment None                                                              remote.py:153
24-10-10 21:55:29.233754 DEBUG     | All environment variables after updating: {'SHELL': '/bin/bash', 'PWD': '/home/jenkins/workspace/Tests_on_bare_metal/tempesta-test', 'LOGNAME': 'root', 'XDG_SESSION_TYPE': 'tty', 'HOME':          remote.py:169
                                  '/home/jenkins', 'LANG': 'en_US.UTF-8', 'LS_COLORS':                                                                                                                                                                
                                  'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.a              
                                  rj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*              
                                  .xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01              
                                  ;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;3              
                                  5:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*              
                                  .m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi              
                                  =01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*              
                                  .midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;9              
                                  0:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-n              
                                  ew=00;90:*.ucf-old=00;90:', 'SSH_CONNECTION': '207.173.132.22 63203 94.242.233.20 10022', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'XDG_SESSION_CLASS': 'user', 'TERM': 'xterm-256color', 'LESSOPEN':                
                                  '| /usr/bin/lesspipe %s', 'USER': 'root', 'SHLVL': '2', 'XDG_SESSION_ID': '26', 'XDG_RUNTIME_DIR': '/run/user/1000', 'SSH_CLIENT': '207.173.132.22 63203 10022', 'XDG_DATA_DIRS':                                   
                                  '/usr/local/share:/usr/share:/var/lib/snapd/desktop', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus',                 
                                  'MAIL': '/var/mail/jenkins', 'SSH_TTY': '/dev/pts/0', 'OLDPWD': '/home/jenkins/workspace', '_': '/usr/bin/sudo', 'SUDO_COMMAND':                                                                                    
                                  '/home/jenkins/workspace/Tests_on_bare_metal/tempesta-test/venv_for_test/bin/python3 run_tests.py -nE -v 6 t_regress.test_stress_failovering.TestStressFailovering.test_failovering_sched_ratio',                   
                                  'SUDO_USER': 'jenkins', 'SUDO_UID': '1001', 'SUDO_GID': '1001'}                                                                                                                                                     
24-10-10 22:02:52.534133 CRITICAL  | Exception in stopping process: None, type: <class 'helpers.error.ProcessKilledException'>, traceback:                                                                                               tf_cfg.py:245
                                  Traceback (most recent call last):                                                                                                                                                                                  
                                    File "/home/jenkins/workspace/Tests_on_bare_metal/tempesta-test/helpers/remote.py", line 187, in run_cmd                                                                                                          
                                      stdout, stderr = current_proc.communicate(timeout=timeout)                                                                                                                                                      
                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                      
                                    File "/usr/lib/python3.11/subprocess.py", line 1209, in communicate                                                                                                                                               
                                      stdout, stderr = self._communicate(input, endtime, timeout)                                                                                                                                                     
                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                     
                                    File "/usr/lib/python3.11/subprocess.py", line 2116, in _communicate                                                                                                                                              
                                      self._check_timeout(endtime, orig_timeout, stdout, stderr)                                                                                                                                                      
                                    File "/usr/lib/python3.11/subprocess.py", line 1253, in _check_timeout                                                                                                                                            
                                      raise TimeoutExpired(                                                                                                                                                                                           
                                  subprocess.TimeoutExpired: Command '/home/jenkins/workspace/Tests_on_bare_metal/tempesta/scripts/tempesta.sh --stop' timed out after 30 seconds                                                                     

                                  The above exception was the direct cause of the following exception:                                                                                                                                                

                                  Traceback (most recent call last):                                                                                                                                                                                  
                                    File "/home/jenkins/workspace/Tests_on_bare_metal/tempesta-test/framework/stateful.py", line 71, in force_stop                                                                                                    
                                      stop_proc()                                                                                                                                                                                                     
                                    File "/home/jenkins/workspace/Tests_on_bare_metal/tempesta-test/helpers/control.py", line 357, in stop_tempesta                                                                                                   
                                      self.node.run_cmd(cmd, timeout=30)                                                                                                                                                                              
                                    File "/home/jenkins/workspace/Tests_on_bare_metal/tempesta-test/helpers/remote.py", line 204, in run_cmd                                                                                                          
                                      raise error.ProcessKilledException() from to_exc                                                                                                                                                                
                                  helpers.error.ProcessKilledException: None                                                                                                                                                                          

24-10-10 22:02:52.544446 WARNING   | Removing /tmp/tempesta/tempesta_tmp.conf: file not found                                                                                                                                            remote.py:267
test_failovering_sched_ratio (t_regress.test_stress_failovering.TestStressFailovering.test_failovering_sched_ratio) ... ERROR
EvgeniiMekhanik commented 1 month ago

It seems that this BUG is because of unlimited waring messages in dmesg. https://github.com/tempesta-tech/tempesta-test/pull/705 fixes this problem

KonsKo commented 1 month ago

Looks like the problem has gone

[64193.588452] Start test: t_regress.test_stress_failovering.TestStressFailovering.test_failovering_sched_ratio
[64193.917019] [tdb] Start Tempesta DB
[64193.958470] net_ratelimit: 196100 callbacks suppressed
[64193.958472] [tempesta fw] Initializing Tempesta FW kernel module...
[64194.111272] [tempesta fw] Warning: Vhost default doesn't have certificate with matching SAN/CN.
                   Maybe that's fine, but it's worth checking the
                   config - if there is no relations between the
                   names, then host name confusion attack is possible.
[64194.167833] [tempesta fw] Configuration processing is completed.
[64194.214632] [tdb] Opened table /opt/tempesta/db/filter0.tdb: size=16777216 rec_size=20 base=0000000031e8f88e
[64194.242880] [tdb] Opened table /opt/tempesta/db/sessions0.tdb: size=16777216 rec_size=312 base=000000009d72e694
[64194.271962] [tdb] Opened table /opt/tempesta/db/client0.tdb: size=16777216 rec_size=624 base=000000009d697641
[64194.283679] [tempesta fw] Open listen socket on: 0.0.0.0:443
[64194.351130] [tempesta fw] Tempesta FW is ready
[64194.585906] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64194.585916] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64194.585978] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64194.586057] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64194.586086] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64194.586100] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64198.959082] net_ratelimit: 1728661 callbacks suppressed
[64198.959083] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64198.959128] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64198.959163] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64198.959168] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64198.959172] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64198.959176] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64198.959179] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64198.959183] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64198.959187] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64198.959190] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64203.963399] net_ratelimit: 2064738 callbacks suppressed
[64203.963401] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64203.963589] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64203.963619] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64203.963622] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64203.963625] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64203.963628] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64203.963630] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64203.963633] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64203.963636] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64203.963638] [tempesta fw] Warning: request dropped: processing error, status 502: 127.0.0.1
[64205.115932] [tdb] Close table 'client0.tdb'
[64205.134930] [tdb] Close table 'sessions0.tdb'
[64205.203479] [tdb] Close table 'filter0.tdb'
[64205.234738] [tempesta fw] exiting...
[64205.314997] [tdb] Shutdown Tempesta DB
[64205.485753] End test:   t_regress.test_stress_failovering.TestStressFailovering.test_failovering_sched_ratio
krizhanovsky commented 1 month ago

Closed by #2263