pawn-lang / YSI-Includes

Just the YSI include files, none of the extra stuff.
211 stars 105 forks source link

Server freezes after updating to YSI 5.x #564

Open SimonOre opened 2 years ago

SimonOre commented 2 years ago

After updating from a previous YSI version 4.00.0001 to 5.x the server freezes at startup. No warnings/errors after compiling

[16:49:31]  
[16:49:31]  
[16:49:31] 
         ==============================================================
         |                                                            |
         |                                                            |
         |     YYYYYYY       YYYYYYY   SSSSSSSSSSSSSSS IIIIIIIIII     |
         |     Y:::::Y       Y:::::Y SS:::::::::::::::SI::::::::I     |
[16:49:31]          |     Y:::::Y       Y:::::YS:::::SSSSSS::::::SI::::::::I     |
         |     Y::::::Y     Y::::::YS:::::S     SSSSSSSII::::::II     |
         |     YYY:::::Y   Y:::::YYYS:::::S              I::::I       |
         |        Y:::::Y Y:::::Y   S:::::S              I::::I       |
         |         Y:::::Y:::::Y     S::::SSSS           I::::I       |
[16:49:31]          |          Y:::::::::Y       SS::::::SSSSS      I::::I       |
         |           Y:::::::Y          SSS::::::::SS    I::::I       |
         |            Y:::::Y              SSSSSS::::S   I::::I       |
         |            Y:::::Y                   S:::::S  I::::I       |
         |            Y:::::Y                   S:::::S  I::::I       |
[16:49:31]          |            Y:::::Y       SSSSSSS     S:::::SII::::::II     |
         |         YYYY:::::YYYY    S::::::SSSSSS:::::SI::::::::I     |
         |         Y:::::::::::Y    S:::::::::::::::SS I::::::::I     |
         |         YYYYYYYYYYYYY     SSSSSSSSSSSSSSS   IIIIIIIIII     |
         |                                                            |
[16:49:31]          |                                                            |
         |                      (c) 2021 MPL v1.1                     |
         |            Alex "Y_Less" Cole and contributors.            |
         |                                                            |
         |                                                            |
[16:49:31]          ==============================================================

[16:49:31]  
[16:49:31]  
[16:49:31]  ========================================== 
[16:49:31]  |                                        | 
[16:49:31]  |   Generating code, this may take a     | 
[16:49:31]  |  little bit of time.  Note that this   | 
[16:49:31]  |  code generation works best with the   | 
[16:49:31]  |     JIT plugin, which you are not      | 
[16:49:31]  |     currently using.  Get it here:     | 
[16:49:31]  |                                        | 
[16:49:31]  |       https://git.io/jit-plugin        | 
[16:49:31]  |                                        | 
[16:49:31]  |             Please wait...             | 
[16:49:40]  |                                        | 
[16:49:40]  |                 Done!                  | 
[16:49:40]  |                                        | 
[16:49:40]  ========================================== 
[16:49:40]  |                                        | 
[16:49:40]  | Server:      0.3.7 - Linux             | 
[16:49:40]  | Compiler:    3.2.3664 - Windows        | 
[16:49:40]  | YSI:         5.05.0505                 | 
[16:49:40]  | Started:     1640731771                | 
[16:49:40]  | JIT:         <none>                    | 
[16:49:40]  | Crashdetect: <warning>                 | 
[16:49:40]  |                                        | 
[16:49:40]  |   Crashdetect has been found, but no   | 
[16:49:40]  |      long-call controls have been      | 
[16:49:40]  |    detected.  If you see a console     | 
[16:49:40]  |  warning about slow execution during   | 
[16:49:40]  |    startup (only), ignore it - this    | 
[16:49:40]  |  message cannot be disabled on Linux   | 
[16:49:40]  |     in some releases due to a bug.     | 
[16:49:40]  |                                        | 
[16:49:40]  ========================================== 
[16:49:40]  |                                        | 
[16:49:40]  | Unable to check the latest YSI version | 
[16:49:40]  | - please watch the forums and github:  | 
[16:49:40]  |                                        | 
[16:49:40]  |          https://git.io/5.x            | 
[16:49:40]  |                                        | 
[16:49:40]  ========================================== 
[16:49:40]  
[16:49:40]   Filterscript '../scriptfiles/callbackfix.amx' loaded.
[16:49:40]  
[16:49:40] ---------------------------------------------------------------------------
[16:49:40] [INFO] Starting GM
[16:49:40] [INFO] MySQL Connection ID: [ID:1]
[16:49:40] [INFO] Loading vehicles
[16:49:40] [INFO] Vehicles loaded. Total 1265
[16:49:40] [INFO] Loading config
[16:49:40] [INFO] Config loaded
[16:53:58] [INFO] Load completed
[16:53:58] ---------------------------------------------------------------------------
[16:53:58] Number of vehicle models: 4
[16:53:59] [connection] incoming connection: 127.0.0.1:55352 id: 0
[16:53:59] [connection] incoming connection: 127.0.0.1:36599 id: 1
[16:54:00] [npc:join] [68ES]Mihaly[VAF] has joined the server (0:127.0.0.1)
[16:54:00] [npc:join] [ISAF]SkyEye has joined the server (1:127.0.0.1)
[16:54:00] 
*** [FCNPC] Info: A newer version (v2.0.4) is available at https://github.com/ziggi/FCNPC/releases ***

The server then starts to use 100% CPU and players cannot join

[16:50:35] Kicking 149.56.x.x because they didn't logon to the game.

Using Ubuntu 21.04 crashdetect.so FCNPC.so streamer.so sscanf.so Whirlpool.so mysql.so gvar.so

#include <a_samp>
#include <sscanf2>
#include <zcmd>
#include <streamer>
#include <YSI_Data\y_iterate>//from YSI\y_iterate
#include <YSI_Coding\y_va>//from YSI\y_va
#include <a_mysql>
#include <gvar>
#include <p_progressbar>
#include <mSelection>
native IsValidVehicle(vehicleid);

native WP_Hash(buffer[], len, const str[]);
r4sheed commented 2 years ago

I have noticed a similar issue with y_dialog. When I start the server the login dialog is fine but after hours it can make the server stuck inside an infinite loop (This is why your server running but players cannot join, and 100% CPU use). No usable information by crashdetect. I am using the latest includes and latest plugins also using sampctl. I also messaged Y_less on Discord DM but did not get any response.

r4sheed commented 2 years ago

[15:12:48] [debug] AMX backtrace:
[15:12:48] [debug] #0 000467b8 in Alloc:Malloc_Allocate (size=79, bool:clear=false) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies\YSI-Includes\YSI_Coding\y_timers\..\y_malloc\y_malloc_funcs.inc:836
[15:12:48] [debug] #1 0007d680 in F@_@:GetLocalFunction (func[]=@00a87f58 "ReportResponse", spec[]=@00a87f94 "iiiis") at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies\YSI-Includes\YSI_Visual\y_dialog\..\..\YSI_Coding\y_inline\y_inline_impl.inc:1079
[15:12:48] [debug] #2 0008041c in Inline_UP_ (func[]=@00a87f58 "ReportResponse", spec[]=@00a87f94 "iiiis") at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies\YSI-Includes\YSI_Visual\y_dialog\..\..\YSI_Coding\y_inline\y_inline_impl.inc:1992
[15:12:48] [debug] #3 0027b044 in bool:Report_ShowAll (playerid=12) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\gamemodes\core\player\report.inc:233
[15:12:48] [debug] #4 0027b29c in ?? (12, 11792116, 0) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\gamemodes\core\player\report.inc:261
[15:12:48] [debug] #5 00073814 in Command_ReProcess (12, 11792084, 0)  at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies\YSI-Includes\YSI_Visual\y_commands\y_commands_impl.inc:541
[15:12:48] [debug] #6 00076d3c in @yH_OnPlayerCommandText@004 (12, 11792084)  at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies\YSI-Includes\YSI_Visual\y_commands\y_commands_impl.inc:1397
[15:12:48] [debug] #7 00034070 in On_PlayerCommandText (12, 11792084) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies\YSI-Includes\YSI_Coding\y_timers\..\y_malloc\..\y_cgen\y_cgen.inc:131
[15:12:48] [debug] #8 0000e690 in public OnPlayerCommandText (12, 11792084) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies\sa-mp-fixes\fixes.inc:14242```
r4sheed commented 2 years ago

Now I changed all Dialog_Show and Dialog_ShowCallback to ShowPlayerDialog and it’s works fine without crashing. I also added performance profiler before and …

image
PatrickGTR commented 2 years ago

Upload a MVCE so it can be replicated otherwise we're playing a guessing game here.

r4sheed commented 2 years ago

I intentionally not added MVCE due the problem hard to reproducible... The server crash is related to the number of players somehow because I ran a test server with 3-4 players for weeks and it was fine.

I don't think the problem is in my code but you can check this function, this is also crashed my server (https://pastebin.com/pLyWeiUd - I cannot paste it here, sorry) I removed y_dialog and y_inline completely from my gamemode and it's working for weeks without any crash.

I repeat this is only happens after hours of uptime, otherwise it works.

Line 132: Dialog_ShowCallback(playerid, using inline Response, DIALOG_STYLE_LIST, va_return("Tippek (%i)", rows), caption, "Módosít", "Kilép");

[21:25:54] *** YSI Info: CMD: Rasheed_Surenos (5) => /tips [21:25:54] [debug] Long callback execution detected (hang or performance issue) [21:25:54] [debug] AMX backtrace: [21:25:54] [debug] #0 00041e98 in Alloc:Malloc_Allocate (size=2143, bool:clear=false) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies\YSI-Includes\YSI_Coding\y_timers..\y_malloc\y_malloc_funcs.inc:825 [21:25:54] [debug] #1 00099b28 in InlineUI (&header=@01b7505c 11752436) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies\YSI-Includes\YSI_Coding\y_inline\y_inline_impl.inc:1955 [21:25:54] [debug] #2 00399cb0 in ?? (5) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\gamemodes\core\server\tips.inc:132 [21:25:54] [debug] #3 00399670 in Indirect_Call () at C:\Users\Admin\OneDrive\westrpg-master\westrpg\gamemodes\core\server\tips.inc:95 [21:25:54] [debug] #4 0002f154 in public Indirect_FromCallback (15552252, 1) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies.resources\sscanf-019253....\indirection\indirection.inc:917 [21:26:55] [debug] Server received interrupt signal while executing server.amx [21:26:55] [debug] AMX backtrace: [21:26:55] [debug] #0 00041e68 in Alloc:Malloc_Allocate (size=2143, bool:clear=false) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies\YSI-Includes\YSI_Coding\y_timers..\y_malloc\y_malloc_funcs.inc:825 [21:26:55] [debug] #1 00099b28 in InlineUI (&header=@01b7505c 11752436) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies\YSI-Includes\YSI_Coding\y_inline\y_inline_impl.inc:1955 [21:26:55] [debug] #2 00399cb0 in ?? (5) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\gamemodes\core\server\tips.inc:132 [21:26:55] [debug] #3 00399670 in Indirect_Call () at C:\Users\Admin\OneDrive\westrpg-master\westrpg\gamemodes\core\server\tips.inc:95 [21:26:55] [debug] #4 0002f154 in public Indirect_FromCallback (15552252, 1) at C:\Users\Admin\OneDrive\westrpg-master\westrpg\dependencies.resources\sscanf-019253....\indirection\indirection.inc:917 [21:26:55] [debug] Native backtrace: [21:26:55] [debug] #0 f7a51f7f in _Z13GetStackTraceRSt6vectorI10StackFrameSaIS0_EEPv () in plugins/crashdetect.so [21:26:55] [debug] #1 f7a4a81d in _ZN11CrashDetect20PrintNativeBacktraceERSoRKN2os7ContextE () in plugins/crashdetect.so [21:26:55] [debug] #2 f7a4ae48 in _ZN11CrashDetect20PrintNativeBacktraceERKN2os7ContextE () in plugins/crashdetect.so [21:26:55] [debug] #3 f7a4c6bb in _ZN11CrashDetect11OnInterruptERKN2os7ContextE () in plugins/crashdetect.so [21:26:55] [debug] #4 f7a517ff in ?? () in plugins/crashdetect.so [21:26:55] [debug] #5 f7f87580 in kernel_rt_sigreturn () in linux-gate.so.1 [21:26:55] [debug] #6 f7a53726 in amx_Exec () in plugins/crashdetect.so [21:26:55] [debug] #7 f7a4c0b4 in _ZN11CrashDetect11ProcessExecEPii () in plugins/crashdetect.so [21:26:55] [debug] #8 f7a4fe6f in ?? () in plugins/crashdetect.so [21:26:55] [debug] #9 f645a932 in ?? () in plugins/streamer.so [21:26:55] [debug] #10 f6625da1 in amx_Exec () in plugins/profiler.so [21:26:55] [debug] #11 f662e890 in _ZN7amxprof8Profiler8ExecHookEPiiPFiP6tagAMXS1_iE () in plugins/profiler.so [21:26:55] [debug] #12 f6628533 in _ZN15ProfilerHandler4ExecEPii () in plugins/profiler.so [21:26:55] [debug] #13 f662711b in ?? () in plugins/profiler.so [21:26:55] [debug] #14 f76a8e84 in ?? () in plugins/mysql.so [21:26:55] [debug] #15 f76ac249 in ?? () in plugins/mysql.so [21:26:55] [debug] #16 f76d3f37 in ?? () in plugins/mysql.so [21:26:55] [debug] #17 f76bb938 in ?? () in plugins/mysql.so [21:26:55] [debug] #18 f76bae22 in ?? () in plugins/mysql.so [21:26:55] [debug] #19 f76bc3b4 in ?? () in plugins/mysql.so [21:26:55] [debug] #20 f76e4887 in ProcessTick () in plugins/mysql.so [21:26:55] [debug] #21 080d1ce2 in ?? () in ./samp03svr [21:26:55] [debug] #22 080aef6c in ?? () in ./samp03svr [21:26:55] [debug] #23 080aa13a in ?? () in ./samp03svr [21:26:55] [debug] #24 f7a9be46 in libc_start_main () in /lib32/libc.so.6 [21:26:55] [debug] #25 0804b4e1 in ?? () in ./samp03svr

Y-Less commented 2 years ago

I've put YSI_NO_HEAP_MALLOC back in. That was always the vastly more stable method.

Y-Less commented 2 years ago

I also just discovered that the default slop space (space reserved in the heap for callback parameters) was very small (only ~256 bytes (just 64 cells)), which will be blown with any reasonable string. So the other thing to try is:

#define YSI_DEFAULT_MALLOC (2047)
#define YSI_YES_HEAP_MALLOC

That should work with the old and new versions. And I just increased the default to this anyway.

Edit: I think the comments were misleading even me and it was 256 cells, not 256 bytes. Still not a huge amount, but does mean that 2047 can probably come down to something more reasonable like 511.

Y-Less commented 2 years ago

Basically, the new version uses the heap, but the VM clears the heap at the end of every top-level callback (easy way to ensure consistency). So every time a new callback is called its reference parameters (i.e. arrays and strings) are inserted at the start of the heap, which clobbers the old malloc values that were there. To combat this I just allocate a block at the start of the malloc memory and abandon it, leaving it free to be used by different callback parameters. I just think there wasn't enough space here for large parameters. There's also code injected in to every public function to restore the previous malloc heap locations.