Closed aazamansari closed 1 year ago
guijemont
commented
7 years ago Thank you for the stack trace. Can you provide more information on the conditions in which this happened? What hardware, what version of WPE, how you compiled it, how you were running it and what code/webpage you were running at the time? Anything that could help us reproduce the bug would be helpful. Also, it looks like you hit an assertion: did any message print regarding this assertion? What was it?
Thanks!
aazamansari
commented
7 years ago We use 5c0c3fd0aa4cdb3693154032cd969b93f88f5bcd from stable branch. Below are the settings that we used:
export WPE_DISK_CACHE_SIZE=10m export WPE_RAM_SIZE=192m export WPE_POLL_MAX_MEMORY='WPEWebProcess:150M,*Process:50M'
This happens on a mips platform. I compile the WPE with -O2 option.
We don't see any asserting logs.
guijemont
commented
7 years ago What is the scenario to reproduce? What webpage, what were you doing?
aazamansari
commented
7 years ago The issue happens when playing live video. It is internal link for linear video(live video) which is not accessible from outside.
aazamansari
commented
7 years ago I see crash in same call without the custom player library that we have.
albertd
commented
6 years ago @aazamansari is this reproducible on the wpe-2017 branch?
guijemont
commented
6 years ago @aazamansari are you able to reproduce with a debug build, or with an -O0 (or maybe -O1) build with debug symbols? With the build+core files that you provided, I haven't been able to see the values of local variables, which makes this very hard to investigate.
Gavriliuk
commented
6 years ago The issue is still being reprodused in the release versions, hundreds crashes every day: https://crashportal.stb.r53.xcal.tv/minidumpReport/list?signature=operationLinkDirectCall
wouterlucas
commented
6 years ago @Gavriliuk and we've been trying to reproduce this on a vanilla Broadcom STB with WPE for hours by different people, different devices without success using refsw 16.x and 17.x.
Without having a reliable way for us to reproduce we can't really help you.
Gavriliuk
commented
6 years ago @wouterlucas Could I help you by providing stacktraces, logs etc. from the crashportal?
guijemont
commented
6 years ago @Gavriliuk a core file from a debug build (and associated build files and source code/revision) could be helpful.
Gavriliuk
commented
6 years ago Hi @guijemont, unfortunately this doesn't look possible
We observe these crashes in the field, I can attach some minidump reports with stacktraces from the recent builds: https://crashportal.stb.r53.xcal.tv/minidumpReport/show/118427400 https://crashportal.stb.r53.xcal.tv/minidumpReport/show/118472731 https://crashportal.stb.r53.xcal.tv/minidumpReport/show/118587423
ncuillery
commented
6 years ago At Youi we have the exact same error (WTFCrash caused by operationLinkDirectCall) on our apps.
We use the 2.19.1 tarball (https://webkitgtk.org/releases/webkitgtk-2.19.1.tar.xz) that we build ourself.
We have been able to narrow down the issue to the following: It only happens on:
@Gavriliuk Does it matches the reports you saw in your dashboard? (it is not public)
Gavriliuk
commented
6 years ago To @ncuillery Thank you Nicolas We observe this kind of crashes on TV boxes (OS Linux, various platforms and architectures) I hope your comment will be helpful for Metro team
albertd
commented
5 years ago @Gavriliuk do we have a public app available to validate?
Gavriliuk
commented
5 years ago @albertd, I actually can't know what do you have :-) But I guess we ourself don't have a such app
This crash happens on our TV STBs - not very rarely, but randomly, no steps to reproduce E.g., 15805 crashes were registered from testing boxes during the recent 30 days I'd attached some example stacktraces above
Thanks Alex
Gavriliuk
commented
5 years ago It seems like the crash is being happened during the XMLHttpRequest sends the asynchronous request or processes its response
nrajan002c
commented
5 years ago Hi @wouterlucas @albertd ,
Again we started analyzing this issue since this crash frequency is higher in field. Originally the SIGUSR1 is being sent inside operationLinkDirectCall api. (i.e. RELEASE_ASSERT(callLinkInfo->isDirect()); code: https://github.com/WebPlatformForEmbedded/WPEWebKit/blob/wpe-20170728/Source/JavaScriptCore/jit/JITOperations.cpp#L999)
operationLinkDirectCall will be called only when the calltype is Direct*.
But before this api is executed, someone else is calling setCallType and changed the callLinkInfo object's callType. So that assert is happened.
Can we modify the code like this to avoid the crash,
void JIT_OPERATION operationLinkDirectCall(ExecState* exec, CallLinkInfo* callLinkInfo, JSFunction* callee) { ..... //RELEASE_ASSERT(callLinkInfo->isDirect()); if(!callLinkInfo->isDirect()) return;
Could you please tell the consequence of this change.
guijemont
commented
5 years ago Can we modify the code like this to avoid the crash,
void JIT_OPERATION operationLinkDirectCall(ExecState* exec, CallLinkInfo* callLinkInfo, JSFunction* callee) { ..... //RELEASE_ASSERT(callLinkInfo->isDirect()); if(!callLinkInfo->isDirect()) return;Could you please tell the consequence of this change. I would need to check more in detail, but this assert is very likely here for a reason, and I would expect that removing it would just move the bug somewhere else.
Interestingly, this week I've been playing with running jsc's tests with qemu, and I found a crash that looks a lot like this one (which wouldn't reproduce on the mips hardware I have). I'll keep on investigating that since now I have a way to reproduce, and I will update this bug once I understand more of what's happening.
nrajan002c
commented
5 years ago Thanks @guijemont for the update. We are seeing this problem in mips platform also. Since this occurs randomly in field, We don't have the concrete reproducing steps. Please let me know if you want to try anything in STB.
Gavriliuk
commented
4 years ago This problem is often observed when load average is too high
It seems the operation block is already destroyed when its call is executed
It leads to SIGSEGV in many cases, and sometimes SIGUSR1 from WTFCrash because of RELEASE_ASSERT on the line 999:
void JIT_OPERATION operationLinkDirectCall(ExecState* exec, CallLinkInfo* callLinkInfo, JSFunction* callee)
{
VM* vm = &exec->vm();
auto throwScope = DECLARE_THROW_SCOPE(*vm);
CodeSpecializationKind kind = callLinkInfo->specializationKind();
NativeCallFrameTracer tracer(vm, exec);
/// Line 999:
RELEASE_ASSERT(callLinkInfo->isDirect());
Balajims88
commented
4 years ago @wouterlucas @woutermeek I am believing this #650 fix this issue. so could you please review and merge this #650
https://github.com/WebPlatformForEmbedded/WPEWebKit/pull/650
CC @aazamansari
guijemont
commented
4 years ago guijemont
commented
4 years ago @Balajims88 @aazamansari have you been able to confirm that the issue is fixed?
kkanag314
commented
4 years ago Hi @guijemont
The JSC crash in operationLinkDirectCall is still not resolved.
[ 10 Thread 0 (crashed) 11 0 libWPEWebKit.so.0.0.20170728!WTFCrash [Assertions.cpp : 270 + 0x8] 12 1 libWPEWebKit.so.0.0.20170728!operationLinkDirectCall [JITOperations.cpp : 999 + 0x4] 13 2 0x71107d54]
The crash has been identified in the boxes like SamsungXG2V2. Exact steps to reproduce is not available as they are occurring random and intermittently.
Balajims88
commented
4 years ago Hi @guijemont , Issue is still reproducible. so the above fix is not fixing properly. so could you please take a look?
guijemont
commented
4 years ago Hi @guijemont , Issue is still reproducible. so the above fix is not fixing properly. so could you please take a look?
Hi @Balajims88! Are you able to provide a way for us to reproduce the issue? This would be the one thing that would allow us to figure things out relatively quickly.
Balajims88
commented
4 years ago @guijemont Sorry, I have a no steps to reproduce this issue.
pkumbh631
commented
3 years ago Hi @guijemont, Please find the below steps to reproduce the issue, Step 1# Push 4.2p12s1 to pace XG2V2. Step 2# Play DASH IPVOD and Peacock Playbacks. Step 3# Monitor for webprocess crash. Signature : operationLinkDirectCall.
Thank you...
guijemont
commented
3 years ago Hi @guijemont, Please find the below steps to reproduce the issue, Step 1# Push 4.2p12s1 to pace XG2V2.
Can you tell me what revision of https://github.com/WebPlatformForEmbedded/WPEWebKit matches "4.2p12s1" ?
I don't have access to "XG2V2", but I can try to reproduce on another device of the same architecture. Is that an arm or mips device?
Step 2# Play DASH IPVOD and Peacock Playbacks.
How can I access that?
Step 3# Monitor for webprocess crash. Signature : operationLinkDirectCall.
Does the crash happen every time? And how quickly does it happen?
pkumbh631
commented
3 years ago The crash usually happens upon launching a web-app. Three most common crash URLs are:
https://xfinity.ccast.api.amazonvideo.com/lrc-vending/html5/index.html?deviceTyp https://www.youtube.com/tv?launch=menu&lmt=0&us_privacy=1-N- https://comcast.play.hbomax.com/?launchpoint=&query=&assetId=&assetType=&entityI
The release assert gets fired that checks if it is a Direct Call : https://github.com/WebPlatformForEmbedded/WPEWebKit/blob/941e5086f2ae4de3448deb2566728214ba668dce/Source/JavaScriptCore/jit/JITOperations.cpp#L1041
`void JIT_OPERATION operationLinkDirectCall(ExecState exec, CallLinkInfo callLinkInfo, JSFunction callee) { VM vm = &exec->vm(); auto throwScope = DECLARE_THROW_SCOPE(*vm);
CodeSpecializationKind kind = callLinkInfo->specializationKind();
NativeCallFrameTracer tracer(vm, exec);
RELEASE_ASSERT(callLinkInfo->isDirect()) // <== HERE `It looks like operationLinkDirectCall is getting called for an unexpected CallLinkInfo call type, possibly as a result of some kind of data corruption. This assert gets fired on MIPS-based devices only, so it can be a MIPS specific error or something related to limited memory of those device types.
glibnovodran
commented
3 years ago @guijemont , The crash seems to happen on both wpe-20170728 and wpe-2.22 More crashes tend to happen to wpe-20170728, but this is probably because more devices run the builds based upon it. The problem seems to be MIPS-specific, but can be a consequence of limited memory amount (a GC-related issue?) Presently wpe-20170728 crashes and wpe-2.22 crashes triggered by the same assertion.
wpe-20170728
1 WTFCrash /usr/src/debug/wpe-webkit/0.4.4+gitAUTOINC+5f899bc2e0-r0/git/Source/WTF/wtf/Assertions.cpp:270 2 operationLinkDirectCall /usr/src/debug/wpe-webkit/0.4.4+gitAUTOINC+5f899bc2e0-r0/git/Source/JavaScriptCore/jit/JITOperations.cpp:999 ...
wpe-2.22 1 WTFCrash /usr/src/debug/wpe-webkit/2.22+gitAUTOINC+686cd2f7df-r0/git/Source/WTF/wtf/Assertions.cpp:255 2 operationLinkDirectCall /usr/src/debug/wpe-webkit/2.22+gitAUTOINC+686cd2f7df-r0/git/Source/JavaScriptCore/jit/JITOperations.cpp:1112 ...
2.22+gitAUTOINC+686cd2f7df-r0/git/Source/JavaScriptCore/jit/JITOperations.cpp:1112 https://github.com/WebPlatformForEmbedded/WPEWebKit/blob/87cbd6cb994c02ba37427c21fa095af7e5ac229d/Source/JavaScriptCore/jit/JITOperations.cpp#L1112
glibnovodran
commented
3 years ago @guijemont, my understanding is @woutermeek has received or will receive soon a more detailed information from Comcast as well as the MIPS-based device to reproduce with.
mbhatt627
commented
2 years ago There are random crash happens on JS Core( WPE -2.22). These crashes are seen only on MIPS devices. Crash happens on operationLinkDirectCall() 1- RELEASE_ASSERT(callLinkInfo->isDirect()); -> This crash is more , in a day 23K ~ 25K crashes 2- RELEASE_ASSERT(callLinkInfo->executable()); -> This less compare to 1. in a day crashes 400~500
There is no reproduction scenario.
WTFCrash_call_stack_isDirect_1112_1.txt WTFCrash_call_stack_isDirect_1112.txt
Attached few callstack from crash.
mbhatt627
commented
2 years ago Most of the times process's vm.peak exceeds vm.size as well vm.rss.peak exceeds vm.rss.size. Ex:
vm.rss.peak 239,296 vm.rss.size 211,668 vm.shared.size 104,340 vm.stack.size 136 vm.swap.size 1,308 vm.vma.peak 778,876 vm.vma.size 774,088
mbhatt627
commented
2 years ago @woutermeek , GCC version on Dunfell build is gcc version 9.3.0, where we see increse in this crash.
mbhatt627
commented
2 years ago @guijemont , Do you have any test case of JSC which can reproduce this crash ?
guijemont
commented
2 years ago @guijemont , Do you have any test case of JSC which can reproduce this crash ?
Not yet, though there is still hope that I could find one among the failures I see with qemu.
modeveci
commented
1 year ago Inactive ticket for long time!
Closing the ticket; if you think it is still relevant and/or valid for the latest version/s. Please do not hesitate to re-open!
0 WTFCrash libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/Assertions.cpp:324 (0x8)
1 operationLinkDirectCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/jit/JITOperations.cpp:953 (0x4)
2 @0x5ab86150
3 deref libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/ThreadSafeRefCounted.h:36 (0x8)
4 _fini libWPEWebKit.so.0.0.20161117
5 execute libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/jit/JITCode.cpp:81 (0xc)
6 executeCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/interpreter/Interpreter.cpp:952 (0x10)
7 call libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:39 (0x0)
8 boundThisNoArgsFunctionCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/JSBoundFunction.cpp:54 (0x14)
9 libWPEWebKit.so.0.0.20161117@0x69858c libWPEWebKit.so.0.0.20161117
10 executeCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/interpreter/Interpreter.cpp:955 (0xc)
11 call libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:39 (0x0)
12 profiledCall libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/runtime/CallData.cpp:59 (0x0)
13 JSObjectCallAsFunction libWPEWebKit.so.0.0.20161117 Source/JavaScriptCore/API/JSObjectRef.cpp:541 (0x38)
14 libCUSTOMPlayer.so.0.0.0@0x7be70 libCUSTOMPlayer.so.0.0.0
15 libCUSTOMPlayer.so.0.0.0@0x6bc58 libCUSTOMPlayer.so.0.0.0
16 libCUSTOMPlayer.so.0.0.0@0x740978 libCUSTOMPlayer.so.0.0.0
17 libCUSTOMPlayer.so.0.0.0@0x6cf40 libCUSTOMPlayer.so.0.0.0
18 libCUSTOMPlayer.so.0.0.0@0x72768 libCUSTOMPlayer.so.0.0.0
19 libCUSTOMPlayer.so.0.0.0@0x1f8228 libCUSTOMPlayer.so.0.0.0
20 libCUSTOMPlayer.so.0.0.0@0x1f8400 libCUSTOMPlayer.so.0.0.0
21 libCUSTOMPlayer.so.0.0.0@0x1f8750 libCUSTOMPlayer.so.0.0.0
22 WPECallbackManager::eventPosted()::{lambda(void)#1}::_FUN(void) libComcastInjectedBundle.so
23 g_timeout_dispatch libglib-2.0.so.0.4800.1 glib-2.0/1_2.48.1-r0/glib-2.48.1/glib/gmain.c:4577 (0x4)
24 g_main_context_dispatch libglib-2.0.so.0.4800.1 glib-2.0/1_2.48.1-r0/glib-2.48.1/glib/gmain.c:3154 (0x0)
25 g_main_context_iterate libglib-2.0.so.0.4800.1 glib-2.0/1_2.48.1-r0/glib-2.48.1/glib/gmain.c:3840 (0x4)
26 g_main_loop_run libglib-2.0.so.0.4800.1 glib-2.0/1_2.48.1-r0/glib-2.48.1/glib/gmain.c:4034 (0xc)
27 run libWPEWebKit.so.0.0.20161117 Source/WTF/wtf/glib/RunLoopGLib.cpp:97 (0x4)
28 ChildProcessMain<WebKit::WebProcess, WebKit::WebProcessMain> libWPEWebKit.so.0.0.20161117 Source/WebKit2/Shared/unix/ChildProcessMain.h:61 (0x4)
29 main WPEWebProcess Source/WebKit2/WebProcess/EntryPoint/unix/WebProcessMain.cpp:52 (0x8)
30 libc-2.19.so@0x194a4 libc-2.19.so
31 libgcc_s.so.1@0x2e18 libgcc_s.so.1
32 ld-2.19.so@0xd80 ld-2.19.so