Crash in TextureAtlas::drawNumberOfQuads on Android Nougat

[haingo-nama]

We see crashes from our new game caused by TextureAtlas::drawNumberOfQuads on Android Nougat (7.x) devices. I can find similar issue here https://github.com/cocos2d/cocos2d-x/issues/18743 but it doesn't seem to have adequate information for the cocos developer to fix it.

When the game was under development, we could reproduce the crash on Samsung Galaxy S6 Android 7.0 with CC_TEXTURE_ATLAS_USE_VAO is 1. Then we set it to 0 and the problem has gone on our device (as suggested in the thread above).

But after releasing the game on Google Play (beta), we see the same crash coming from other Android 7.x devices:

• Moto G (5th generation)
• Moto G Plus (5th generation)
• Redmi 4X
• Galaxy A9 Pro There could be more devices sharing the same fate as the game has just been released a few days.

Below is the environment setup and steps to reproduce the issue.

Environment setup:

• cocos2d-x version: 3.17
• Devices: Android Nougat (7.x)
• NDK version: 17.2.4988734

Steps to Reproduce:

1. Open the app on Android Nougat device
2. Watch a Video Ad
3. The app might crash after finishing watching the video ad

Crash log: /system/lib/libc.so (memcpy+96) /system/vendor/lib/egl/libESXGLESv2_adreno.so (_ZN20EsxVertexArrayObject18UpdateInternalVbosEPK17EsxDrawDescriptorjPK16EsxAttributeDesc+1206) /system/vendor/lib/egl/libESXGLESv2_adreno.so (_ZN20A5xVertexArrayObject11CalcVfdRegsEPK17EsxDrawDescriptorP10A5xVfdRegsi+94) /system/vendor/lib/egl/libESXGLESv2_adreno.so (_ZN10A5xContext16ValidateVfdStateEPK17EsxDrawDescriptori+28) /system/vendor/lib/egl/libESXGLESv2_adreno.so (_ZN10A5xContext13ValidateStateEPK17EsxDrawDescriptor+2400) /system/vendor/lib/egl/libESXGLESv2_adreno.so (_ZN10A5xContext18HwValidateGfxStateEPK17EsxDrawDescriptor+4) /system/vendor/lib/egl/libESXGLESv2_adreno.so (_ZN10EsxContext16ValidateGfxStateEPK17EsxDrawDescriptor+758) /system/vendor/lib/egl/libESXGLESv2_adreno.so (_ZN10EsxContext21DrawElementsInstancedE11EsxPrimTypej10EsxPixTypePKvji+338) /system/vendor/lib/egl/libESXGLESv2_adreno.so (_ZN10EsxContext14GlDrawElementsEjijPKv+68) /system/vendor/lib/egl/libESXGLESv2_adreno.so (glDrawElements+34) /data/app/com.amanotes.guitar-1/lib/arm/libnama.so (_ZN7cocos2d12TextureAtlas17drawNumberOfQuadsEii+460) /data/app/com.amanotes.guitar-1/lib/arm/libnama.so (_ZN7cocos2d5Label6onDrawERKNS_4Mat4Eb+380) /data/app/com.amanotes.guitar-1/lib/arm/libnama.so (_ZN7cocos2d8Renderer16visitRenderQueueERNS_11RenderQueueE+872) /data/app/com.amanotes.guitar-1/lib/arm/libnama.so (_ZN7cocos2d8Renderer6renderEv+96) /data/app/com.amanotes.guitar-1/lib/arm/libnama.so (_ZN7cocos2d5Scene6renderEPNS_8RendererEPKNS_4Mat4ES5_j+504) /data/app/com.amanotes.guitar-1/lib/arm/libnama.so (_ZN7cocos2d5Scene6renderEPNS_8RendererERKNS_4Mat4EPS4_+28) /data/app/com.amanotes.guitar-1/lib/arm/libnama.so (_ZN7cocos2d8Director9drawSceneEv+468) /data/app/com.amanotes.guitar-1/lib/arm/libnama.so (_ZN7cocos2d8Director8mainLoopEv+132) /data/app/com.amanotes.guitar-1/oat/arm/base.odex

Please give us suggestion to fix this issue.

[crazyhappygame]

Try change ndk to ndk16. All Android prebuilds wers built with ndk16

[haingo-nama]

Thank you, we will try with NDK 16.

Please note that it will take some time get the results as we need to release the game then check if the crash is still there on users' devices or not.

[haingo-nama]

So we built the game with NDK16 and the crash still happens. With CC_TEXTURE_ATLAS_USE_VAO = 1, the crash happens more often. With CC_TEXTURE_ATLAS_USE_VAO = 0, the crash doesn't happen any more on Galaxy S6 Android 7.0, but it still on our Xiao Mi Redmi 5 Android 7.1.2.

I'm guessing this is not caused by NDK. We'll try with any other suggestion you give.

[crazyhappygame]

latest github version especially this pull request https://github.com/cocos2d/cocos2d-x/pull/19013 + ndk16?

[haingo-nama]

Yeah we will try that. But we might have to give you update next Monday as pulling the latest github and build the game with it is not a trivial task while it's now Friday afternoon our time.

[haingo-nama]

So we tried with latest github and NDK 16. The same crash still happens.

[haingo-nama]

Seems like we found a solution for this issue. The problem might be the callback function is triggered after the video is closed (after fully watched). That callback alters the scene (add/remove some nodes) and is triggered in the thread managed by 3rd party ads sdk and different from the update thread. We suspect this is the root of the problem. So we decided to have an (atomic) variable indicating the state of video ad (loading, playing, failed, watched, closed, etc, ...) which will be updated by 3rd ads callbacks. Then we process this variable in update thread and change the scene accordingly.

After using this solution, we can't reproduce the crash on our devices. But to be sure, we will update this thread after monitoring the game when it goes live.

[aqaq1990123]

did you fix it?

[haingo-nama]

Hi everyone, the crash is gone with the solution we mentioned above. We don't see this crash coming from users in Google Play Console any more. I think we can close this issue now :)

Thank you for your support.

[stephenhurry]

@haingo-nama Hi may you give more hints on the implementation of atomic variable? I am encountering the same issue, thanks!

[haingo-nama]

@haingo-nama Hi may you give more hints on the implementation of atomic variable? I am encountering the same issue, thanks!

Just don't modify the scene in any other thread rather than one managed by cocos. Whenever you call new Node, new Label, ... or any statement that modifies the scene, you should call it in Init, OnEnter, Update, ... or any cocos function that you know it's managed by cocos.

In our case, we have to show reward to the user after they watch a video. The callback of rewarded video is in another thread (which is managed by the ads SDK). So we implemented it like:

` class RewadedVideoButton : Node { atomic<bool> videoWatched;

void init()
{
    videoWatched = false;
}

void showRewardedVideo()
{
    ads.showRewadedVideo(CC_CALLBACK(this, RewadedVideoButton::onRewadedVideoWatched));
}

void onRewadedVideoWatched()
{
    // Modifying the scene here would make the game crash as it's called in a thread managed by ads
    // So we just mark the video as watched then it will be processed in cocos thread
    videoWatched = true;
}

void Update()
{
    if (videoWatched)
    {
        // Process the reward here. Show some fantastic image and text
        ...
        Label* text = new Label("Congratulation ...");
        ...
        videoWatched = false;
    }
}

}

`

This is not really our implementation as it's more complicated. But it's similar to how we fixed the issue.

[stephenhurry]

@haingo-nama Thanks! It definitely would help.

[drelaptop]

Same crash occurs to me at this situation:

1. back to the game after the SDK wroks finish
2. put works like new Label into the SDK callback directly

and the workaround I got is put the new Label works to next frame

seems like some issue in Label about the gl context restore

[zhongfq]

Recently, i find some reason about these crash, this is because you call openGL api(ex: create new label) before android Activity.onResume when openGL context is not ready, to fix these crash, you should put you call(call from java) into next frame:

JNIEXPORT void JNICALL Java_kernel_LuaJ_call (JNIEnv *env, jclass cls, jint jfunc, jstring jargs, jboolean jonce) { CC_UNUSED_PARAM(env); CC_UNUSED_PARAM(cls);

if (!xgame::runtime::isRestarting()) {
    std::string args = cocos2d::JniHelper::jstring2string(jargs);
    int func = (int)jfunc;
    bool once = (bool)jonce;
    auto listener = new EventListenerCustom();
    listener->autorelease();
    listener->init(Director::EVENT_BEFORE_UPDATE, [func, args, once, listener](EventCustom *event){
        Director::getInstance()->getEventDispatcher()->removeEventListener(listener);
        lua_State *L = olua_mainthread(NULL);
        int top = lua_gettop(L);
        olua_geterrorfunc(L);
        olua_getref(L, func);
        if (!lua_isnil(L, -1)) {
            lua_pushstring(L, args.c_str());
            lua_pcall(L, 1, 0, top + 1);
        } else {
            xgame::runtime::log("attempt to call nil: %d %s", func, args.c_str());
        }
        if (once) {
            olua_unref(L, func);
            xgame::runtime::log("remove func ref: %d", func);
        }
        lua_settop(L, top);
    });
    Director::getInstance()->getEventDispatcher()->addEventListenerWithFixedPriority(listener, 1);
}

}