Thread affinity for gstreamer-gl contexts?

[asajeffrey]

Building magicleap servo with the render-android PR (#308) works, but running it and viewing video produces errors:

09-12 13:23:21.998 16889     4 I NoTag   : ** (Servo2D:1): CRITICAL **: 13:23:21.998: gst_gl_context_thread_add: assertion 'context->priv->active_thread == g_thread_self ()' failed

According to @xclaesse this is caused by the gstreamer-gl context being created in one thread and used in another. The code that creates the context is https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/blob/master/gstreamer-gl/src/gl_context.rs#L25-30

    pub unsafe fn new_wrapped<T: IsA<GLDisplay>>(
        display: &T,
        handle: uintptr_t,
        context_type: GLPlatform,
        available_apis: GLAPI,
    ) -> Option<GLContext> {
        from_glib_full(gst_gl_sys::gst_gl_context_new_wrapped(
            display.as_ref().to_glib_none().0,
            handle,
            context_type.to_glib(),
            available_apis.to_glib(),
        ))
    }

which is called at https://github.com/servo/media/pull/308/files#diff-c7f05fb6a8c34d92da3ac21df409cee9R88-R93

The problem is that that initialization code is called from the script thread, as you can see from this backtrace:

#0  0x0000aaaab87d71e0 in servo_media_gstreamer_render_android::RenderAndroid::new::he5ddafe679ae0dff (app_gl_context=...)
#1  0x0000aaaab87d590c in servo_media_gstreamer::render::platform::create_render::h66f42d164255c2ed (gl_context=...)
    at /Users/ajeffrey/github/asajeffrey/media/backends/gstreamer/render.rs:39
#2  0x0000aaaab87cfad8 in servo_media_gstreamer::render::GStreamerRender::new::hf1ceb3060956f2f5 (gl_context=...)
    at /Users/ajeffrey/github/asajeffrey/media/backends/gstreamer/render.rs:99
#3  0x0000aaaab6fe5420 in servo_media_gstreamer::player::GStreamerPlayer::new::ha157902e3f41e1c0 (id=0, context_id=0x40004c9f47f0, 
    backend_chan=..., stream_type=Seekable, observer=..., renderer=..., gl_context=...)
    at /Users/ajeffrey/github/asajeffrey/media/backends/gstreamer/player.rs:387
#4  0x0000aaaab6fba5e8 in _$LT$servo_media_gstreamer..GStreamerBackend$u20$as$u20$servo_media..Backend$GT$::create_player::h42c4c77f63ac7bd6 (
    self=0x40003da22c60, context_id=0x40004c9f47f0, stream_type=Seekable, sender=..., renderer=..., gl_context=...)
    at /Users/ajeffrey/github/asajeffrey/media/backends/gstreamer/lib.rs:154
#5  0x0000aaaab751eac4 in script::dom::htmlmediaelement::HTMLMediaElement::setup_media_player::h55ed4bb8f3d5fc52 (self=0x400047ee0000, 
    resource=0x40004c9f4d88) at components/script/dom/htmlmediaelement.rs:1337
#6  0x0000aaaab751bcf0 in script::dom::htmlmediaelement::HTMLMediaElement::resource_fetch_algorithm::h1b126da150aab687 (self=0x400047ee0000, 
    resource=...) at components/script/dom/htmlmediaelement.rs:880
#7  0x0000aaaab7519d84 in script::dom::htmlmediaelement::HTMLMediaElement::resource_selection_algorithm_sync::h1d06a7c580275871 (
    self=0x400047ee0000, base_url=...) at components/script/dom/htmlmediaelement.rs:774
#8  0x0000aaaab7525b1c in _$LT$script..dom..htmlmediaelement..MediaElementMicrotask$u20$as$u20$script..microtask..MicrotaskRunnable$GT$::handler::h3b3efcaca733b88e (self=0x400047e0e5c8) at components/script/dom/htmlmediaelement.rs:2372
#9  0x0000aaaab22b7208 in script::microtask::MicrotaskQueue::checkpoint::hd51e4bb8e0e54cab (self=0x400047e18010, cx=..., target_provider=..., 
    globalscopes=...) at components/script/microtask.rs:99
#10 0x0000aaaab307ece4 in script::script_thread::ScriptThread::perform_a_microtask_checkpoint::h08fa3fc73b0065a2 (self=0x40004c9fc7f8)
    at components/script/script_thread.rs:3839
#11 0x0000aaaab305f2d8 in script::script_thread::ScriptThread::handle_msgs::hdb141a07048a3a91 (self=0x40004c9fc7f8)
    at components/script/script_thread.rs:1538
#12 0x0000aaaab305cfc0 in script::script_thread::ScriptThread::start::h04f610f1d4d641d0 (self=0x40004c9fc7f8)
    at components/script/script_thread.rs:1354
#13 0x0000aaaab1b8b8c4 in _$LT$script..script_thread..ScriptThread$u20$as$u20$script_traits..ScriptThreadFactory$GT$::create::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h79250945d7231557 () at components/script/script_thread.rs:800
#14 0x0000aaaab2ff9428 in profile_traits::mem::ProfilerChan::run_with_memory_reporting::h0738dff972ea7c3e (self=0x40004c9fc7c8, f=..., 
    reporter_name=..., channel_for_reporter=..., msg=0x0) at /Users/ajeffrey/github/asajeffrey/servo/components/profile_traits/mem.rs:88
#15 0x0000aaaab1b8bf60 in _$LT$script..script_thread..ScriptThread$u20$as$u20$script_traits..ScriptThreadFactory$GT$::create::_$u7b$$u7b$closure$u7d$$u7d$::h68c6b269cc8d0c46 () at components/script/script_thread.rs:798
#16 0x0000aaaab7ed791c in std::sys_common::backtrace::__rust_begin_short_backtrace::h2fc8dcbcd681c66b (f=...)
    at /rustc/521d78407471cb78e9bbf47160f6aa23047ac499/src/libstd/sys_common/backtrace.rs:77
#17 0x0000aaaab78d1220 in std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hba7d578e6a9471c6
    () at /rustc/521d78407471cb78e9bbf47160f6aa23047ac499/src/libstd/thread/mod.rs:470
#18 0x0000aaaab2d40b1c in _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h909e5fab292c8955 (self=..., _args=0) at /rustc/521d78407471cb78e9bbf47160f6aa23047ac499/src/libstd/panic.rs:315
#19 0x0000aaaab7c27580 in std::panicking::try::do_call::h31c15f0246044556 (data=0x40004c9fea58 "")
    at /rustc/521d78407471cb78e9bbf47160f6aa23047ac499/src/libstd/panicking.rs:296
#20 0x0000aaaab6d10264 in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:80
#21 0x0000aaaab7bb1368 in std::panicking::try::hd82502bf334749ae (f=...)
    at /rustc/521d78407471cb78e9bbf47160f6aa23047ac499/src/libstd/panicking.rs:275
#22 0x0000aaaab2d7e87c in std::panic::catch_unwind::h38095efb1f7d9876 (f=...)
    at /rustc/521d78407471cb78e9bbf47160f6aa23047ac499/src/libstd/panic.rs:394
#23 0x0000aaaab78cf8c4 in std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h6cc3de819432004c ()
    at /rustc/521d78407471cb78e9bbf47160f6aa23047ac499/src/libstd/thread/mod.rs:469
#24 0x0000aaaab28a9e74 in core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hfaa5465cfad021bf ()
    at /rustc/521d78407471cb78e9bbf47160f6aa23047ac499/src/libcore/ops/function.rs:235
#25 0x0000aaaab6cfbc94 in _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::h54133a55e7b72b3d ()
    at /rustc/521d78407471cb78e9bbf47160f6aa23047ac499/src/liballoc/boxed.rs:922
#26 0x0000aaaab6d0fb70 in _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::h329f858a0894a5e8 ()
    at /rustc/521d78407471cb78e9bbf47160f6aa23047ac499/src/liballoc/boxed.rs:922
#27 std::sys_common::thread::start_thread::h7a4d7f539a8f95e5 () at src/libstd/sys_common/thread.rs:13
#28 std::sys::unix::thread::Thread::new::thread_start::hdba27bbdb682597e () at src/libstd/sys/unix/thread.rs:79
#29 0x000040003abf56f8 in __pthread_start(void*) ()
   from /Users/ajeffrey/github/asajeffrey/servo/support/magicleap/Servo2D/.out/debug_lumin_clang-3.8_aarch64/libc.so
#30 0x000040003aba87d0 in __start_thread ()
   from /Users/ajeffrey/github/asajeffrey/servo/support/magicleap/Servo2D/.out/debug_lumin_clang-3.8_aarch64/libc.so

So we have the gstreamer-gl context being created in the script thread, but then used in the gstreamer render thread.

[jdm]

It seems strange that this has not appeared to be a problem on Linux, given this similar code: https://github.com/servo/media/blob/f9aed4e41457321b8b646cf1f5501c6e0c1e8532/backends/gstreamer/render-unix/lib.rs#L165

[asajeffrey]

Indeed, I'm hoping to try this on a linux box when I get home. It would be that the threading requirements are different between GL/EGL?

[asajeffrey]

The threading requirements at https://gstreamer.freedesktop.org/documentation/gl/gstglcontext.html say "As a result of the limitation on OpenGL context, this object is not thread safe unless specified and must only be activated in a single thread."

I think what we're doing is creating it in the script thread, then activating it in the gstreamer render thread, which according to ^ is safe, but apparently not.

[asajeffrey]

Ah... https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/blob/master/gst-libs/gst/gl/gstglcontext.c#L1580-1581

void
gst_gl_context_thread_add (GstGLContext * context,
    GstGLContextThreadFunc func, gpointer data)
{
  GstGLWindow *window;
  RunGenericData rdata;

  g_return_if_fail (GST_IS_GL_CONTEXT (context));
  g_return_if_fail (func != NULL);

  if (GST_IS_GL_WRAPPED_CONTEXT (context))
    g_return_if_fail (context->priv->active_thread == g_thread_self ());

That last assertion is what's failing, and is due to the test if (GST_IS_GL_WRAPPED_CONTEXT (context)), that is you can't use gst_gl_context_thread_add to call a function in the active thead of a wrapped GL context (which is our case, since the context is created with gst_gl::GLContext::new_wrapped()).

[xclaesse]

As far so know the problem is not in which thread you create the GST gl context, but where you activate it.

[asajeffrey]

@xclaesse yes, but it looks like you can only use gst_gl_context_thread_add on unwrapped contexts?

[asajeffrey]

OK, if the problem is that we're using a wrapped context, perhaps the thing to do is create an unwrapped context which shares with the wrapped context, and use that as the context for the player?

[ceyusa]

As @xclaesse said, the problem is in which thread the wrapped gl context is activated:

https://github.com/servo/media/pull/308/files#diff-c7f05fb6a8c34d92da3ac21df409cee9R106

We should look por a place where to activate it, but inside media-servo. The problem is, if we don't activate it before the gst pipeline uses it, the pipeline will probably fail (since gstgl cannot access the gl vtable)

[asajeffrey]

So replacing that wrapped GL context with an unwrapped context makes that problem go away:

    let wrapped_context = unsafe {
        gst_gl::GLContext::new_wrapped(
            &display,
            context,
            gst_gl::GLPlatform::EGL,
            gl_api,
       )
    };
    let unwrapped_context = gst_gl::GLContext::new(&display);
    let _ = unwrapped_context.create(wrapped_context.as_ref());
    (Some(unwrapped_context), Some(display))

but then this produces an error:

09-13 11:59:49.579  2363    48 E NoTag   : Player error: "Error from element /GstPlayBin:playbin/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstAmcVideoDec-OmxNvidiaH264Decode:amcvideodec-omxnvidiah264decode0: GStreamer encountered a general supporting library error.\nGStreamer encountered a general supporting library error.\n../gst-build/subprojects/gst-plugins-bad/sys/androidmedia/gstamcvideodec.c(1061): _gl_sync_render_unlocked (): /GstPlayBin:playbin/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstAmcVideoDec-OmxNvidiaH264Decode:amcvideodec-omxnvidiah264decode0:\nFailed to update tex image: -1086652416"
09-13 11:59:51.316  2363     5 I NoTag   : thread '<unnamed>' panicked at 'Caught GL error 502 at bind_texture', /Users/ajeffrey/.cargo/git/checkouts/webrender-c3596abe1cf4f320/4646982/webrender/src/device/gl.rs:1253:17

My guess is there are some points where the GL context known in the script thread is required to be the same GL context as the one used by the GL render thread?

[asajeffrey]

Doing a search for activate in servo/media, the only hit is https://github.com/servo/media/blob/6e864ed1b3c39e26fa39faad2ab99ef79d83b64d/backends/gstreamer/render-unix/lib.rs#L131 which is only in the unix back end, not the android back end, and is also activating the context on the script thread rather than the gstreamer render thread.

[ceyusa]

android backend also activates the context on the scripth thread:

https://github.com/servo/media/compare/master...ceyusa:render-android?expand=1#diff-c7f05fb6a8c34d92da3ac21df409cee9R106

you can try delete that block of code

[asajeffrey]

Oh good point, the android back end is in a PR, not checked in yet. I'm going to see if I can replicate the issue on a linux box.

[asajeffrey]

OK, I'm failing to work out why this works on unix but not magicleap. I understand why it's failing on magicleap, it's the "why does it work on unix" part that I'm struggling with. We create a wrapped context, but since it's a wrapped context we can't call gst_gl_context_thread_add on it, which is why the magicleap app is failing. But for some reason the unix app is happy, despite using the same logic.

[asajeffrey]

OK, setting a couple of breakpoints and running the unix program in lldb produces:

Process 28776 launched: '/home/ajeffrey/github/asajeffrey/servo/target/debug/servo' (x86_64)
1 location added to breakpoint 1
1 location added to breakpoint 2
Process 28776 stopped
* thread #43: tid = 29000, 0x00007ffff796b29e libgstgl-1.0.so.0`gst_gl_context_activate(context=&0x7fff8c0dc070L, activate=1) + 14 at gstglcontext.c:746, name = 'ScriptThread Pi', stop reason = breakpoint 2.1
    frame #0: 0x00007ffff796b29e libgstgl-1.0.so.0`gst_gl_context_activate(context=&0x7fff8c0dc070L, activate=1) + 14 at gstglcontext.c:746
(lldb) c
Process 28776 resuming
Process 28776 stopped
* thread #64: tid = 29104, 0x00007ffff796b29e libgstgl-1.0.so.0`gst_gl_context_activate(context=&0x5555630ba120L, activate=1) + 14 at gstglcontext.c:746, name = 'gstglcontext', stop reason = breakpoint 2.1
    frame #0: 0x00007ffff796b29e libgstgl-1.0.so.0`gst_gl_context_activate(context=&0x5555630ba120L, activate=1) + 14 at gstglcontext.c:746
(lldb) c
Process 28776 resuming
Process 28776 stopped
* thread #65: tid = 29103, 0x00007ffff796bcc3 libgstgl-1.0.so.0`gst_gl_context_thread_add(context=&0x5555630ba120L, func=0x00007ffff7962310, data=0x00007fff8c0fc460) + 35 at gstglcontext.c:1562, name = 'typefind:sink', stop reason = breakpoint 1.1
    frame #0: 0x00007ffff796bcc3 libgstgl-1.0.so.0`gst_gl_context_thread_add(context=&0x5555630ba120L, func=0x00007ffff7962310, data=0x00007fff8c0fc460) + 35 at gstglcontext.c:1562
(lldb) c
Process 28776 resuming
Process 28776 stopped
* thread #65: tid = 29103, 0x00007ffff796bcc3 libgstgl-1.0.so.0`gst_gl_context_thread_add(context=&0x5555630ba120L, func=0x00007ffff7962310, data=0x00007fff8c0fc0f0) + 35 at gstglcontext.c:1562, name = 'typefind:sink', stop reason = breakpoint 1.1
    frame #0: 0x00007ffff796bcc3 libgstgl-1.0.so.0`gst_gl_context_thread_add(context=&0x5555630ba120L, func=0x00007ffff7962310, data=0x00007fff8c0fc0f0) + 35 at gstglcontext.c:1562
(lldb) c
Process 28776 resuming
Process 28776 stopped
* thread #74: tid = 29227, 0x00007ffff796bcc3 libgstgl-1.0.so.0`gst_gl_context_thread_add(context=&0x5555630ba120L, func=0x00007ffff79640a0, data=0x00005555630bc120) + 35 at gstglcontext.c:1562, name = 'vqueue:src', stop reason = breakpoint 1.1
    frame #0: 0x00007ffff796bcc3 libgstgl-1.0.so.0`gst_gl_context_thread_add(context=&0x5555630ba120L, func=0x00007ffff79640a0, data=0x00005555630bc120) + 35 at gstglcontext.c:1562
(lldb) c
Process 28776 resuming
Process 28776 stopped
* thread #67: tid = 29131, 0x00007ffff796bcc3 libgstgl-1.0.so.0`gst_gl_context_thread_add(context=&0x5555630ba120L, func=0x00007ffff7962ec0, data=0x00007fff591fb460) + 35 at gstglcontext.c:1562, name = 'multiqueue0:src', stop reason = breakpoint 1.1
    frame #0: 0x00007ffff796bcc3 libgstgl-1.0.so.0`gst_gl_context_thread_add(context=&0x5555630ba120L, func=0x00007ffff7962ec0, data=0x00007fff591fb460) + 35 at gstglcontext.c:1562
(lldb) c
Process 28776 resuming
Process 28776 stopped
* thread #64: tid = 29104, 0x00007ffff796bcc3 libgstgl-1.0.so.0`gst_gl_context_thread_add(context=&0x5555630ba120L, func=0x00007ffff7962ec0, data=0x00007fff5da9a140) + 35 at gstglcontext.c:1562, name = 'gstglcontext', stop reason = breakpoint 1.1
    frame #0: 0x00007ffff796bcc3 libgstgl-1.0.so.0`gst_gl_context_thread_add(context=&0x5555630ba120L, func=0x00007ffff7962ec0, data=0x00007fff5da9a140) + 35 at gstglcontext.c:1562
(lldb) c
Process 28776 resuming
Process 28776 stopped
* thread #67: tid = 29131, 0x00007ffff796bcc3 libgstgl-1.0.so.0`gst_gl_context_thread_add(context=&0x5555630ba120L, func=0x00007ffff7962ec0, data=0x00007fff591fb460) + 35 at gstglcontext.c:1562, name = 'multiqueue0:src', stop reason = breakpoint 1.1
    frame #0: 0x00007ffff796bcc3 libgstgl-1.0.so.0`gst_gl_context_thread_add(context=&0x5555630ba120L, func=0x00007ffff7962ec0, data=0x00007fff591fb460) + 35 at gstglcontext.c:1562

that is the context that is activated in the script thread is not the one being activated in the render thread and having gst_gl_context_thread_add called on it. When I'm back in the office I can see what's happening on magicleap.

[asajeffrey]

Sigh, that would be too easy wouldn't it? The magicleap build has no symbol gst_gl_context_thread_add (presumably it got inlined) and when gdb traps the error, the backtrace isn't much use:

Thread 2 "vqueue:src" received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 20232.20355]
0x000040002b95fbfc in tgkill ()
   from /Users/ajeffrey/github/asajeffrey/servo/support/magicleap/Servo2D/.out/debug_lumin_clang-3.8_aarch64/libc.so
(gdb) bt
#0  0x000040002b95fbfc in tgkill ()
   from /Users/ajeffrey/github/asajeffrey/servo/support/magicleap/Servo2D/.out/debug_lumin_clang-3.8_aarch64/libc.so
#1  0x000040002b95cf44 in pthread_kill ()
   from /Users/ajeffrey/github/asajeffrey/servo/support/magicleap/Servo2D/.out/debug_lumin_clang-3.8_aarch64/libc.so
#2  0x000040002b9167bc in raise ()
   from /Users/ajeffrey/github/asajeffrey/servo/support/magicleap/Servo2D/.out/debug_lumin_clang-3.8_aarch64/libc.so
#3  0x000040002b2aac20 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[xclaesse]

gst_gl_context_thread_add is part of the ABI, so it shouldn't have been inlined. Maybe you have stripped/optimized build?

[asajeffrey]

@xclaesse the build is the one produced by https://github.com/servo/servo/tree/master/support/magicleap/gstreamer, which is I believe is optimized but not stripped.

[asajeffrey]

Spent all of today working out how to get gdb to play ball with gstreamer on magicleap. https://github.com/servo/servo/pull/24224/commits/b711d12344b034acf43e3a9657d1998f205fe91e

[asajeffrey]

Argh, and after all that, yes I can find where the error is happening, but no I'm not sure it's any more use because the backtrace is still garbled:

Thread 71 "gstglcontext" hit Breakpoint 3, gst_gl_context_activate (context=0x40003b349070, activate=1)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:746
746   g_return_val_if_fail (GST_IS_GL_CONTEXT (context), FALSE);
(gdb) 
Continuing.
[Switching to Thread 22847.22944]

Thread 74 "typefind:sink" hit Breakpoint 4, gst_gl_context_thread_add (context=0x40003b349070, 
    func=0x4000345d5e8c <gst_gl_base_filter_gl_start>, data=0x40003b0a4480)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.

...

Thread 72 "GstPlayer" hit Breakpoint 4, gst_gl_context_thread_add (context=0x40003b349070, func=0x40002d14bc6c, data=0x40003b6ff310)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[Switching to Thread 22847.22964]

Thread 75 "vqueue:src" hit Breakpoint 4, gst_gl_context_thread_add (context=0x40003b349070, func=0x4000345d81e4 <_map_data_gl>, 
    data=0x400045cf88e8) at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[New Thread 22847.22975]

Thread 75 "vqueue:src" hit Breakpoint 4, gst_gl_context_thread_add (context=0x40003b385070, func=0x40002d14a500, data=0x400045cf93d8)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) bt
#0  gst_gl_context_thread_add (context=0x40003b385070, func=0x40002d14a500, data=0x400045cf93d8)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
#1  0x000040002d149c6c in ?? ()
#2  0x000040003b10c920 in ?? ()

There is a bit of a clue, in that Servo happily runs along with context 0x40003b349070 but then context 0x40003b385070 comes out of nowhere.

[asajeffrey]

Confirming that on Linux there's only one GstGLContext argument to gst_gl_context_thread_add. @xclaesse any ideas where the new context might be coming from?

[xclaesse]

The fact that it doesn't have the function symbol for the caller of gst_gl_context_thread_add() indicates it's probably called from Servo and not from GStreamer, otherwise it would have the symbol like all others. Could you grep the code base to see if that's called anywhere? Or try teaching gdb where to look for Servo symbols too.

You could add breakpoint in gst_gl_context_new() and gst_gl_context_new() and print the returned pointer, so you could have an indication who created that context.

[asajeffrey]

It's a bit annoying that there's no debug symbol for the caller. I'll have a look to see if I can find the context creator.

[asajeffrey]

OK, here's the trace featuring the calls to gst_gl_context_new*:

Thread 2 "ScriptThread Pi" hit Breakpoint 6, 0x0000aaaab178b984 in gst_gl_context_new_wrapped@plt ()
(gdb) fin
Run till exit from #0  0x0000aaaab178b984 in gst_gl_context_new_wrapped@plt ()

Thread 2 "ScriptThread Pi" hit Breakpoint 5, gst_gl_context_new_wrapped (display=0x40002bc4f840, handle=70369465933856, 
    context_type=GST_GL_PLATFORM_EGL, available_apis=GST_GL_API_GLES2)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:394
394   GstGLWrappedContext *context_wrap = NULL;
(gdb) finish
Run till exit from #0  gst_gl_context_new_wrapped (display=0x40002bc4f840, handle=70369465933856, context_type=GST_GL_PLATFORM_EGL, 
    available_apis=GST_GL_API_GLES2) at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:394
return value in registers
return value in registers
0x0000aaaab7a30528 in gstreamer_gl::gl_context::_$LT$impl$u20$gstreamer_gl..auto..gl_context..GLContext$GT$::new_wrapped::haf23710c831ae933 (
    display=0x400039bf3ee0, handle=70369465933856, context_type=..., available_apis=...)
    at /Users/ajeffrey/.cargo/registry/src/github.com-1ecc6299db9ec823/gstreamer-gl-0.14.0/src/gl_context.rs:25
25          from_glib_full(gst_gl_sys::gst_gl_context_new_wrapped(
Value returned is $1 = (GstGLContext *) 0x40003a43e070
(gdb) c
Continuing.
[New Thread 1781.1898]
[New Thread 1781.1895]
[New Thread 1781.1896]
[New Thread 1781.1897]
[Switching to Thread 1781.1898]

Thread 70 "typefind:sink" hit Breakpoint 4, gst_gl_context_new (display=0x40002bc4f840)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:326
326   GstGLContext *context = NULL;
(gdb) bt
#0  gst_gl_context_new (display=0x40002bc4f840) at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:326
#1  0x000040001f5db598 in gst_gl_display_create_context (display=0x40002bc4f840, other_context=0x40003a43e070, p_context=0x40002bd426c8, 
    error=0x4000327f62e0) at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstgldisplay.c:521
#2  0x000040001f5c98a4 in gst_gl_base_filter_find_gl_context (filter=0x40002bd42480)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglbasefilter.c:443
#3  0x000040001e89d4d8 in ?? ()
#4  0x000040002bd4ce30 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) finish
Run till exit from #0  gst_gl_context_new (display=0x40002bc4f840)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:326
return value in registers
return value in registers
0x000040001f5db598 in gst_gl_display_create_context (display=0x40002bc4f840, other_context=0x40003a43e070, p_context=0x40002bd426c8, 
    error=0x4000327f62e0) at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstgldisplay.c:521
521   context = gst_gl_context_new (display);
Value returned is $2 = (GstGLContext *) 0x40002bd55070
(gdb) c
Continuing.
[New Thread 1781.1902]
[New Thread 1781.1903]

Thread 70 "typefind:sink" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, 
    func=0x40001f5c9e8c <gst_gl_base_filter_gl_start>, data=0x40002bd42480)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.

Thread 70 "typefind:sink" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, 
    func=0x40001f5c9e8c <gst_gl_base_filter_gl_start>, data=0x40002bd42110)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[New Thread 1781.1928]
[New Thread 1781.1909]
[New Thread 1781.1910]
[New Thread 1781.1911]
[New Thread 1781.1912]
[New Thread 1781.1915]
[New Thread 1781.1916]
[New Thread 1781.1918]
[New Thread 1781.1917]
[New Thread 1781.1921]
[New Thread 1781.1926]
[New Thread 1781.1927]
[New Thread 1781.1929]
[New Thread 1781.1930]
[Switching to Thread 1781.1928]

Thread 76 "vqueue:src" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001f5ce2c8 <_reset_gl>, 
    data=0x40002bfc2100) at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[Switching to Thread 1781.1926]

Thread 86 "amcvideodec-omx" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001f5cae24 <_mem_create_gl>, 
    data=0x4000356fdd50) at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.

Thread 86 "amcvideodec-omx" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001e9a49d0, data=0x40002cbb7da0)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[Switching to Thread 1781.1928]

Thread 76 "vqueue:src" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001e9a6c6c, data=0x40002cf65710)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.

Thread 76 "vqueue:src" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001e9a6c6c, data=0x40002cf65690)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[New Thread 1781.1931]

Thread 76 "vqueue:src" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001e9a6c6c, data=0x40002cf65610)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[New Thread 1781.1932]
[Switching to Thread 1781.1926]

Thread 86 "amcvideodec-omx" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001e9a6c6c, data=0x40002bd04990)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[Switching to Thread 1781.1896]

Thread 72 "GstPlayer" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001e9a6c6c, data=0x40002cf65490)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[Switching to Thread 1781.1928]

Thread 76 "vqueue:src" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001e9a6c6c, data=0x400040c863d0)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.

Thread 76 "vqueue:src" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001e9a6c6c, data=0x400040c86450)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[New Thread 1781.1933]

Thread 76 "vqueue:src" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001e9a6c6c, data=0x40002bd04990)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[Switching to Thread 1781.1926]

Thread 86 "amcvideodec-omx" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001e9a6c6c, data=0x40003a70d0d0)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[New Thread 1781.1934]
[Switching to Thread 1781.1896]

Thread 72 "GstPlayer" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001e9a6c6c, data=0x400040c861d0)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.
[Switching to Thread 1781.1928]

Thread 76 "vqueue:src" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40002bd55070, func=0x40001f5cc1e4 <_map_data_gl>, 
    data=0x400035ef58e8) at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.

Thread 76 "vqueue:src" hit Breakpoint 3, gst_gl_context_thread_add (context=0x40003a43e070, func=0x40001e9a5500, data=0x400035ef63d8)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1562
1562      g_return_if_fail (GST_IS_GL_CONTEXT (context));
(gdb) 
Continuing.

The context that is working came from gst_gl_base_filter_find_gl_context, the one that didn't came from RenderAndroid::new.

I'm back to not understanding how this is working in Linux, since it's essentially the same code.

[asajeffrey]

Ah, a different error, this one looking more sensible...

09-17 11:44:41.458  3097     5 I NoTag   : ** (Servo2D:1): CRITICAL **: 11:44:41.458: gst_gl_display_egl_new_with_egl_display: assertion 'display != NULL' failed

so now I need to work out why the display provided by magicleap (which is display 1) isn't making it to gstreamer.

[asajeffrey]

And that would be because https://github.com/servo/servo/blob/540a73ed2970948e916db0c53880ca20cb5083e9/ports/libmlservo/src/lib.rs#L176-L177 doesn't pass along the display and context it was given as arguments.

[asajeffrey]

OK, back to the original problem...

_$LT$servo_media_gstreamer_render_android..RenderAndroid$u20$as$u20$servo_media_gstreamer_render..Render$GT$::build_frame::h731dab0ed5eba503 (
    self=0x4000438292a0, sample=...)
    at /Users/ajeffrey/.cargo/git/checkouts/media-c23a3cd5aa97076a/d6f799e/backends/gstreamer/render-android/lib.rs:149
149         let info = gst_video::VideoInfo::from_caps(caps).ok_or_else(|| ())?;
(gdb) 
151         if self.gst_context.lock().unwrap().is_some() {
(gdb) 
152             if let Some(sync_meta) = buffer.get_meta::<gst_gl::GLSyncMeta>() {
(gdb) 
153                 sync_meta.set_sync_point(self.gst_context.lock().unwrap().as_ref().unwrap());
(gdb) 
158             gst_video::VideoFrame::from_buffer_readable_gl(buffer, &info).or_else(|_| Err(()))?;
(gdb) 
160         if self.gst_context.lock().unwrap().is_some() {
(gdb) 
160         if self.gst_context.lock().unwrap().is_some() {
(gdb) 
161             if let Some(sync_meta) = frame.buffer().get_meta::<gst_gl::GLSyncMeta>() {
(gdb) 
162                 sync_meta.wait(&self.app_context);
(gdb) s
_$LT$gstreamer..meta..MetaRef$LT$T$GT$$u20$as$u20$core..ops..deref..Deref$GT$::deref::h9dfaef36b5a0a91d (self=0x4000437f3d40)
    at /Users/ajeffrey/.cargo/registry/src/github.com-1ecc6299db9ec823/gstreamer-0.14.0/src/meta.rs:86
86          self.meta
(gdb) s
87      }
(gdb) s
gstreamer_gl::gl_sync_meta::GLSyncMeta::wait::h0dbd68e4e1961c1f (self=0x40003b577dc0, context=0x4000438292b0)
    at /Users/ajeffrey/.cargo/registry/src/github.com-1ecc6299db9ec823/gstreamer-gl-0.14.0/src/gl_sync_meta.rs:44
44                  &self.0 as *const _ as *mut _,
(gdb) s
45                  context.as_ref().to_glib_none().0,
(gdb) n
45                  context.as_ref().to_glib_none().0,
(gdb) n
43              gst_gl_sys::gst_gl_sync_meta_wait(
(gdb) s
[New Thread 4569.4710]
gst_gl_sync_meta_wait (sync_meta=0x40003b577dc0, context=0x400048c31070)
    at ../gst-build/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglsyncmeta.c:234
234   if (sync_meta->wait)
(gdb) s
235     sync_meta->wait (sync_meta, context);
(gdb) s

Thread 2 "vqueue:src" received signal SIGTRAP, Trace/breakpoint trap.

so the problem is https://github.com/servo/media/blob/d6f799e7d12e7d701d2cc38d40b40c57ddf75286/backends/gstreamer/render-android/lib.rs#L162

        if self.gst_context.lock().unwrap().is_some() {
            if let Some(sync_meta) = frame.buffer().get_meta::<gst_gl::GLSyncMeta>() {
                sync_meta.wait(&self.app_context);
            }
        }

since app_context is a wrapped GLContext, but calling wait calls https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/blob/dff5399e8750bf0f48efd23370f09aeddeb1d7d8/gst-libs/gst/gl/gstglsyncmeta.c#L231-239

void
gst_gl_sync_meta_wait (GstGLSyncMeta * sync_meta, GstGLContext * context)
{
  if (sync_meta->wait)
    sync_meta->wait (sync_meta, context);
  else
    gst_gl_context_thread_add (context,
        (GstGLContextThreadFunc) _wait, sync_meta);
}

and oh dear gst_gl_context_thread_add isn't allowed on wrapped GL contexts.

[xclaesse]

You're waiting on the wrong context. See here how to get GStreamer's context:

https://gitlab.freedesktop.org/xclaesse/gstreamer_demo/blob/master/VideoScene.cpp#L296

[asajeffrey]

IRC chat: https://mozilla.logbot.info/servo-magicleap/20190917#c16622502

[asajeffrey]

OK, a workaround is https://github.com/servo/media/pull/311 but I'm not convinced it's the Right Thing To Do, since it's only sync'ing the render thread, not the main thread, but since the main thread is pretty oblivious to gstreamer I don't see how to achieve that. Should the media stack have an API similar to webrender's for sync'ing textures?

[asajeffrey]

@jdm raises a good point over at https://mozilla.logbot.info/servo/20190917#c16622943

jdm wonders why the syncing is required on android and not on linux

[ceyusa]

Doing this implies to solve https://github.com/servo/servo/issues/24211 too