Android build fails on java bindings

[tezet]

Cerbero Android armv7 build is failing on openwebrtc recipe with following error:

make[3]: Entering directory `/root/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java'
PYTHONDONTWRITEBYTECODE=1 ../../bindings/java/gen_jni.py \
    --gir=Owr-0.3.gir \
    --c-out=../../bindings/java/owr_jni.c \
    --j-out=../../bindings/java/owr
reading from gir file "Owr-0.3.gir"
saving generated C source to "../../bindings/java/owr_jni.c"
saving generated Java source in "../../bindings/java/owr"
-------- BEGIN ---------
Traceback (most recent call last):
  File "../../bindings/java/gen_jni.py", line 195, in <module>
    sys.exit(main())
  File "../../bindings/java/gen_jni.py", line 175, in main
    namespaces = gir_parser.parse_full(type_registry)
  File "/root/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/gir_parser.py", line 602, in parse_full
    return [Namespace(type_registry, tag) for tag in self.xml_root.findall(TAG_NAMESPACE)]
  File "/root/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/gir_parser.py", line 551, in __init__
    self.callbacks = [Callback.from_tag(type_registry, t) for t in tag.findall(TAG_CALLBACK)]
  File "/root/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/gir_parser.py", line 369, in from_tag
    params=Parameters.from_tag(type_registry, tag),
  File "/root/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/gir_parser.py", line 205, in from_tag
    return_value = parse_tag_value(type_registry, tag.find(TAG_RETURN_VALUE), 'result')
  File "/root/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/gir_parser.py", line 145, in parse_tag_value
    typ = lookup_type(type_tag)
  File "/root/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/gir_parser.py", line 130, in lookup_type
    return type_registry.lookup(gir_type, c_type)
  File "/root/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/type_registry.py", line 81, in lookup
    raise LookupError("type lookup failed (gir_type=%s, c_type=%s)" % (gir_type, c_type))
**LookupError: type lookup failed (gir_type=None, c_type=GstContext*)**
make[3]: *** [owr_jni.c] Error 1
make[3]: Leaving directory `/root/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/root/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/cerbero/build/sources/android_armv7/openwebrtc-0.3.0'
make: *** [all] Error 2

I am building it on Ubuntu 16.04, same result on Ubuntu 14.04. It used to build fine few weeks ago.

[tezet]

The issue was introduced by this commit: https://github.com/EricssonResearch/openwebrtc/commit/0b28b080d61af3adb1f779e693fc029f9c1ad499

For some reason the entry for the VideoRendererRequestContextCallback in Owr-0.3.gir file is not properly generated.

[socay1]

The same problem on Centos 7, any solution to fix it?
@tezet

[superdump]

@philn ?

[DblA]

I tried to checkout commits before 0b28b08 when build failed, but the result was the same (on 16.04 LTS).

Tag at line 2355 at '~/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/Owr-0.3.gir' is missing name attribute. fixing line to <type name="none" c:type="GstContext*"/> gets things to compile. I have no idea if the name should be 'none' or something else though.

[Zubnix]

I think the core issue here is that Owr (secretly) exposes the GstContext type. The intermediate bindings representation tries to find a name in the owr namespace for type 'gstcontext' and does not find it because it's actually in the 'gst' namespace which owr obviously does not generate bindings for.

It's also very odd that the offending header/commit does not include gst.h which has the real gstcontext type, but instead simply typedefs an opaque gstcontext struct. It's almost as if the author did not want to include any gst dependencies... and then secretly still does...

The only way afaik to properly fix this is to define the gstcontext as a gpointer or maybe a gobject, or somehow provide and link the missing gst bindings.

[Rugvip]

Other non-standard types have been added to the bindings generator in bindings/java/gen_jni.py. Afaik it should be possible to add another dummy type similar to GMainContextDummy, https://github.com/EricssonResearch/openwebrtc/blob/b05d861b2dca146180ed01ab5ea3a753b18c3eb3/bindings/java/gen_jni.py#L136

[Zubnix]

That indeed fixes the issue on how to map a custom type to Java. However we still need to tell the scanner what name it should use when it encounters a gstcontext as currently it simply ommits the name attribute in the gir xml. (How does GLib.MainContext gets its name? I don't see any gir annotations).

[Rugvip]

Before the full Gir parsing and code generation is done, all known types are added to a type registry. Each type added to the type registry contains information about what type and variable names to use, how to convert type to and from java, how to clean up memory, etc.

When generating the OWR bindings, all types are added here. A bunch of standard types are registered from standard_types.py, e.g. numerical, string, list, hashmap. The gir file is also parsed for types, which is when all OWR types are added. A few custom types are also added, WindowHandleType and GMainContextDummy, which is the interesting bit in this case.

By using custom types it is possible to manually add types that are incomplete or not present in the gir file. In the case of GMainContextDummy, the gir and C names are set as a class attributes, and some dummy conversion rules are added. It should be possible to do the same for GstContext, although it's possible that a Java representation needs to be implemented in this case.

Btw, there's really no difference between custom and standard types, other than where they are defined. So looking at the standard types for inspiration makes sense.

[Zubnix]

@Rugvip thanks for the explanation.

It seems this commit: https://github.com/pererikb/openwebrtc/commit/a45e78395db04d63e325d2de5123b350cad49a5b

Fixes the missing name attribute. Together with your explanation, I think I should be able to make something work.

[Zubnix]

I've committed the missing parts on my fork: https://github.com/udevbe/openwebrtc/commit/142aeec473ac2df77a96e73a3632638aed994015

I haven't tested the fix, but looking at the C & Java code, it all seems to look ok and the build now properly succeeds.

[nakyup]

@Zubnix i tested it but same error

make[3]: 디렉터리 '/home/alex/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java' 들어감 PYTHONDONTWRITEBYTECODE=1 ../../bindings/java/gen_jni.py \ --gir=Owr-0.3.gir \ --c-out=../../bindings/java/owr_jni.c \ --j-out=../../bindings/java/owr reading from gir file "Owr-0.3.gir" saving generated C source to "../../bindings/java/owr_jni.c" saving generated Java source in "../../bindings/java/owr" -------- BEGIN --------- Traceback (most recent call last): File "../../bindings/java/gen_jni.py", line 203, in sys.exit(main()) File "../../bindings/java/gen_jni.py", line 180, in main namespaces = gir_parser.parse_full(type_registry) File "/home/alex/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/gir_parser.py", line 602, in parse_full return [Namespace(type_registry, tag) for tag in self.xml_root.findall(TAG_NAMESPACE)] File "/home/alex/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/gir_parser.py", line 551, in init self.callbacks = [Callback.from_tag(type_registry, t) for t in tag.findall(TAG_CALLBACK)] File "/home/alex/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/gir_parser.py", line 369, in from_tag params=Parameters.from_tag(type_registry, tag), File "/home/alex/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/gir_parser.py", line 205, in from_tag return_value = parse_tag_value(type_registry, tag.find(TAG_RETURN_VALUE), 'result') File "/home/alex/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/gir_parser.py", line 145, in parse_tag_value typ = lookup_type(type_tag) File "/home/alex/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/gir_parser.py", line 130, in lookup_type return type_registry.lookup(gir_type, c_type) File "/home/alex/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/bindings/java/type_registry.py", line 81, in lookup raise LookupError("type lookup failed (gir_type=%s, c_type=%s)" % (gir_type, c_type)) LookupError: type lookup failed (gir_type=None, c_type=GstContext*)

[Zubnix]

@nakyup Did you make sure your cerbero build updated the openwebrtc sources with the fix? Your error looks like it did not pick up https://github.com/pererikb/openwebrtc/commit/a45e78395db04d63e325d2de5123b350cad49a5b

Inside your cerbero dir (this is an build-time generated file): find . -name Owr-0.3.gir

Open that file and check if you see:

<return-value transfer-ownership="none">
      <doc xml:space="preserve">the GstContext</doc>
      <type name="GstContext" c:type="OwrGstContext*"/>
 </return-value>

[nakyup]

@Zubnix apply pererikb@a45e783 and check Owr-0.3.gir

  <return-value>
    <type c:type="GstContext*"/>
  </return-value>

something wrong?

[Zubnix]

@nakyup You have to make sure your cerbero project picks up the changes. By default cerbero is configured to checkout (and with it overwrite any changes) the default openwebrtc repo.

In your cerbero folder there is a file: cerbero/recipes/openwebrtc.recipe In that file you will see a section

remotes = {
        'origin': 'origin': 'https://github.com/EricssonResearch/openwebrtc.git'
    }

Make sure it points to a repository (ie your own personal repo) that has the fixes applied (ie you can fork the ericsson repo and apply the fixes to your personal fork).

[nakyup]

@Zubnix thanks!

1. create new cebero

2. edit cebero/recipes/openwebrtc.recipe remotes = { 'origin': 'origin': 'https://github.com/EricssonResearch/openwebrtc.git' } to 'origin': 'https://github.com/udevbe/openwebrtc.git' } commit = 'origin/openwebrtc-java' or https://github.com/nakyup/openwebrtc.git

3. check openwebrtc-0.3.0/bindings/java/Owr-0.3.gir it's ok

   
     <return-value transfer-ownership="none">
       <doc xml:space="preserve">the GstContext</doc>
       <type name="GstContext" c:type="OwrGstContext*"/>
     </return-value>

but build fail

libtool: link: arm-linux-androideabi-gcc -fuse-ld=gold -std=gnu99 -shared  -fPIC -DPIC  .libs/owr_jni.o   -L/home/alex/cerbero/build/sources/android_armv7/openwebrtc-0.3.0/owr/.libs -L/home/alex/cerbero/build/dist/android_armv7/lib -L/home/alex/cerbero/build/dist/android_armv7/lib -L/home/alex/cerbero/build/android-ndk-r12b/platforms/android-9/arch-arm/usr/lib -L/home/alex/cerbero/build/dist/android_armv7/lib -L/home/alex/cerbero/build/android-ndk-r12b/platforms/android-9/arch-arm/usr/lib -landroid ../../owr/.libs/libopenwebrtc.so -L/home/alex/cerbero/build/dist/android_armv7/lib/gstreamer-1.0/static -lsupc++ -lgirepository-internals /home/alex/cerbero/build/dist/android_armv7/lib/libnice.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstbadvideo-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstbadbase-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libgraphene-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libopenh264.so /home/alex/cerbero/build/dist/android_armv7/lib/libgnustl.so -lstdc++ /home/alex/cerbero/build/dist/android_armv7/lib/libopus.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstnet-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libgio-2.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libusrsctp.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstsctp-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libsrtp.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstcodecparsers-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstpbutils-1.0.so -lvpx /home/alex/cerbero/build/dist/android_armv7/lib/libgstcontroller-1.0.so -lOpenSLES /home/alex/cerbero/build/dist/android_armv7/lib/libssl.so /home/alex/cerbero/build/dist/android_armv7/lib/libcrypto.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstrtp-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstaudio-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libgsttag-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libz.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstapp-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstgl-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstvideo-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstallocators-1.0.so -lGLESv2 -lEGL /home/alex/cerbero/build/dist/android_armv7/lib/libgstbase-1.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libgstreamer-1.0.so -ldl /home/alex/cerbero/build/dist/android_armv7/lib/libgobject-2.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libffi.so /home/alex/cerbero/build/dist/android_armv7/lib/libgmodule-2.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libgthread-2.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libglib-2.0.so /home/alex/cerbero/build/dist/android_armv7/lib/libintl.so -lc /home/alex/cerbero/build/dist/android_armv7/lib/libiconv.so /home/alex/cerbero/build/dist/android_armv7/lib/liborc-0.4.so -llog -lm  -g -g -Os --sysroot=/home/alex/cerbero/build/android-ndk-r12b/platforms/android-9/arch-arm -fstack-protector -mthumb -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 --sysroot=/home/alex/cerbero/build/android-ndk-r12b/platforms/android-9/arch-arm -Wl,-no-undefined -Wl,-z -Wl,noexecstack -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,--gc-sections -Wl,-dynamic-linker -Wl,/system/bin/linker -Wl,-z -Wl,nocopyreloc -march=armv7-a -Wl,--fix-cortex-a8   -pthread -Wl,-soname -Wl,libopenwebrtc_jni.so -o .libs/libopenwebrtc_jni.so

owr_jni.c:11013: error: undefined reference to 'owr_video_renderer_set_request_context_callback' collect2: error: ld returned 1 exit status

[Zubnix]

No idea where that error is coming from. :confused: It looks like the linker is unable to find a symbol from the vanilla openwebrtc lib. Can you try with https://github.com/Zubnix/openwebrtc ? That's the original openwebrtc repo with the patches applied, unlike the udevbe ones, which is modified for a pure non-android Java build. Als please make sure you clean your build first.

[nakyup]

@Zubnix try https://github.com/Zubnix/openwebrtc same error

owr_jni.c:11014: error: undefined reference to 'owr_video_renderer_set_request_context_callback'

[nakyup]

Build successful!!

libopenwebrtc.exp add owr_video_renderer_set_request_context_callback

@Zubnix Thanks!!!

[BitterSymphony]

I am also ran into the same issue, I tried to add owr_video_renderer_set_request_context_callback into libopenwebrtc.exp, but still no luck. Isn't that the libopenwebrtc.exp file is generated by gen_jni.py? It seems that the .exp file will be overwritten every time after compilation.

Hope you could help me out. Thanks a lot

[nakyup]

@superdump merge plz! 11092b8

@BitterSymphony
try edit cebero/recipes/openwebrtc.recipe remotes = { 'origin': 'origin': 'https://github.com/EricssonResearch/openwebrtc.git' } to https://github.com/nakyup/openwebrtc.git

[BitterSymphony]

@nakyup Thank you for your help.

Updated cebero/recipes/openwebrtc.recipe from:

remotes = {
        'origin': 'https://github.com/Zubnix/openwebrtc'
    }

to

remotes = {
        'origin': 'https://github.com/nakyup/openwebrtc.git'
    }

Now I'm getting back to this exception: Exception: Please update your openwebrtc build for the host darwin; it's out of date

[BitterSymphony]

build success after creating a new cebero and cloning @nakyup 's repository. Thanks for the help.