Cannot get video to play

[anesuc]

Hi!

I've gotten the default videotestsrc to work however I tried playing an mp4 and have not gotten anything to show up. The pipeline description I used was:

"filesrc location=/home/... ! appsink name=sink"

Thanks

[RSATom]

videotestsrc produces raw (i.e uncompressed) video frames. But mp4 has compressed content, and also it's wrapped into container. So you have to deal with all that things.

I think you should get it working with something like this: filesrc location=/home/... ! decodebin ! appsink name=sink

[anesuc]

Thanks for the fast reply! Really appreciate it! Ok, tried that and got an instant crash with the error:

terminate called after throwing an instance of 'Napi::Error' what(): Cannot read property '1' of undefined

I'm on Nw.Js version 0.45.5 btw

[RSATom]

I didn't try to use it with NW.js. If you will provide me minimal app to reproduce this problem, I would try to fix it.

[anesuc]

Ok. Hopefully you are on Linux. I've uploaded a zip with modifications to your git. It's very rough just because I was changing what was necessary. https://drive.google.com/file/d/14IIUzbL_e0B0rletrqNtt5s4v367-Gpc/view?usp=sharing

Run ./nw and you should be ready to go

[RSATom]

Ok, I'll look at it a little bit later.

[RSATom]

@anesuc I've investigated your question a little bit, it looks like GStreamer inside application can't load elements to decode video, and gives your application only audio frames, that's why some properties related to video frames just not available. I'm trying to understand why it happens.

[anesuc]

Interesting. I was only testing an audio-less video anyway so that's interesting.

[RSATom]

@anesuc It looks like there are some conflict between avdec_h264 GStreamer element and NW.js internals. Since working in minimal test application works well, but gives error with NW.js. Tbh, I don't want spend much time on NW.js, so I would recommend you try Electron - it's possible there are no this issue there.

[RSATom]

file_example_MP4_480_1_5MG.mp4 You've provided has audio track, it contains silence, but it exists.

[anesuc]

Oh well that sucks. Nah, I can't switch to Electron since this is part of a massive project that relies on Nw.Js at this point haha

Oh yeah thats true about the audio track in the video

[RSATom]

The most possible reason Nw.js executable contains libav/ffmpeg inside, and some exports conflict with libav used in GStreamer.

[RSATom]

Maybe some workaround exists, but it require investigation and there are no 100% result...

[anesuc]

Yep that fixed it! Give me a sec to find the source, but it was a fix from another library I was also trying haha

[anesuc]

Ok the fix for anyone looking for it was actually also for electron. But works for Nw.Js. To workaround it you need to either replace libffmpeg.so with empty wrapper linked to libav*:

gcc -Wl,--no-as-needed -shared -lavformat -o /path/to/Nw.js-libffmpeg.so

Will fix the issue. I'm doing additional testing though, but it doesn't crash anymore

[anesuc]

@RSATom Yeah I still don't get any sound. How did you even get it to play without crashing before?

[RSATom]

to get sound you just need add something like autoaudiosink element to pipeline. I.e. video will go through NW.js, but audio GStreamer will output directly to system.

[RSATom]

If you need get raw audioframes in NW.js, you can add another appsink to pipeline and add another callback for it in JS.

[anesuc]

I tried adding autoaudiosink and it just does nothing now. I assume it's a context error thing? I just added "! autoaudiosink". How do you go about the raw audioframes code wise?

[RSATom]

I'm afraid it's not that simple... What source do you want to use? Local file, or some Url?

[anesuc]

Local

[RSATom]

I think you need something like this:

filesrc location=file_example_MP4_480_1_5MG.mp4 ! decodebin name=d  d. ! queue ! videoconvert ! appsink name=sink  d. ! queue ! audioconvert ! autoaudiosink

[anesuc]

Ok that works! Thanks! Really appreciate you taking your time to help with this!

[RSATom]

@anesuc

How do you go about the raw audioframes code wise?

If you need to get raw audioframes in NW.js - I can give you some hints. But only tomorrow. I have to go right now...

[anesuc]

Yes when you can please do!

[anesuc]

I want to add that although my solution works, it pretty much bricking playback on video tags etc is kind of a problem haha

[RSATom]

@anesuc to get raw audioframes alongside with video frame you will need something like this:

wcjs_gs.parseLaunch("filesrc location=file_example_MP4_480_1_5MG.mp4 ! decodebin name=d  d. ! queue ! videoconvert ! appsink name=sink  d. ! queue ! audioconvert ! appsink name=audiosink");
wcjs_gs.addAppSinkCallback("sink",
        function(type, frame) {
                if(type == wcjs_gs.AppSinkNewSample) {
                        renderContext.render(frame, frame.width, frame.height, frame.planes[1], frame.planes[2]);
                }
        } );
wcjs_gs.addAppSinkCallback("audiosink",
        function(type, frame) {
                if(type == wcjs_gs.AppSinkNewSample) {
                    console.log(type, frame);
                }
        } );

I didn't try run it myself, so some issues are possible, but overall concept should be correct.

[anesuc]

Hey sorry to bump this, It might be a new issue worthy, but is it possible to build this statically? THis would supposedly resolve my problem that I attempted to resolve using that temporary fix. There is another module that does something similar to this and mentions that linking the libraries statically during build will resolve the conflict issue. It's here: https://github.com/Kagami/mpv.js

That's where I initially got that temporary fix. However, I kind of need the normal video tags to still work. To quote it:

To workaround it you need to either replace libffmpeg.so with empty wrapper linked to libav: gcc -Wl,--no-as-needed -shared -lavformat -o /path/to/libffmpeg.so Or use libmpv with statically linked libav

(The last part being important)

I am just having issues getting it to compile though with the static libs hence why I am back here haha. I didn't create a new issue as I did mention the temporary fix here, but if you think it's worth a new issue I will gladly create one, depending if this is a solvable issue haha.

[RSATom]

@anesuc If you mean link GStreamer statically to plugin - I think yes, it's possible, but I never used GStreamer as static library, and it will require pretty big amount of time to investigate how to do it, and prepare repeatable build procedure.

[anesuc]

@RSATom Yeah that's what I meant and that's understandable. If you end up having some free time to try it let me know if something works out

[RSATom]

@anesuc sorry, don't think I will have time for this in the near future... All I can do - create issue and mark it help wanted...

[anesuc]

@RSATom Ok thats fine Saw that you already added a comment that adds more information in https://github.com/RSATom/wcjs-gs/issues/1 . I'll just comment there so that I get notified in case someone stumbles on it haha