vlc video audio

[passenger94]

made a "video" branch on my repository

working for me using system libvlc 2.2.0 should work on Windows too according to vlc doc with a method change from libvlc_media_player_set_xwindow to libvlc_media_player_set_hwnd, there's also a cocoa method : libvlc_media_player_set_nsobject ...

tested with simple-video.rb with a remote and a local file

[ccoupe]

VLC has distribution problems for packaged Shoes - which libs to include in the package and where to put them on the users system. Very difficult issue when it gets to codecs which may not be FOSS. That's the main reason it was ripped out of Shoes long ago.

[passenger94]

I was suspecting something like that ... reason why i used installed vlc, could be set as an option though, like it was, except it could possibly work :-) let the user decide if it can/could deal with the distribution problems. Of course it may not worth the maintenance work, though it was rather easy to rebuild the feature, i was missing the ability to have an easy solution for sounds inside shoes (i know there is gems for that but vlc is quite an amazing tool, there's also mpv but i found it to be more difficult to install headers)

[ccoupe]

For audio, the portaudio gem/extension source is still in Shoes - it's just not compiled and the libs aren't copied into the Shoes place. PITA if you get the wrong version of portaudio installed with apt-get. VLC video also had threading problems with 1.9.1 (many things did) and given the distribution issues (and window issues at the time) no one wanted to rewrite the Shoes C code to deal with the new VLC api. I

[passenger94]

just to let you know it works nice here, made a video_gtk3 branch which is current with master (main point of this message) so if you want to foresee merging issues about gtk3 branch into master ... (result + history) cheers

[ccoupe]

Considering that the work that is going into #54 is based on the old Video code we should probably probably put the code back into Shoes.

[ccoupe]

I don't know how practical the following idea is. I don't want to build/maintain Shoes dependency for libVLC (and all it's components) and I certainly don't want to bloat the Shoes download size by including it. However, we could include a fiddle/ffi based Ruby interface to the libVLC api and try to load the so/dylib/dll from known locations ( in cache.rb or shoes.rb ) if It loads and the version numbers are good enough then enable creation of Shoes Video Viideo widgets for Shoes scripts

VLC still has an external display window option. Obviously getting that done in Shoes is a bit of a challenge after all these years of neglect. Less clear is how to handle Control buttons like start/stop/pause/ff/rew and associated callback and threading issues. However we all use pthreads now so that might not be a blocker.

The load locations of VLC are knowable for all platforms (according to VLC). The real trick is creating the fiddle/ffi interface - once but usable for all platforms. Furthermore, we would have to use ruby calls (rb_eval etc ) to the fiddle api so we have no static inker dependencies.

[ccoupe]

@passenger94 - would you create a video branch on the gitthub site and push to it so I can look at what you've done already.

[passenger94]

done, pending pull request on video_vlc branch (still working by my side)

if i remember correctly, the only change for Osx and Windows might be lines 351; 352 https://github.com/Shoes3/shoes3/pull/189/files#diff-153758cf9bd8a68fbd3d25b2079b22aaR351 That could be an option to use user installed vlc, in Loose Shoes environment (no library management troubles, it used to be that way, before)

[ccoupe]

@passenger94. Thank you. I like that most of the VLC code is isolated into shoes/video/ instead of cluttering up ruby.c Looking at your code has also helped me to think about my proposed fiddle/ffi method of dealing with Video. In that magical universe , the shoes/video/* vlc code would be replaced with C that calls Ruby (using rb_eval) and the ruby code would use a dynamic ffi/fiddle built bindings to talk to the vlc library - assuming the user has installed vlc on their system. Some methods that deal with general Shoes things (top, left, move) would still be C. Any of your methods (for example) that call into vlc_apis would have to cross the Shoes.C- > Ruby/ffi -> vlc C barriers.

The goal is to not to use C to compile vlc calls and thereby create a link dependency for vlc libs to be includes with Shoes. Side effect - you wouldn't need VLC installed to build a distributable Shoes that could handle VLC if installed on the users system. It's a lot to think about. I have no idea if Gtk3/Windows shoes_native_surface() returns something acceptable to vlc.

[passenger94]

:-D Now, That's what i'm calling Fiddling ! MAN !!! this is heavy stuff !!! Lots of fun :-D arid documentation + almost bare bones programming !!! ouch ! and i' have only discovered the beginning of fiddle -> libvlc side, that is a pledge to be quite a funky experience when trying to render on a cairo surface !

[ccoupe]

Its very bare bones, learn as you go. Thankfully you have working code so it could be done one vlc call at a time until the last thing to do is delete #include "vlc.h" Getting the structs properly configured is important and make sure nothing is lost in conversion from C into Ruby. There are a couple of Projects the have done ffi&vlc in Node.js and Scheme. I wouldn't try Cairo drawing/manipulation/transforms this way but we need those do we for video widgets? A lot of work.

Might be a way to handle other strange native widgets in the future, if there are any.

[passenger94]

ok, Thanks to the github goldmine, having a vlc instance playing a video/audio in an external window is quite straightforward, that's one thing, but interacting with structures and unpacking what's pointers are pointing to ........ there are some limitations apparently too, for example a union inside structure is not allowed in fiddle, we have to use a workaround for this.... create a place-holder and then squirm around fiddle pointers and sisters... lots to learn :-D

[passenger94]

something is working :-) fiddle + libvlc on the ruby side of Shoes nothing in shoes C specific to vlc, generic api, but uses Shoes placements routines (here with margins) Sound and Vision !

2 lines of code in ruby, specific to linux / windows / osx (using libvlc that is)

[ccoupe]

Amazing! Wonderful!

[ccoupe]

I have some head scratching ahead for Windows, It kind of works, I get the audio track for an mp4 but error messages on the dos commandline and no video,

E:\shoes3\samples>cshoes.exe videoFiddleVlc01.rb
[03da64c4] core video output error: video output creation failed
[03d84d3c] core decoder error: failed to create video output
[03da64c4] core video output error: video output creation failed
[03d84d3c] core decoder error: failed to create video output

VLC (standalone app) does show the video, so it's definitely s Shoes/Window problem. Probably the gdk call. to get a Handle.

[passenger94]

the libvlc hook to the drawing area is not yet totally finished (still missing some proper thickness to the widget to act like other widgets) but should work ... head scratching ... any chance to capture an actually helpful message ? There was some complain about h264 (mp4) rendering ...

[ccoupe]

Yes, I've set the hook for Windows. Different error message but we have video. It's got new oddities but I'm convinced the fiddle concept you implemented will work. Even with the Windows issues, this will get into Shoes 3.3.1, as proof of concept, samples/very-expert-video.rb. or a widget. I need to get the OSX code working at least as good as Window. Tomorrow, I hope.

[passenger94]

what do i need to get quickly set, on windows 7, to be able to mess around with c code ? if such a request has a meaning on windows ...

[ccoupe]

You will need RubyInstaller and the matching devkit. That should get you the MingW msys environment which is a kind of like the unix shell. It's important to get the 32 bit versions of both. So Ruby 2.2.3 is their latest and the 32 bit devkit for it.

http://rubyinstaller.org/downloads/

You'll probably want 7zip utilitiy so you can untar things from Shoe website. Plus some patience. The terminal is pretty lame re: cut/paste/select.

On 01/30/2016 01:07 AM, passenger94 wrote:

what do i need to get quickly set, on windows 7, to be able to mess around with c code ?

— Reply to this email directly or view it on GitHub https://github.com/Shoes3/shoes3/issues/133#issuecomment-177097650.

[IanTrudel]

Is this part of 3.3.1 release? Some issues might need tagging.

[ccoupe]

Actually it's not tagged because it's not completely certain it's going to work or when it's going to work well enough. I think it's pretty close but I'm an optimist about schedules and features. We all know feature schedules are moveable.

[ccoupe]

The OSX code is going to take awhile. The new shoes_native_surface-* functions are a lot different from what cocoa.m has. I think the newer code can be done in Cocoa and makes more sense once I get back my head into Objective-C mode. Plus I've got other bugs to fix on OSX.

[passenger94]

@ccoupe , Is Windows part working now ? Looking at cocoa, makes me wondering if video was ever an option on Shoes osx ? The ancient code doesn't make much sense, it's missing a proper canvas to draw on, the equivalent of a GtkDrawingArea , at the moment "shoes_native_surface_new" returns the entire app canvas ... Apart from that, it should be relatively straightforward (from my totally objective-C blind perspective, that is)

[ccoupe]

It should be just a case of creating an NSView with a NRect from the width and height and enablling/disabling certain event callbacks . I agree that VLC may have never worked on OSX.

The Windows code is still misbehaving. I got tired of working on Windows and decided to give the OSX codebase some attention.

[passenger94]

i'm concerned with this : https://github.com/passenger94/shoes3/blob/video_fiddle_vlc/lib/shoes/videoffi.rb#L256

it works nice and we HAVE to wait for parent canvas to be drawn in order to accesss the drawable for vlc But it might conflict with a user supplied start event in a shoes script (happened to me) The only other solution, i found, is something like that but it's ugly... any idea ?

 Thread.new do
        sleep 0.5
        drID = @video.drawable  # xlib window / HWND / NSView  id
        if libvlc_media_player_get_xwindow(@player) == 0
            libvlc_media_player_set_xwindow(@player, drID)
                # libvlc_media_player_set_hwnd       on Windows
                # libvlc_media_player_set_nsobject   on osx
         end

        play if @loaded && @autoplay
   end

Also waiting in a loop for app.started? (in a Thread) but it's not enough when app is started, asynchronous drawing is not yet finished ...

EDIT: we might add an internal (private, not documented, reserved for Shoes) canvas.started? function, emitted at the same time as start event ...

[ccoupe]

@passenger94 - sorry I didn't get back to you sooner. I'm OK with adding undocumented/private methods. We've got draw methods that should never be called from general purpose scripts.

[passenger94]

No problem :)

On windows rake is complaining about not finding " ../mingw/lib/libmsvcrt-ruby220.so" when trying to compile Shoes, i thought it was looking for a dll - i do have a "msvcrt-ruby220.dll" in Ruby bin folder and a "libmsvcrt-ruby220.dll.a" in Ruby lib folder - ?! (i can compile gem extensions, so i guess installation is fine)

[ccoupe]

Sounds like your Ruby install is OK. Shoes is more complicated. See https://github.com/Shoes3/shoes3/wiki/Building-Shoes-on-Windows

1) you need the precompiled dependencies, (http://shoes.mvmanila.com/public/shoesdeps/mingw/ShoesDeps.zip - 96MB) 2) you need a win7-custom.yaml - see https://github.com/Shoes3/shoes3/wiki/Custom.yaml. with your settings for your system. 3) Patience - it does work.

[passenger94]

yeah i followed that route, after remembering you took a lot of time to write Wiki pages about it ... !! i'm sure it does work but the whole thing is kind of frightening to say the least ...

There is only a "libwinpthread.dll.a" inside ShoesDeps.zip, is it ok ? i was expecting a lot more (directories, dlls ...) it seems like the win7-custom.yaml is called only if cross compiling, no ? i'm lost there btw i'm trying to do a loose build, hoping it to be more easy, but i might be wrong probably because then i need all the dependencies installed , right ?

[ccoupe]

There is no loose build for Windows. They don't have sym links. So you have to do rake win32:setup:win7 . Then you can rake, rake clean like normal. win7-custom.yaml protects any changes you make to the rakefile from being blown away by a git refresh. You want one.

There should be lots of files in ShoesDep.zip, bin,lib include directories with things that make sense for unix folk. It's always possible the file is bad.

[ccoupe]

Yes, the ShoesDep file is bad. I'll replace it.

[ccoupe]

The file is mingw.tar.gz . You'll need to use 7-Zip to unpacking. Do NOT move the files into mingw/ruby directories. Whatever directory you unpack to belongs in the win7-custom.yaml

Deps: E:/shoesdeps/mingw
Ruby: C:/Ruby22
Gemloc: E:/gems/shoes
Extloc: E:/gems/shoes
Exts:
  - ftsearch
  - chipmunk
Gems:
  - hpricot
  - sqlite3
InclGems:
  - sqlite3
  - nokogiri-1.6.7.1-x86-mingw32
  - mini_portile2-2.0.0
  - byebug-8.2.1
  - rb-readline-0.5.3
  - win32-shortcut-0.2.5
MS-Theme: true 
GtkVersion: 3

That's my win7-custom.yaml

[passenger94]

Thanks ! :-))

[ccoupe]

You'll also want to download the prebuilt gems unless you like to suffer.

[passenger94]

hmmmmmmm ..... i think i might take the download route :-D , which is named ?

[ccoupe]

http://shoes.mvmanila.com/public/shoesdeps/prebuilt-gems/win7-gems.tar.gz

If you use a web browser you can navigate to all the files available for download. ;^)

[passenger94]

oh yes a web browser !!! ;-D sorry

[passenger94]

ok things are moving forward, But ! i have some "Package 'libpng' has no Name: field" which stops compilation, does it tell something to you ? $ rake build Package 'libpng' has no Name: field ... ./shoes/app.h:13:19: fatal error: cairo.h: No such file or directory happens also if i do #{PKG_CONFIG} --cflags cairo.chomp (in irb) in compile command, i do have some includes and libs like pango for example .... but no cairo ...

also the paths are hard-coded to your system in the .pc files, is it a problem ?

[ccoupe]

As I remember, @backOrder has used those deps (and .pc) files recently for 3.3.0 and I use them for both cross compiled (xwin7) and win7 builds which are pretty different. There are several versions of pkg-config for windows, however. I build from the dos console, Not from msys (and I think pkg-config is one of the reasons).

[ccoupe]

I just built win7 (master) myself. Send me email ccoupe@cableon.net if you want to work in a env where I send attachments.

[passenger94]

getting closer : cairo.pc wants libpng16 not libpng on my system ... now it almost compile, i'm having a video flag on the way, i think i need to git checkout to the right branch ...

[ccoupe]

Yes, it has not been updated for video (xwin7/env.rb) the changes for that branch. libpng.pc and libpng16.pc are duplicates.

I've fixed my win7/env.rb now. When I push it to github, and you pull, you are going to be happy you have an win7-custom.yaml

[passenger94]

ok i'm there ! still not compiling though, because i need to figure out the video windows part now ... looking in your branch where you have it i think

[passenger94]

that strange error will wait for tonight ! ./shoes/config.h:83:3: error: unknown type name 'HINSTANCE'

[ccoupe]

That error suggests there are missing Windows .h files which should be in the devkit/msys.

That gives me a few hours to deal with merge some branches. To really do some work on video for OSX, I want some of the changes I've made in master. It does compile, and load the libvlc.dylib (which complains about no plugins) and then dies early in your fiddle setup.

I'm going to try merging master into video_vlc branch. Expect a big commit.

[passenger94]

Compiles fine now, Thanks ! Hell, you sure reach level 1000000000000000+ in skills, tenacity and patience departments for managing all this madness !!!!!

EDIT; works ! the method libvlc_media_player_get_hwnd returns a void pointer not a int like the equivvalent in linux so if you get rid of the if guard and set directly the hwd : libvlc_media_player_set_hwnd(@player, drID) it works (should be the same on osx)

on another front i have problems with the new way to wait for drawable to be ready (in a Thread with started attribute) apparently we're stuck in an infinite loop there..., works on linux ...

[passenger94]

ok, it works for me, PR to test

i used animate method in lieu of Thread.new approach, Windows wasn't happy with it, i think because of threading issue between gtk main loop and that one ... Timers are using g_timeout_add some dedicated routine for threads in gtk (i hope it's the same in osx)

[ccoupe]

Works for my Windows too! Congratulations!

The rake files were a mess before I started 'fixing' them and they are still a mess. One day when there is nothing to add to Shoes, we can clean them up. You do not want to look at what happens for cross compiling gems and exts.

[ccoupe]

OSX fun: Had to resort to inserting raises into videoffi.rb:260 . in initialize I did

        @vlci = libvlc_new(0, nil)
        @version = libvlc_get_version
    raise "vlc version#{@version} #{@vlci.inspect}"
        @player = libvlc_media_player_new(@vlci)

and the shoes console shows

Error in <unknown> line 0 | 2016-02-07 23:17:42 -0700
vlc version2.2.1 Terry Pratchett (Weatherwax) #<Fiddle::Pointer:0x007fb780d13590 ptr=0x00000000000000 size=0 free=0x00000000000000>
/Users/ccoupe/build/xmavericks/Shoes.app/Contents/MacOS/lib/shoes/videoffi.rb:260:in `initialize'

If i remove the 'raise' then the next line, libvlc_media_player-new crashes Shoes (osx) something awful. Linux returns

Error in <unknown> line 0 | 2016-02-07 23:57:39 -0700
vlc version2.1.6 Rincewind #<Fiddle::Pointer:0x000000029b2b50 ptr=0x0000000299e990 size=0 free=0x00000000000000>
/home/ccoupe/Projects/shoes3/dist/lib/shoes/videoffi.rb:260:in `initialize'

Windows (working) is also 2.2.1 so that can't be the problem. Suspicious people might wonder if there is something wrong with the OSX vlc code, since fiddle.ptr is null for osx.

[passenger94]

Nice ! (for windows) , ouchh ! i can barely imagine the cross compile hardcore poetry !

googling leads me to put some faith into believing that it possibly looks like osx might eventually need arguments to livlc_new, maybe try with this one : https://github.com/strukturag/VLCKit/blob/master/Sources/VLCLibrary.m#L126 or maybe ENV variable for plugins path .....

Also, you can check for null pointer with null? method btw Fiddle is a tremendous library, nice finding !