search

filoe / cscore

An advanced audio library, written in C#. Provides tons of features. From playing/recording audio to decoding/encoding audio streams/files to processing audio data in realtime (e.g. applying custom effects during playback, create visualizations,...). The possibilities are nearly unlimited.
Other
2.19k stars 454 forks source link

Linux and OSX Support #94

Open opcon opened 8 years ago

opcon commented 8 years ago

Progress

filoe commented 8 years ago

Using ffmpeg as decoder on non-windows platforms, was on my todo list for months. I've just had a quick look at it. Unfortunately it is quite hard to implement seeking. Also, we would have to check the license of ffmpeg. I'm not sure whether it is compatible with the ms-pl or mit license.

I've added a few extra points to your progress list above.

opcon commented 8 years ago

FFmpeg is licensed under LGPL which is compatible with MS-PL and MIT. If we don't distribute FFmpeg with CSCore and rely on a system installation of ffmpeg, then calling PInvoke on the system installation of FFmepg is the same as dynamically linking to it, which means the FFmpeg license shouldn't have any effect.

I think CSCore would fall under this section of the LGPL license (found here):

A program that contains no derivative of any portion of the Library, but is designed to work with the Library by being compiled or linked with it, is called a "work that uses the Library". Such a work, in isolation, is not a derivative work of the Library, and therefore falls outside the scope of this License.

I'm not a lawyer though, so I could be wrong.

Thanks for the added progress points.

opcon commented 8 years ago

I have made some progress with decoding compressed audio on OSX using MacCore, a project that binds Apple's Objective-C API to C#. I just need to clean up the code and implement an IWaveSource decoder.

The resulting MacCore binary is quite big (~8 MB), so to avoid redistributing the binary on every platform I thought it might be a good idea to have a separate OSX dll: CSCore.OSX.dll?

This way all the OSX dependent code could be kept separate from the main CSCore dll.

filoe commented 8 years ago

Would you still need a reference on the MacCore assemblies? Or would you put all OSX relevant code into the CSCore.OSX project?

opcon commented 8 years ago

At the moment the OSX specific code references the MacCore assemblies, however it should be possible to just pull the relevant code from MacCore directly into CSCore, since it only uses a small section of the library.

If that's possible than maybe there's no point having a separate CSCore.OSX.dll, and we can have it in the main project directly?

filoe commented 8 years ago

Well, if we could include it directly it would be perfect. But we could start with using MacCore assemblies as seperate files. What would you say?

opcon commented 8 years ago

I think that's a good idea.

I should have the OSX code working soon, I'll put it in a CSCore.OSX project for now, and we can always combine them later.

opcon commented 8 years ago

I have implemented a CoreAudio IWaveSource which works perfectly with the OpenAL output!

Video here

Note the audio is a little choppy because I'm running OSX in a VM.

filoe commented 8 years ago

Wow thats awesome! I'm going to have a look at the linux build soon. For testing out OSX, I would have to set up a VM first(and learn how to deal with a OSX system ;)).

filoe commented 8 years ago

Just merged your openal pull request into the openal branch. I've fixed already a few bugs. I've also created a new linux branch, containing the CSCore.Linux project which contains links to the main CSCore project: http://fs5.directupload.net/images/160528/9brj96dh.png

opcon commented 8 years ago

I had a look at the linux branch, nice job! Thanks for catching those bugs in the OpenAL implementation

I also edited the checklist to reflect the OSX implementation status

opcon commented 8 years ago

Hi @filoe, just an update on the OSX implementation. I've written a post-build step that merges CSCore.OSX.dll with MonoMac.dll (the MacCore assembly) and removes unused code from MonoMac.dll, resulting in a self-contained CSCore.OSX.dll that is only ~40kb in size. I've been using this merged dll in my projects and have not run into any issues.

I feel like it's a good idea to keep CSCore.OSX as a separate project, similar to CSCore.FFmpeg, and it could have it's own nuget package.

What are your thoughts on merging my branch osx back into the main codebase? It contains some fixes to the OpenAL implementation to correctly flush buffers after stopping, as well as the OSX implementation.

I don't know if you'd prefer to merge your openal/linux branches into master first, or if I should merge/rebase directly into one of those?

filoe commented 8 years ago

Hello @opcon,

since I don't want to include linux and osx support in 1.2 release, I would prefer to not merge the extra branches to master. Is it possible to seperate OpenAL from linux/osx?

Regards

opcon commented 8 years ago

Yeah I can do that, the actual commits themselves are separate so I can just pick the OpenAL ones.

I'll cherry pick them on to your current OpenAL branch and then submit a new pull request?

filoe commented 8 years ago

That would be great. I would like to seperate OpenAL from linux/osx/... development. As soon as OpenAL is stable enough, I can merge it back to master.

filoe commented 8 years ago

In order to pass the unit tests and make it possible to play audio on multiple devices at once, I've did some major changes to the AL implementation. Unfortunately, it seems like the windows OpenAL implementation is really bad. I was not able to use multiple contexts in one appdomain yet. If I switch the context, it stops playing the old context. If did some tests with ubuntu and it seems like it works fine on ubuntu. But I don't really see a huge problem in it. If it works on mac, I'm happy with it. The user would have to use Wasapi on Windows and OpenAL on the other platforms. Hopefully microsoft fixes this ...

opcon commented 8 years ago

Thank you for your effort, that's a massive rewrite!! It looks good!

Yeah, Windows doesn't have the greatest OpenAL support. I agree that the main use for OpenAL is to support Mac and Linux, Windows users have other options.

I found a spelling mistake in ALSoundOut, CheckForDisposed() line 613:

if (_disposed)
    throw new ObjectDisposedException("WasapiOut")

WasapiOut -> ALOut

Additionally it appears that restarting a stopped source does not work unless I call Initialize() again. It seems like it's something to do with the buffer filling, since all I hear is the same few milliseconds of audio repeated over and over.

I couldn't find a fix in the quick time I looked at it, but it's not a drastic bug since calling Initialize() before restarting a stopped source is easy to do.

Thank you again for your rewrite!

filoe commented 8 years ago

Thanks for reporting that bug. Did not recognice that. Gonna have a look at it.

Am 28.07.2016 um 03:08 schrieb Patrick Yates notifications@github.com:

Thank you for your effort, that's a massive rewrite!! It looks good!

Yeah, Windows doesn't have the greatest OpenAL support. I agree that the main use for OpenAL is to support Mac and Linux, Windows users have other options.

I found a spelling mistake in ALSoundOut, CheckForDisposed() line 613:

if (_disposed) throw new ObjectDisposedException("WasapiOut") WasapiOut -> ALOut

Additionally it appears that restarting a stopped source does not work unless I call Initialize() again. It seems like it's something to do with the buffer filling, since all I hear is the same few milliseconds of audio repeated over and over.

I couldn't find a fix in the quick time I looked at it, but it's not a drastic bug since calling Initialize() before restarting a stopped source is easy to do.

Thank you again for your rewrite!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Mr-Crabman commented 7 years ago

Is the linux branch functional?, I'm finding this kinda ambiguous.

Bond-009 commented 7 years ago

115 Support for .Net core?

filoe commented 7 years ago

The linux branch is currently functional but not with all features (see http://fs5.directupload.net/images/160528/9brj96dh.png). Therefor it is currently not released.

Mr-Crabman commented 7 years ago

Thanks for the information, how do I download/use it though?, I'm new to Github.

EDIT: the linux branch I mean, I probably should have specified.

filoe commented 7 years ago

After some time, there is finally some kind of progress: The usage of ffmpeg shared libraries is ready to use. It was already tested on windows and linux(Ubuntu). Since I don't have any access to OSX systems I can't test it.

opcon commented 7 years ago

Wonderful news! I can test the ffmpeg output on OSX in the next couple of days and let you know how it goes.

Would you be happy to include the OSX output in the same release as the ffmpeg support? If so, I'll create a pull request with my OSX code.

In addition I changed the CodecFactory class to allow adding additional decoders to it at runtime, as well as coding some checks to prevent loading Windows APIs on Linux/OSX. I'll create a separate pull request for these.

filoe commented 7 years ago

Would be great if you could test it on OSX.

allquixotic commented 7 years ago

Hi, I'm on macOS Sierra, and I'm having difficulty using the filoe/cscore git master version of CSCore.Ffmpeg. I have it compiled successfully using Visual Studio for Mac beta, and I'm trying to use Mono 4.8 (part of .NET Core SDK from Microsoft) to run it.

The problem I'm having is related to DLL mapping.

I see the Interop classes try to load the following: avformat-57, avutil-55, avcodec-57, and swresample-2.

So I wrote a test.exe.config as follows:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
        <dllmap dll="avformat-57" target="libavformat.57.dylib"/>
        <dllmap dll="avcodec-57" target="libavcodec.57.dylib"/>
        <dllmap dll="avutil-55" target="libavutil.55.dylib"/>
        <dllmap dll="swresample-2" target="libswresample.2.dylib"/>
</configuration>

Unfortunately, even after setting the env var DYLD_FALLBACK_LIBRARY_PATH (equivalent of Linux LD_LIBRARY_PATH on macOS) to /usr/local/lib:., and having Homebrew FFMpeg libs (which are those same exact versions mentioned above, 57/55/2 accordingly) in /usr/local/lib and copied to the bin/Debug dir with the executable, it still doesn't work.

I get a DllNotFoundException on avformat-57.

Any tips?

EDIT: Right after posting this, I had the genius idea of renaming test.exe.config to CSCore.Ffmpeg.dylib.config. Now I'm getting somewhere:

[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeInitializationException: The type initializer for 'CSCore.Ffmpeg.FfmpegCalls' threw an exception. ---> System.DllNotFoundException: libavformat.57.dylib
  at (wrapper managed-to-native) CSCore.Ffmpeg.Interops.ffmpeg:av_register_all ()
  at CSCore.Ffmpeg.FfmpegCalls..cctor () [0x000a8] in <8897487fb95445f2be1733c8853021bf>:0
   --- End of inner exception stack trace ---

...At least it's actually trying to find the lib I specified in the .config file! Now to hack on it until it works.

allquixotic commented 7 years ago

My previous problem was that the .NET Core SDK ships a 32-bit version of Mono, and homebrew installs a 64-bit version of ffmpeg. Argh! So I used homebrew's 64-bit Mono instead.

Now a trivial 1-line test program hangs indefinitely.

new FfmpegDecoder(new MemoryStream()) hangs (without error, even with all debug/verbose stuff opened up) in the following call stack:

FfmpegDecoder(Stream) constructor -> new AvFormatContext(FfmpegStream) constructor -> FfmpegCalls.AvformatOpenInput(pformatContext, stream.AvioContext) ->

this line in FfmpegCalls.cs::AvformatOpenInput() hangs:

int result = ffmpeg.avformat_open_input(formatContext, "DUMMY-FILENAME", null, null);

"DUMMY-FILENAME" doesn't sound good; that looks like the code isn't complete. The FFmpeg libavformat docs don't say anything about "if you pass DUMMY-FILENAME, blah blah will happen".

This is on macOS with 64-bit ffmpeg 3.2.2 and 64-bit mono 4.6.2 and git master CSCore.

filoe commented 7 years ago

The ffmpeg component was never tested on mac. I currently don't even have mac build of ffmpeg. I'm planing to do that till the next cscore 1.3. DUMMY-FILENAME is correct because in that cause the avioContext contains the callbacks, setup by the FfmpegStream class. It is definitly finished. The only problem you're currently facing is probably the missing build of ffmpeg for Mac.

filoe commented 7 years ago

"Support for channel-conversion based on a channel matrix" is done. Is there any progress on the MacOSX implemenation @opcon?

Mr-Crabman commented 7 years ago

How did you get the visualization sample working?, trying to play a wav file just makes the program shut down.

Compiling has no errors of any kind either.

EDIT: The xterm window displays a message when the shutdown happens, but for some reason I can't copy it, it does mention System.Windows.Forms a lot though.

opcon commented 7 years ago

@filoe The MacOS implementation is complete (for decoding audio files at least). I need to go through and clean up/produce an OSX branch suitable for merging, I still haven't had time to do that unfortunately. Hopefully I'll get to it in a week or so.

What's the status on merging your OpenAL branch into master?

Also I have made a few changes to the way CodecFactory works in my branch which makes supported codec detection for the current platform easier, I'll create a separate pull request for those and we can discuss those changes separately.

Mr-Crabman commented 7 years ago

Does anyone know how to get the winformsvisualization sample working on Linux?, I'm getting System.DllNotFoundException: libopenal.so.1, and I'm completely confused, as I'm sure I built it properly.

filoe commented 7 years ago

Does your system has openal installed. Which system are you using?

Am 25.01.2017 um 17:03 schrieb VortexGator notifications@github.com:

Does anyone know how to get the winformsvisualization sample working on Linux?, I'm getting System.DllNotFoundException: libopenal.so.1, and I'm completely confused.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Mr-Crabman commented 7 years ago

Openal doesn't come preinstalled as part of the OS?, well that explains it.

I'm getting a different error now, yay!.

shm_open() failed: No such file or directory
AL lib: (WW) alc_initconfig: Failed to initialize backend "pulse"
shm_open() failed: No such file or directory
ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Protocol error
CSCore.SoundOut.AL.ALException: Could not open device "ALSA Default".

Not sure what you mean by system?.

filoe commented 7 years ago

I am sorry. I ve meant, which operating system are you using?

Am 25.01.2017 um 20:54 schrieb VortexGator notifications@github.com:

Openal doesn't come preinstalled as part of the OS?, well that explains it.

I'm getting a different error now, yay!.

shm_open() failed: No such file or directory AL lib: (WW) alc_initconfig: Failed to initialize backend "pulse" shm_open() failed: No such file or directory ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Protocol error Not sure what you mean by system?.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Mr-Crabman commented 7 years ago

Xubuntu.

Mr-Crabman commented 7 years ago

Strange, when I restart the computer, it works temporarily, but later (few minutes) gives the error:

shm_open() failed: No such file or directory
AL lib: (WW) alc_initconfig: Failed to initialize backend "pulse"
shm_open() failed: No such file or directory
ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Protocol error
CSCore.SoundOut.AL.ALException: Could not open device "ALSA Default".

After some time.

filoe commented 7 years ago

Is there some kind of stacktrace?

Am 01.02.2017 um 15:14 schrieb VortexGator notifications@github.com:

Strange, when I restart the computer, it works temporarily, but later (few minutes) gives the error:

shm_open() failed: No such file or directory AL lib: (WW) alc_initconfig: Failed to initialize backend "pulse" shm_open() failed: No such file or directory ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Protocol error CSCore.SoundOut.AL.ALException: Could not open device "ALSA Default". After some arbitrary amount of time.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Mr-Crabman commented 7 years ago

@filoe

Is there some kind of stacktrace?

Not sure what you mean?, like this?:

Thread started:  #2
Thread finished:  #2
Thread started:  #3
Thread finished:  #3
shm_open() failed: No such file or directory
AL lib: (WW) alc_initconfig: Failed to initialize backend "pulse"
shm_open() failed: No such file or directory
ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Protocol error

AL lib: (EE) ALCplaybackAlsa_open: Could not open playback device 'default': Connection refused
CSCore.SoundOut.AL.ALException: Could not open device "ALSA Default".
  at CSCore.SoundOut.AL.ALDevice.get_DeviceHandle () [0x00000] in cscore-linux/CSCore/SoundOut/AL/ALDevice.cs:31 
  at CSCore.SoundOut.AL.ALContext..ctor (CSCore.SoundOut.AL.ALDevice device) [0x00000] in cscore-linux/CSCore/SoundOut/AL/ALContext.cs:69 
  at CSCore.SoundOut.ALSoundOut.Initialize (CSCore.IWaveSource source) [0x00000] in cscore-linux/CSCore/SoundOut/ALSoundOut.cs:327 
  at WinformsVisualization.Form1.openToolStripMenuItem_Click (System.Object sender, System.EventArgs e) [0x00000] in cscore-linux/Samples/WinformsVisualization/Form1.cs:70
filoe commented 7 years ago

I've created an issue. But it will be quite hard to reproduce and find the error. See #210.

ststeiger commented 4 years ago

Why don't you just directly use the ALSA API (libasound) to play a sound on Linux? There's a C# wrapper (don't know how complete): https://github.com/atsushieno/alsa-sharp

Example playback program (in plain C): https://gist.github.com/ghedo/963382/815c98d1ba0eda1b486eb9d80d9a91a81d995283

You could also use PulseAudio, but PulseAudio is just a sorry-construct ontop of ALSA, and not necessarely present on small distros, while ALSA is in the kernel.

For iOS/Android, take a look at XamarinAudioManager

OSX you seem to already have covered with CoreAudio, more or less.

allquixotic commented 4 years ago

Why don't you just directly use the ALSA API (libasound) to play a sound on Linux? There's a C# wrapper (don't know how complete): https://github.com/atsushieno/alsa-sharp

Example playback program (in plain C): https://gist.github.com/ghedo/963382/815c98d1ba0eda1b486eb9d80d9a91a81d995283

You could also use PulseAudio, but PulseAudio is just a sorry-construct ontop of ALSA, and not necessarely present on small distros, while ALSA is in the kernel.

For iOS/Android, take a look at XamarinAudioManager

OSX you seem to already have covered with CoreAudio, more or less.

The point of an audio API like cscore is to abstract away platform differences, so the developer can release software for multiple platforms with only one audio playback implementation in their app. Furthermore, this can be an issue if, like you said, there’s more than one valid audio API for a given platform and you want to support each. Only supporting ALSA on Linux is a really bad idea because most desktop systems do use PulseAudio, and the shim that lets you play audio back to PulseAudio using the ALSA API is very inefficient, error-prone, and restricts you to a subset of the ALSA API. Accessing PulseAudio directly is so much better.

There’s even more to it: on more than one occasion a platform owner has introduced a new audio API and officially deprecated an old one. If an application directly codes against a platform sound API, well, that’s a rewrite of their sound handling code to support the new API. If they use an abstraction later like cscore, the new API can be added as a backend to cscore and your app can bring in that support with minimal or no new code.

Your comment sounds like an attempt to evade this real problem, which is strange since I would normally assume that someone contributing ideas to a sound abstraction layer would at least advance arguments acknowledging the reason for that abstraction layer to exist. Indeed, if every app developer just coded against ALSA directly, we could just delete this project from GitHub and move on.

ststeiger commented 4 years ago

@allquixotic: Well, I think you misunderstood something there. I didn't mean he should code using ALSA instead of CSCore. I meant CSCore should use ALSA as backend instead of OpenAL. That way, you don't need an additional native library (which may not be present on some systems). Also, that way you don't have to depend (=wait for) on OpenAL for platform support.

[Alsa-Architecture[1]

filoe commented 4 years ago

There is no problem with supporting ALSA directly as playback interface. Unfortunatly I currently have very limited resources on this project. I will try to maintain bugs, ... but in order to support such big new features, I would have to rely on the help or on contributions of others. So if there is anyone willing to contribute or even maintain the library, that would be great in order to bring cscore stable to linux and mac. There is already the netstandard branch which acts as a base for that.