Handling of external libraries

[cochrane]

We need to converge with this discussion somewhere because right now, it is all over the place. So here are the possibilities for handling the dependencies our code has that I've been able to think of:

• Store the code directly in the repository. Pro: Very easy; no setup required; inherently works on all platforms. Can make own modifications. Cons: Can make own modifications that aren't visible to the outside world; update difficult.
• Store the binaries directly. Pro: Update becomes easy. Con: Need at least three versions of everything (Mac, Linux, Windows), possibly more depending on 32/64 bit. Any new platform (does FreeBSD have OpenGL?) increases the pain.
• Hope the user has installed the libraries. Pro: Nothing to do on our part except edit the Readme.md. Con: No idea who uses what version of what library.
• Use external Git references to the libraries (to known tags). Pro: Using official version; no modifications make it back. Same ease of use as storing in repository. Con: Git external references are really hard. Especially if some library we want doesn't have a git repository; there's some Git/SVN bridge but I've never tried it.

A few more notes:

• SDL is funny because the Mac version is a different format (Framework) than any other library. That's my duty to figure out, of course, for everyone else that's not an issue.
• How big a problem is incompatibility really? Probably not that big; incompatible OpenGL drivers hurt way more. But of course it takes only one issue to make the whole thing less fun.
• We definitely need some way to deal with our own version of OpenAL. It does things the normal version doesn't, and I don't want to throw that away without very good reason. We could put it in a separate repository here and then use it like any other third-party library, of course.
• My personal vote would be for Git external references, I just have very little experience with actually using them.

[Lwmte]

Cochrane, is it possible to install/compile/link OpenAL Soft (http://kcat.strangesoft.net/openal.html) for Mac along with stock OpenAL? OpenTomb-bundled version wasn't very different from default OpenAL Soft, except it used custom-written SDL backend (which I will really miss - but I wrote a letter to Chris Robinson asking if he can implement SDL backend officially, or at least XAudio2 output for Windows).

I have more concerns regarding recently introduced sndfile library. I won't babble about how easy and simple it was without it and with only using native SDL functions to read and decompress audio samples, but what is the whole point in using sndfile along with OGG/Vorbis, if sndfile supports OGG/Vorbis itself?! Basically, we now have FOUR sound library dependencies (SDL, OpenAL Soft, OGG/Vorbis and now sndfile)! I'm sorry, but OpenTomb isn't a sound editing or video editing software, we need it light and simple - so either get rid of OGG/Vorbis or sndfile.

By the way, if sndfile can provide EASY interface for streaming MS-ADPCM wave files (TR3-TR5), then it should absolutely stay in place. I just had no idea how to program MS-ADPCM streaming when I wrote audio engine.

[Lwmte]

P.S.: Sorry, but maybe I got something wrong... Maybe sndfile requires OGG/Vorbis lib to read .ogg files? If that's the case, then self-written OGG streaming code should go away, and sndfile streaming should be implemented instead.

[stohrendorf]

MS-ADPCM is possible (see http://www.mega-nerd.com/libsndfile/); and from what I can tell from the sources (https://github.com/erikd/libsndfile/tree/master/src), it requires external libogg and libvorbis, so we still need to link against it.

[Lwmte]

Ok then, I thought libsndfile supports OGG/Vorbis itself! Maybe using libsndfile for streaming will solve current loading lag issue with soundtracks.

Generally - I'm all for removing third-party libs from source code, and using third approach offered by Cochrane - just put an info about version requirements into Readme, and hope that user will install libraries for himself. Because it's already happening anyway (I needed to install SDL and zlib anyway, and in case of zlib, I had funny time finding pre-compiled libs on the internet, as I'm too lazy and dumb to compile them myself).

Not sure if this approach will break nightly builds, we need to ask vobject about that... So far it worked with external SDL and zlib.

[stohrendorf]

Maybe we should set up a wiki page with dev requirements, like limiting boost to only use headers, external library handling and versioning, coding style, whatever.

[cochrane]

The wiki page is a good idea to document whatever we come up with here, once we have something.

@Lwmte: Installing a custom version of OpenAL is no problem on OS X, and I think actually what most games do. But again, throwing out our existing patched version needs strong justification that I just don't see.

[cochrane]

I'm not fond of having everyone install their own libraries. Everything that makes it more difficult for new developers to join is a problem, not an advantage.

[vvs-]

Actually, I see two problems here:

• a development environment
• a deployment environment

They can have different setups. Actually, there is a special MinGW case already for cross-compilation.

[stohrendorf]

To summarize: my last merge request eliminated the need for SDL Audio, so the only depencies are now OpenAL and libsndfile (which in turn requires ogg/vorbis). Both libraries are available on all target platforms. The correct way of using XAudio in windows is to abstract the audio engine in OpenTomb so that on windows it uses XAudio directly, while using OpenAL on all other platforms. Patching XAudio into OpenAL is surely the wrong way, as they both are abstraction layers with the same underlaying layer. And well, if developers cannot even install a needed lib, then they are not worth the title "developer".

Sorry, but I still haven't seen any reasonable argument that let me even consider to leave the code in.

[vvs-]

And well, if developers cannot even install a needed lib, then they are not worth the title "developer".

That's a very strong statement that I'd like to avoid. We don't need to alienate people without system administration experience. I know many professional Java developers working on Windows who don't even know what CMD.EXE is. Don't laugh - it's true.

[stohrendorf]

I have my very strong opinion about Java "developers", and I'm not laughing, as I know that's true.

But we're talking about C/C++ developers here. If you want to attract people that do not know how to use/build/install external libs, this project is going to be doomed, as such people usually write code that's ... well, weird and not very error-prone.

Please don't think I'm only ranting. But the need for including the libs in-source comes from reasons that will lead to great problems in the long term.

[cochrane]

I don't mean to be rude here, but the first and only time we've had absolutely massive problems with libraries was when someone decided to break all existing processes for dealing with them because it looked cleaner. I like insulting people as much as anybody, but in the end it has rarely, if ever, solved problems.

If you think you have a solution that works better than what we had, great, show it to me. So far I haven't seen it.

[vvs-]

To be fair, nobody openly opposed that move. So, I think it's a scapegoating to blame the author of the pull request.

[stohrendorf]

It's pretty simple, but requires some work: Implement two audio engines, one for OpenAL, one for XAudio. This needs an abstraction/indirection layer in OpenTomb, but it's the only clean way, because patching XAudio support into OpenAL is like patching DirectX support into OpenGL.

[cochrane]

To be fair, nobody openly opposed that move. So, I think it's a scapegoating to blame the author of the pull request.

You're right. @stohrendorf : Sorry.

It's pretty simple, but requires some work: Implement two audio engines, one for OpenAL, one for XAudio. This needs an abstraction/indirection layer in OpenTomb, but it's the only clean way, because patching XAudio support into OpenAL is like patching DirectX support into OpenGL.

I don't really understand your objection here. Yeah, it is a bit weird, but it is also code that is already tested and works well. You propose to replace that well-tested code and replace it with a huge new thing that needs to be written and tested again and which will expose an interface that nobody is familiar with, for no actual benefit other than that a description of it sounds nicer.

I understand the desire for clean solutions, but maintenance cost is a real thing, too. Now, if you want to write that translation layer, be my guest. But I just don't see the benefit.

[stohrendorf]

Then there's FMOD Ex; it isn't open source, but a fairly advanced sound engine for win/mac/linux. A colleague who worked for a game developer told me about that.

[vvs-]

I'd second the maintenance concerns.

Then there's FMOD

Wouldn't that be incompatible with OpenTomb license?

[vvs-]

To add to my previous post. There are such big projects as OpenStack and Xen which use their own development environment (DevStack and Raisin respectively). They have just the same concerns about consistency of their development environment for the developers and easy maintenance. Though, it has a big cost of designing and constructing such reference environment in the first place.

But having all reference libraries as an optional git submodules used by the developers can ease the pain considerably.

[vvs-]

I think it requires some elaboration.

The idea is to move all external libraries into submodules to make it optional. Those who require special library versions should explicitly checkout the submodules. That means cross-compiling or some platform specific versions which should be hosted in the opentomb organization. Others shouldn't do anything special and use system libraries by default.

We can even have different branches for that, i.e. development or stable which could differentiate between debug and release specific libraries.

[ablepharus]

Why not use the DirectSound backend of openal-soft on Windows and leave XAudio alone for now? If you want to have XAudio, work on openal-soft.

[Lwmte]

In fact, I got reply from Chris Robinson (who works on OpenAL Soft), and he told me that there's no benefit in XAudio2, when compared to mmdevapi (actually, people say that XAudio2 is an abstraction layer over mmdevapi), and OpenAL Soft already uses mmdevapi as a backend for Windows systems since Vista. So it's no more actual.

[stohrendorf]

And the mac-issues with missing extensions could also be solved with b3cb74737a542eef011b6ceea8ef62e059b7a267, as I had missing extensions here on windows, too, until I implemented probing all output devices for EFX support.

[Lwmte]

By the way, can we store OpenAL Soft as a reference too? OpenTomb makes no use of native SDL audio features, and now, when XAudio2/mmdevapi business is sorted out, we can use official version without modifications. P.S.: Chris also said to me that using SDL as a backend with OpenAL is unsafe, even in SDL2. There is a risk that OpenAL may still be shutting down while SDL is already destroyed on exit, which is dirty.

If you need a link to OpenAL git repo, here is it: http://repo.or.cz/w/openal-soft.git

[Lwmte]

Also, what about SDL2, SDL_Image, zlib and Lua? I don't remember whole story, but why these were left as links to pre-compiled libs, if we decided to convert everything to references? Or there is a reason for such setup?

[vvs-]

As far as I remember, you and TeslaRus said that the only patched libraries were bullet and openal. Cochrane also needs patched openal for Mac OSX. Why freetype is there I don't remember. It was decided that it's better to use system libraries for everything else. Actually, I think that developers and users should have different setups.

[Lwmte]

Well, currently we're not using patched Bullet anymore (by the way, Lua was also patched, but it is now referenced as external library, not as submodule), and seems we don't need patched OpenAL either after all this discussion about XAudio. So effectively, only patched library we currently need is stohrendorf's LuaState fork (ironically, as Steffen proposed removing patched libs in the first place).

Cochrane had problems with stock OpenAL for Mac (I have no idea what's the version, probably it's a fork of last open-source Creative OpenAL, which should be ridiculously ancient), what he really needs is any version of OpenAL Soft, and there is no real difference between submodule reference to current git repo of OpenAL Soft and bundled patched version. Only difference is backend.

So... I believe it's time to finally decide how we manage libs. If we've decided to use submodules, we should get rid of system libs completely (meaning - SDL2, SDL_Image, zlib and Lua should be also added as submodules... ah, I also forgot about libsndfile and ogg/vorbis). Right?

[Lwmte]

And I'll say again - I believe there is an ongoing issue with different includes of Bullet modules, as I have seen <> and "" links to them roaming around.

[vvs-]

I don't see why it should be either one or the other. Why not both? You can keep only patched libs as submodules and rely on the system libs for everything else. I should point out that if OpenTomb will ever get packaged for distribution, then it should rely on the system libraries.

As for includes I think it's better to keep them global as <>.

[Lwmte]

Ok then, basically, we need only LuaState as submodule then.

[vvs-]

Sure. We can always add anything if needed.

[vvs-]

BTW, LuaState is header only. It doesn't need runtime.

[Lwmte]

But then we return to the problem described by Cochrane in the first post - what if someone has different library version? I must say that so far it wasn't an issue for me, cause I never encountered compile problems because of wrong lib versions. Only ogg/vorbis were picky about particular flag settings (fseeko64 function), but these were integrated all along (I should confess, it was me integrating ogg/vorbis sources, because I couldn't find pre-compiled lib binaries for MinGW).

[Lwmte]

And, as I said earlier, I'm also in favor of this approach (system libs). Only headache for me would be compiling ogg/vorbis.

[vvs-]

what if someone has different library version?

We need to choose something. And every choice will have its pros and cons. We can change our decision later. Right now it's not a primary concern.

[Lwmte]

For me it's the primary concern now, as I want to finally build this source code! :smile:

[vvs-]

If I understood correctly, you'll need to build all the libraries anyway no matter if they are in-tree or outside.

[Lwmte]

Yes, and building libraries themselves is something I never did before... Actually I just googled for compiled .a/.la files and just put them into /lib directory, and headers into /include directory! But this won't work for ogg/vorbis and bullet, because there's no pre-compiled library binaries on the net (except outdated versions).

[vvs-]

If there is CMakeLists somewhere then consider it done. If not then CygWin could help. In any case there should be tutorials somewhere on the Net for building it under Windows. If nothing will work just ask here and may be someone will have an answer.

[Lwmte]

I have successfully built both Bullet and Ogg/Vorbis. It was actually piece of cake, no errors or warnings, only thing I needed to do is to change build type in CodeBlocks from "commands only" to "static library". Now I'll try to build latest OpenAL Soft! :sweat_smile:

[Lwmte]

Ok, I have built all the libraries, and now I'm trying to compile engine using CMake-generated CodeBlocks project. I should also note that I've removed integrated OpenAL and put freshly compiled libs and includes into my minGW folder. The problem is, CMake links to OpenAL dll from Windows system directory, but I want it to link to my static library from minGW folder. How can I do this?

P.S.: I believe that my problem is somewhat similar to Cochrane's problem, as CMake points me to system OpenAL dll, which is "genuine" (yet ancient) dll from Creative in Windows\System32 folder. But I want the same thing as Cochrane, i. e. ignore system OpenAL and use OpenAL Soft instead.

[Lwmte]

It's a miracle! I have successfully compiled it without warnings and errors! How it's possible? Every time people introduced something new before, I had to break my head trying to find what's missing, like flags, includes, build options, etc. But now it looks like some kind of CMake magic... I think I'll use CMake only for now.

However... Seems that OpenAL Soft isn't the only one which tries to link dynamically... Glew also wants to be a dll, how can I prevent that? Is there any command for CMake which forces certain lib to be identified and linked as static?

[cochrane]

Sorry for taking soong. Yes, my problem with OpenAL is that the version included with Mac OS X is ancient. Any version of OpenAL Soft should work just fine for me.

[cochrane]

*so long (can't edit my comments on the mobile version of Github)

[Lwmte]

Yes, then I think we should just leave OpenAL Soft and LuaState as submodules (OpenAL Soft for Mac only). However, how can we use OpenAL as static lib in win/linux, while using it as submodule on mac? How to manage includes in such case?

[Lwmte]

P.S.: Just found CMake command BUILD_SHARED_LIBS, will it prevent linking dynamically? P.P.S.: I actually see no benefit in linking statically, it just looks cleaner. I'm fed up with DLL hell created by original engines and their patchers, so it's always better if user just have .exe file in his engine directory. But if we eventually make a setup, it makes no difference. I can live with this! :smile:

[Lwmte]

...However, on second thought, I had to put these DLLs in the same directory as binary:

• glew32.dll
• libjpeg-9.dll
• libpng16-16.dll
• libtiff-5.dll
• libwebp-4.dll
• libsndfile-1.dll
• lua53.dll
• SDL2_image.dll (magically, SDL2.dll isn't required)
• zlib1.dll

Don't you think it's too much?

[vvs-]

Oh, libpng, libtiff and libjpeg are dependencies too. It's a standard set of libraries which seems to always be on any Linux system, so I didn't even notice that they are required. That shows that to build a complete set of requirements we need to build it on all supported systems. And I think cross-compiling should be included too.

[vvs-]

Just found CMake command BUILD_SHARED_LIBS, will it prevent linking dynamically?

"_BUILD_SHARED_LIBS Global flag to cause addlibrary to create shared libraries if on. If present and true, this will cause all libraries to be built shared unless the library was explicitly added as a static library. This variable is often added to projects as an OPTION so that each user of a project can decide if they want to build the project using shared or static libraries."

[Lwmte]

This is exactly what we need. Although, I tried to modify CMakeLists.txt to make static libraries, but it had no effect... I think someone with better knowledge of CMake should implement this.

P.S.: Please check recent changes - I have removed Bullet and freetype, and made OpenAL Soft submodule. It works correctly on my side, although engine is completely broken (Lara looks like abomination, textures are messed up, entities are missing, etc.)

[vvs-]

But we already identified these bugs (at least partially):

Lara looks like abomination,

That's #143. Not only Lara, just any animation.

textures are messed up,

That's #124. I've reverted the whole texture atlas in my branch.

entities are missing, etc.)

That's #145. Also, #137.

You can temporary workaround #137 by patching frustum so that isPolyVisible always return true.