add dynamic linker support

[mohaalak]

Why do we need it?

with the work of @reduz and alket soon we can have an asset sharing ( addon ) but the missing system is that this addons is all in gd script but there are so many works that can be done in modules like add Admob,IAP,OpenCV,CameraCaputre,VRsupport and so many more , this modules should be compiled and it's scary for beginners just to clone the godot repo add the module then download dependencies and compile it but think about it if the modules is compiled and can be added to engine it will be great.

Problems

• first of all that this modules should be compiled with compilers that the godot was compiled
• every version of godot can load just the modules for that version
• our last problem how can we implement it that it will be simple and easy to use both for module developers and engine users

my solution is that we just support those release on steam and in site the stable versions and if someone wants a custom version he should download the modules code and compile it themselves, for compatibility issue the modules should have a supported version and for different platform the compiled version should exist.

@akien-mga I will appreciate it if you add feature request and discussion label to this issue

[neikeq]

Related to #2701

[reduz]

I understand, but this is too much work for too little gain.

What we could do is configure Godot so you can set a path to your Godot repository that you compile yourself, and downloading the addon will add a module into Godot, that you can recompile.

On Sat, Mar 5, 2016 at 4:46 PM, Ignacio Etcheverry <notifications@github.com

wrote:

Related to #2701 https://github.com/godotengine/godot/issues/2701

— Reply to this email directly or view it on GitHub https://github.com/godotengine/godot/issues/3936#issuecomment-192718298.

[mohaalak]

I think it's too little gain is for you and me but think about the developers that they hope with godot they can bring food for their children :stuck_out_tongue_closed_eyes: and does not know anything about android sdks , ndk, gradle, or some of them even does not anything about git , I just say if it is lots of work just direct me in the right direction let me support this system but if you think that it is bad design for godot and you don't want that in godot then I think I should close this issue.

[Zylann]

Would having a C API make the problem easier to solve?

[ghost]

@Zylann it would, in fact #3943 would make it possible to write wrappers directly in GDScript with no performance penalties.

You could (for example) just distribute an OpenCV module written in GDScript itself and it would call C code directly, much like it happens in other languages with FFI support (like this). In short, there would be no need for compiling.

But it should also be possible to write modules and link them dynamically. I think this is a huge issue Godot has - having to compile your own C++ modules along with all of Godot makes no sense - but perhaps the problem is being cross-platform. What about something like Pluma?

[bojidar-bg]

Linking modules dynamically is not impossible (even without additional libraries), but might be a pain to setup correctly. I have discussed the issue with @est31 once and we actually went rather deep into it.

I might do this one day, but given the current state of affairs, for the next few months I have enough other issues to tackle first. If anyone of you wants to take it, I'm free to speak about it on IRC.

[reduz]

The problem is not linking modules dynamically, this is super easy. The problem is creating modules for every single platform Godot supports.

On Wed, Jun 8, 2016 at 2:13 PM, Bojidar Marinov notifications@github.com wrote:

Linking modules dynamically is not impossible (even without additional libraries), but might be a pain to setup correctly. I have discussed the issue with @est31 https://github.com/est31 once and we actually went rather deep into it.

I might do this one day, but given the current state of affairs, for the next few months I have enough other issues to tackle first. If anyone of you wants to take it, I'm free to speak about it on IRC.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/godotengine/godot/issues/3936#issuecomment-224661922, or mute the thread https://github.com/notifications/unsubscribe/AF-Z25uZVzcO9fhZqKDHirjbOIhYiBpbks5qJvgsgaJpZM4HqEgW .

[bojidar-bg]

@reduz That would be the module developers' problem, not ours. So not a big issue, since even now, the modules have to be compiled for every single platform.

[reduz]

Yeah, the problem is that it's a lot of work for the module developer.

On Wed, Jun 8, 2016 at 2:41 PM, Bojidar Marinov notifications@github.com wrote:

@reduz https://github.com/reduz That would be the module developers' problem, not ours. So not a big issue, since even now, the modules have to be compiled for every single platform.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/godotengine/godot/issues/3936#issuecomment-224670379, or mute the thread https://github.com/notifications/unsubscribe/AF-Z2_-qH36U3CKDl09COFfwWIzGNxPwks5qJv64gaJpZM4HqEgW .

[neikeq]

@reduz I think a developer won't mind that if it means more users will use the module (since users don't need to build it nor the engine). Also a developer may just release the source and the users could share their builds (like unofficial builds of Godot).

[reduz]

An alternative I thought about could be to make a simplified, generated from bindings, godot API for this to link from C or C++. It would at least make it easier to link and keep compatibility between versions

On Wed, Jun 8, 2016 at 3:02 PM, Ignacio Etcheverry <notifications@github.com

wrote:

@reduz https://github.com/reduz I think a developer won't mind that if it means more users will use the module (since users don't need to build it nor the engine). Also a developer may just release the source and the users could share their builds (like unofficial builds of Godot).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/godotengine/godot/issues/3936#issuecomment-224676665, or mute the thread https://github.com/notifications/unsubscribe/AF-Z285dFHCuBTDnh5wr0iGAwUs9-0uuks5qJwOxgaJpZM4HqEgW .

[ghost]

@reduz You mean like a libgodot.so which can be called from C++?

[reduz]

yes, exactly, you link your program against a wrapper header/lib and generate a dll/so. Godot then loads this .so dynamically for every platform

On Wed, Jun 8, 2016 at 3:40 PM, paper-pauper notifications@github.com wrote:

@reduz https://github.com/reduz You mean like a libgodot.so which can be called from C++?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/godotengine/godot/issues/3936#issuecomment-224687867, or mute the thread https://github.com/notifications/unsubscribe/AF-Z2y0eIsGDfvLhf2OMazSX0AXb0-82ks5qJwyTgaJpZM4HqEgW .

[ghost]

@reduz That's a pretty good idea! Does this approach have any limitations? It would speed up C++ development a lot. +1

[neikeq]

@paper-pauper the limitations is that you have the same API access as GDScript.

[ghost]

@neikeq that would not affect wrappers (for libraries like OpenCV and MongoDB), which would only need to call the register hook. I think this is a good solution (but still think writing wrappers in GDScript itself via #3943 should be implemented at some point for easier distribution, much like languages like Python have wrappers written both in C++ via the API and in Python itself via ctypes/ffi).

[Zylann]

I don't really like the fact that GDScript would be required to bind dynamic modules, because GDScript is a module itself, it is optional. Modules depending on other modules is not a bad thing, but for such a core feature the engine's API should be enough.

[ghost]

@Zylann For what @reduz proposed, GDScript is not required at all.

For what I proposed in #3943, you raise a good concern but GDScript is always bundled with Godot, so it'd be the best way to distribute wrappers (aside from C++, which is always possible but requires compilation, which can be less comfortable). What other language aside from those two would you suggest for wrappers? Would you rather not have wrappers at all than have them in GDScript?

[Zylann]

I wouldn't use an intermediate language, actually, or maybe use the reflection/Variant system, which is already a dynamic interface to the engine. I understand that it would reduce the requirements for the module creator. However, since inheritance it's heavily used in the engine, how would you inherit from Node, Reference, or Resource?

Also, theoretically, if you setup a standard build system for a module on each platform you want to support, you can easily update modules for newer versions of Godot.

At the same time, I'm curious about what will be the best solution. I've worked on a few projects involving dynamic modules and realized a lot of problems need to be solved. My last engine project was litteraly designed around loading modules dynamically. Just like with Godot, they had access to all the engine (which was itself a library), assuming they won't change during execution because it's too much work to maintain validity of objects. So adding or removing native modules requires to restart the engine, but from a user's point of view, that's way better already than having to setup a whole environment to build everything for any platform.

[reduz]

I'll revive this thread with a proposal. Let me know what you guys think about it.

First a bit of context:

Distributing dlls/so files in Godot as external modules is to difficult due to the many different compilers, versions, ABI changes, etc. In a bit more detail.

1) C++ is terrible for binary compatibility in shared objects, the ABI is different between compilers and compiler versions. 2) Even if the same compiler was used, C++ requires symbols (class) sizes to be the same in order to link properly

This makes adding extra functionality as a .dll linking against Godot too difficult to even attempt.

An idea lurked around about creating a more stable mirrored C++ API that replicates API functions using a safer API, but this would only solve 2), not 1).

So, how about the following? Some functionality must of course be used as an extension, such as if you want to add a module that might replace the physics engine, but for the large amount of libraries and other stuff that might be nice to bind (ie, SQLite, ODBC, DBUS, Kinect, FMOD, whathever), having access via the script API to this would be fine. I think this could be solved via the following:

1) Create a very simple dynamic API via C, not C++ which can access most godot functionality, but only provide a minimal API. 2) C ABI is much more stable and compatible between compilers and compiler versions than C++. 3) C ABI would be simple and access most of Godot via the reflection API instead of headers, ie

godot_class c = class_db_get_class("Node2D");
godot_instance ins = godot_instance(c);
godot_call_method( ins, "set_pos", godot_vec2(10,22) );

the idea would be to simply add scripts using C, not sure how this would work exactly (CScript could be a class where you edit which which C class to instance) .

This would allow us to bind all sort of C and C++ libraries without any risk, given the ABI will always be C.

What do you think?

[reduz]

To add a bit more depth on how this would be used, we would just add a specific type of addon that is a CPlugin or somethin like this, like

addons/sqlite/sqlite.Linux.x86.32.so addons/sqlite/sqlite.Linux.x86.64.so addons/sqlite/sqlite.Windows.x86.32.so addons/sqlite/sqlite.Windows.x86.64.so addons/sqlite/sqlite.android.arm7.so

etc

Even WebAssembly has shared object loading in the roadmap.

This way it would be super simple to distribute this kind of content, and even distribute it in the asset library.

[SuperUserNameMan]

I've reread twice, but still not sure to understand what you mean by "the idea would be to simply add scripts using C".

For instance, let's say I want to make a Text To Speech plugin. My first intuition would be that my plugin would have to create or extend an AudioStream with some custom methods and properties. Would it be possible ? How would I have to do ?

[neikeq]

Would you be able to create and register object types and bind its methods with this approach? If so, how would inheritance work?

[reduz]

My first intuition would be that my plugin would have to create or extend an AudioStream with some custom methods and properties. Would it be possible ?

By design no, since there is not really any way to extend AudioStream and provide audio from a script. The script APIs are not low-level enough.

However, for such special cases like mixing audio in low level, or any other case where you want direct C access, it would be pretty easy to add a special API to do this as an exception..

[reduz]

Would you be able to create and register object types and bind its methods with this approach? If so, how would inheritance work?

yes and no. It would work the same way as scripts. If you want to create a custom button in C (not a very useful example :P ), you would specify that your C script (likely via API) is intended to go into a Button, then it will receive the same information as a script placed on a Button.

In the scene tree, you make it appear as custom types via EditorPlugin API, the same way GDScript scripts appear there,.

[reduz]

Again, what I am proposing is basically a "Use C as scripting language"

[reduz]

The usage will be really simple,. we' ll just a provide a single .h file for C programs to use with the Godot reflection API exposed in C. (again, not all classes and functions in a header, just the reflection API ).

This should be enough to make very simple shaderd objects/dlls we can even provide a small SConstruct to build those projects standalone.

[neikeq]

I like the idea. If it does not limit the extension to .c then I would be able to create scripts with D using extern(C) :)

[reduz]

I like the idea. If it does not limit the extension to .c then I would be able to create scripts with D using extern(C) :)

you literally would use anything that interacts with C

[ghost]

I think this is a great idea, and it could really bring a lot of stuff to Godot. +1

[SuperUserNameMan]

Ah okay, I think I understood better now.

So for instance, this would allow us to use gdscript for prototyping complex AI algorithms, and later convert the gdscript into a faster compiled "C script".

And for instance, in case we don't have the ability to compile our Cscript for Macintosh or for Emscripten, we could just provide the original gdscript version instead of the "C version".

Sounds great !

[reduz]

So for instance, this would allow us to use gdscript for prototyping complex AI algorithms, and later convert the gdscript into a faster compiled "C script".

I think for your own project, a C++ module will be way more easy to use than this. This method of using C directly will be a lot more cumbersome than writing a C++ module, but the benefit will be being able to distribute compiled C code readily to be included in all exportable platforms.

[Zylann]

So when you write a C script, you don't really have native access to the "Godot standard lib"? No DVector, no memory tracking, basically you need to recode everything?

If I were to convert my Voxel Tools module to this API, how would I generate voxel Meshes with SurfaceTool with the reflection API without slowing down the algorithm because of reflection calls hashing strings and creating Variants? (How would you have Variant in a C API btw?) And how should I provide nodes such as VoxelTerrain or VoxelSprite?

If I follow the audio example earlier in the thread, there could be exceptions. Keeping this optimized means more exceptions :s

[mohaalak]

@Zylann reudz's idea is best to use when you have a library and you want to set an api for it in gdscript without compiling the whole engine. but I think if you want to have a voxel mesh and voxel tool you should write modules then compile the whole engine.

[reduz]

@Zylann If your voxel tools are meant to be distributed as an addon, I don't think it would be too troublesome to move code to C, but you would probably only want to move the voxel generation part. Should still be pretty fast though.

[neikeq]

Regarding @Zylann's comment, the downside of @reduz proposal is that godot_call_method(ins, "method_name") uses variant call. IMO it would be better to provide method calls for every method in the API which call their respective methods directly instead of using variant call. I don't think it would increase the size that much, 2~ MB at worst. The problem would be automatically generating such glue API, because there is no way to automatically know the real method name. Another problem is that some of those methods are private. This is the same problem I faced when writing my C wrappers and forced me to use variant call for now.

[Zylann]

@reduz I think passing thousands of vertices to build meshes only using the reflection layer with the C API will generate a lot of overhead. A project I worked on before had the same idea to make a particle simulator module to communicate with the engine, it delivered terrible results compared to what can be done with a native API. Also, as I said, making my own module would also require to include my own base classes, including STL, and being unable to see it in the Godot profiler (memory/CPU).

Aside from my particular case, not being able to native link directly is kind of frustrating, because, even if I know C++ binary compatibility is worse, I always used cross-platform libraries without having to recompile them, from small ones like SFML to Qt, which is huge. Just pick the correct version. This can be done. Why can't Godot follow this scheme? Is it too big to maintain? Does the code changes too frequently?

[reduz]

IMO it would be better to provide method calls for every method in the API which call their respective methods directly instead of using variant call.

This would simply add a lot of size overhead to Godot unnecessarily. The point of this API is to use external libraries, not so much improve performance (even though you can still do it). I can see it justified for Mono but not for this.

[reduz]

I think passing thousands of vertices to build meshes only using the reflection layer with the C API will generate a lot of overhead.

@Zylann I don't think so, generating geometry is just filling up a few DVectors. This should be really fast as the C API will have functions to lock them/unlock them and access the raw data inside. Calling Mesh.add_surface() through variant_call won't happen nearly as often, so it should not be a performance limitation.

[neikeq]

Would it be possible to have a function like godot_get_method(type, "set_pos") that returns a MethodBind pointer? This way we could cache a method when needing to call it many times.

[reduz]

@neikeq yeah, don't see why not

[reduz]

For reference to those reading the thread, here's some documentation of why C must be used instead of C++:

http://www.mingw.org/wiki/Interoperability_of_Libraries_Created_by_Different_Compiler_Brands

[Zylann]

With the C API method, is it possible to have modules that depend on others? For example, a basic noise generation module, then a voxel module using the first one on a native level?

[reduz]

Here's a draft of how the C API would look like. It would be a single file: http://pastebin.com/Qepcp5Nm

[ghost]

@reduz Looks clean and easy to use. Can't wait to see it in practice!

@Zylann I think it would be possible simply by using the "extern" keyword, since it'd be all dynamically linked.

[bojidar-bg]

@reduz I hope those binding are going to be created automatically, is this correct? :smile:

[Zylann]

@paper-pauper the problem is, if a module links another at compile-time as a dependency, how would it be usable as a Godot C module? My last engine project supported this case by simply letting the OS load the dependencies, putting the libs next to each other. The system keeps a reference count, so it doesn't matters if the engine loads it a second time after. Just wondering if that would be possible with Godot :)

@reduz I guess it will need some boilerplate for making C++ modules easier to develop.

[reduz]

@bojidar-bg the bindings will give you access to the regular Godot reflection API, you don't have to do anything from Godot side

@Zylann I don't think one of these modules will link another one

[Zylann]

@reduz It might not be that frequent, but it can happen, and is certainly possible to do. I already have module ideas in that would rely on this. Otherwise, the same issue that led Godot to support C modules would be present in modules again.

[ghost]

What about using dlopen to load shared libraries at runtime instead of linking?

In this case, where is the issue? All libraries will be loaded in the same address space, so simply using extern or the equivalent in other languages will allow modules to call other modules. (Or maybe my C is the issue, and I'm getting rusty)