[DISCUSSION] Plugin Functions (what will do)

[nonunknown]

While we discuss UX at #1 I think its important to decide the plugin functions, I mean what it will be able to do. Also a lot of objectives was mentioned in godot-proposals 119, but here we need to specify more things, and more specific things too. Here's my list:

• Copy source files to project folder e.g library/HelloWorld.cpp to godotproject/cpp/HelloWorld.cpp
• Support Cross-compiling for all platforms
• Support Rust Language (Done yet!)
• Support Nim Language (to study)

[willnationsdev]

Support Rust Language (Done yet!) Support Nim Language (to study)

I would try and make language-specific operations an async/one-way dependency. After all, the end-goal is to have whatever plugin design we make be something that could conceivably be integrated into the official engine in a future release (once the design is finalized). So, the code should try to work off a generic interface to perform all of its operations, and then each language binding should provide its own implementation of that interface to plug its own operations into the required methods that the plugin depends on. This is referring to the various operations that 119 mentions.

So, rather than the quoted stuff above, it would just be something that bindings include in their own repos that implements a class that we have defined (and they could maintain that in a separate branch, or their master, or however they want to do that).

[SIsilicon]

How about the ability to support custom language, as long as you have the templates and build script for them. That way, supporting languages like Rust and Nim can be done more efficiently, and can be maintained apart from the main plugin.

Edit: Basically what @willnationsdev said.

[SIsilicon]

While we are taking about Plugin functions, where we should place all the data for the following?

• source code per Library
• binaries per Library
• the languages' data (templates and build scripts)
• bindings (per language?)

For reference, I have the following setup in my plugin now.

• source code under addons/gdnative_data/<Library>/src
• binaries under addons/gdnative_data/<Library>/bin
• language data under <Godot Data Folder>/native_languages
• godot cpp bindings under <Godot Data Folder>/native_languages

[willnationsdev]

Cross-compilation, also, should be something that is mostly handled by the target language since they are the ones defining what compilation tools should be used.

All our code should do is ascertain what the current platform is and provide an interface for

1. What platforms are available for target_language?
2. Which platform out of those options does the user currently want to target?

These would both be questions that are answered by methods on a GDNativeBuild object since the object defines the entirety of the build configuration (or something to that effect - we might wanna distinguish between a Library and its source code files vs. the configuration of parameters to pass to a given compiler tool.

[nonunknown]

The problem is , @SIsilicon, that CPP and Rust, and probably Nim, workflow are totally different, there's no way to create a pattern between them, I recommend you to check godot-rust bindings, there you will see how it works.

[willnationsdev]

@SIsilicon My proposal handles all that stuff in a GDNativeBuild resource (generic interface) with language-specific implementations of those scripts. Then, the user should be the one to decide where they want to store their source code/build files (res://, user://, 3rd-party location). It should all be something the user decides.

[willnationsdev]

@nonunknown Isn't the basic pattern of how builds work all the same under-the-hood? Even if the exact files are different and the exact compilers, etc. you still have some collection of source files, an optional set of header/definition files (C/C++/F# all have things like that), and a list of dependent library files. Then you need the compiler tool, the target platform, the target build type (release/debug), and whatever other arbitrary list of compiler options are needed (which the language should be able to specify). Do C/C++, Rust, and Nim behave in different ways in these regards?

[SIsilicon]

The problem is , @SIsilicon, that CPP and Rust, and probably Nim, workflow are totally different, there's no way to create a pattern between them, I recommend you to check godot-rust bindings, there you will see how it works.

Just checked out the rust bindings. I think that as long as the process of building the code can be automated from a python script (assuming we're using Python here), then any language can be implemented as a "custom language". After all, they all still build the same thing: a binary that Godot can use.

[willnationsdev]

@SIsilicon Idk if we want to assume a dependency on python though. I suppose we could, but I do know that there are people who want to compile Godot with their own custom compiler options (not scons) so they don't necessarily have to be tied to Python. But if it is necessary to use python in order to have git-less fetching/unpacking of git repos for source code, then demanding that dependency might be worth it.

[SIsilicon]

Yeah, after all, you don't need Scons for every language. the language can simply request that Scons should be installed. Plus, using python instead of a batch file or shell script makes it easier to cross compile, plus you get more built-in features such as tkinter, which I use to prompt the user to locate the android ndk for compiling to Android.

[nonunknown]

@willnationsdev as you mentioned under the hood is the same, but the workflow changes, rust need to recompile the bindings everytime (when I tested, maybe I can be wrong) but it already generate everything the workflow is way easier than cpp

@SIsilicon There's no need to automate with python, when can do everything executing shell commands this way we avoid more dependencies as will mentioned.

Rust workflow:

cargo --lib lib_name_here modify your code cargo build done

CPP workflow: Generate Bindings Create your code Create Gdlib.cpp (register classes) Compile Code Done

[SIsilicon]

Yes, but when you use shell commands, you then require to make a version for the three different desktop platforms, which means more code to maintain (batch for Windows, shell for Linux and Mac).

[willnationsdev]

@SIsilicon Ah, true. Yeah, python would definitely make that easier. And make just writing the code a lot easier (geez, I really dislike bash/batch scripting - Python's a lot better).

[willnationsdev]

Wait, does Rust not have any sort of bindings generation process? How are you supposed to use Rust in master versions of the engine? Do they not generate their API automatically?

[willnationsdev]

Here we go. They do still have the ability to generate bindings for custom builds: https://godot-rust.github.io/book/advanced-guides/custom-bindings.html

[SIsilicon]

Whether they generate their bindings or not shouldn't matter to the plugin. If it can be automated in a batch file, or any script for that matter, all we have to do is expose enough parameters for scripts to build with to support most languages.

[willnationsdev]

If we are going to have compatibility with Python scripts to handle executing operations, I'd still want the data to be editable from the Godot Editor. In which case, we would likely need to have a Resource type that saves its data in a JSON format which Python can then also access without needing to know anything about Godot's Resource file format.

That way, a user can use the Inspector to generate the GUI for editing data related to their builds, but the execution of the logic is converted into a Python-readable format and delegated to a Python script.

[SIsilicon]

Such data can be passed to python via command line arguments.

[willnationsdev]

Ah, that's true too. Simpler as well. So just use a Resource in Godot, and then use OS.execute to run a python script where you pass in all the information from the resource as arguments? I think actually did something similar with build tools in my original godot-builder addon.

[SIsilicon]

Yeah that's basically what I'm suggesting. Any custom command arguments can be appended to what the builder already passes in (platform, architecture, etc...).

[willnationsdev]

Okay, here's a tenative/preliminary list of features:

(Just assume that most language-specific operations that interact with the operating system are just generally handled by Python script calls)

1. Plugin defines a core GDNativeLanguage script (extending Reference probably)
   • get the name of the language
   • get source file extension
   • get header/declaration file extension (may not be applicable to a given language)
   • get available bindings versions (e.g. C++ has 1.0, 1.1)
     • this would be for officially supported versions, independent of custom-built ones from the end-user
2. Plugin defines a core GDNativeBindings script (extending Reference probably)
   • callback to get version number
   • callback to get which versions of Godot are compatible with the given version number
   • callback to get whether the given version is compatible with the current version of Godot (values of Yes, No, Unknown for custom builds)
   • callback to generate bindings from an API JSON file.
   • callback to fetch bindings from a third-party location.
   • callback to build bindings to make a static library. Optional to store the static library in a centralized location or in a project.
   • callback to generate a list of project template names
   • callback to generate files for a template project (with a given template project name selected)
   • same as above two, but for template classes associated with the language. A GDNativeBuild would also need to be added as a parameter so that the generated class files can be added to the language-specific project files and/or added to the GDNativeBuild resource's GUI.
     • e.g. if I tell a GDNativeBuildCpp instance to add_class("MyNode"), I'd expect to get a my_node.h, my_node.cpp, and my_node.gdns. I would also expect that the Build object is keeping track of the current name/path for the GDNativeLibrary file it intends to generate, and making a .gdns file would automatically set the field accordingly. Further, if the GDNativeLibrary name/path is ever edited, the GDNativeBuildCpp should automatically detect this and respond by updating all of the stored .gdns filepaths so that the files at those locations begin pointing to the new GDNativeLibrary path without having to manually go to each one and change it.*
3. Create a GDNativeBuild resource
   • GDNativeBuild resources can have child GDNativeBuild resources.
     1. Configure dependencies
     2. Add/Remove source files (or add directories for recursive adding of files based on file extension)
        • Maybe render this using a Tree in the Inspector? Right-click to "Add File" vs. "Add Folder"? That, or just have 2 fields in a property group, one for export(Array, String, GLOBAL_DIR) and one for export(Array, String, GLOBAL_FILE).
     3. Add/Remove header/declaration files (same as above)
     4. Add/Remove library files (for any dependent static/dynamic libraries - these need to be two separate lists?)
        • Any attempt to build a GDNativeBuild resource will recursively merge all "dependencies" of the build into master lists at the top level which the top GDNativeBuild then passes to a Python script.
     5. Configure build arguments
        • get list of available build configurations from language-specific implementation (debug/release/etc.).
        • property to choose from that build configuration list (a string enum).
        • get list of available bindings libraries.
        • property to choose from that bindings libraries list (a string enum).
        • same as above two pairs, but for architecture (32bit/64bit)
        • similar pattern for whatever other build arguments you can think of.
4. When a particular GDNativeBuild resource is selected...
   • Add/Remove a class to/from the build

* - For any operation that is expected to take an especially long time, it ideally shouldn't trigger in a synchronous OS.execute call. It should be an async operation in a separate thread (just make a separate thread first and then do a synchronous OS.execute call within that Thread callback. Use a custom Thread type that emits a signal when it finishes. Then you can yield for that signal, thereby stalling local execution, and when the signal is done, you can finish waiting for the Thread to clean it up. This way, the main Editor thread doesn't stall while the background operation is executing, and when it is done executing, a signal is emitted which you can respond to for updating the GUI with a different icon or re-enabling temporarily disabled buttons, etc. (e.g. can't build again while already building / can't build while generating bindings, etc.).

We might also want to separate out the files, folders, and libraries associated with a build vs. the release/debug, architecture, and other build arguments since one will conceivably want to be able to easily switch between those things without having to have a duplicate larger resource for the entire "native project". Not sure what to call it though. Maybe have GDNativeBuild and GDNativeBuildArgs? And the former has an array of the latter plus another field for one being selected?

WDYT?

[SIsilicon]

Pretty well thought out. However, I do feel like some parts are needlessly complicated. For starters, I don't think we'll need a whole class for the building threads. It would be best if we built each library one at a time to avoid clashes, so having just one thread should do the job. Secondly, since binding would be handled by each language separately, we could just have it so that the python scripts handle the bindings data themselves. They can also query the user for what version of the binary to use, if they want to. They can also handle creating a sort of config file to store all those settings. Said config file can also be created and edited per library/build.

Tl;dr I don't think we need a resource for everything.

Having a GdnativeBuild resource extending GDNativeLibrary sounds like a great idea though. It could even have a property to switch between debug and release, stores the language, and may have a custom config file as mentioned above.

[nonunknown]

Wow huge list, I think we should've to start with little and the increasing the scope, in my mind just compiling code to some languages would be okay, then after some iterations we start to add extra features to personalize builds and stuff like that!

Also I got an answer from Godot-rust binding guys https://github.com/godot-rust/godot-rust/issues/637

[willnationsdev]

@SIsilicon

I don't think we need a resource for everything.

That may be true (don't know everything we will need yet), but we will want all data that should be user-editable to be placed in a Resource so that users don't have to dig around in config files to mess with the data for their native libraries. A user should never be required to tinker with text files to properly configure the basic needs of getting started with a language. That should be something reserved for power users. At least, in my opinion.

For starters, I don't think we'll need a whole class for the building threads.

The extra class extending Thread is only necessary if we have the main Editor thread needing to respond, visually, to the completion of any launched threads. Which likely will happen if we want to give the users proper visual feedback on things like, "I clicked a button to fetch/build bindings to get a static library. How do I know if that operation is still loading? Has it completed yet? Should I have the ability to attempt to build that same language while the bindings are still being prepared?" Those are all questions the Editor should be answering for the user visually in response to the progress of threaded operations.

It would be best if we built each library one at a time to avoid clashes, so having just one thread should do the job.

You could do all of it in the main thread, but that would mean that literally the entire Editor would be locked until the operation completes. That leads to a worse user experience since the user can't navigate the Editor in any capacity or perform separate actions whilst the operation is executing. I really wouldn't recommend that.

Secondly, since binding would be handled by each language separately, we could just have it so that the python scripts handle the bindings data themselves.

I could potentially see this happening. So long as the input data can be configured from visual stuff in the Editor. As mentioned, I don't think people should need to mess with text files if they ever need to adjust something. Chances are, a new user won't care to use the non-latest-stable version of a particular set of bindings, so it would probably be fine to leave it as something in a text file, and possibly not changed often. So, if it ends up being better in the code maintainability to just keep all bindings management in the Python script(s), then so be it. But if it isn't that much more complicated to include the information in some "advanced" settings in the GUI, then there isn't much of a reason to take it out imo. Just a "we'll see what it's like when we get into the nitty-gritty of implementing it".

Having a GdnativeBuild resource extending GDNativeLibrary sounds like a great idea though.

I don't think it should extend the GDNativeLibrary resource necessarily since they fulfill different purposes. The Library exists to match up multiple dynamic library files with one conceptual/abstracted library so that it matches up to different targeted platforms. The GDNativeBuild would be something that defines the arguments that a library should be built with. One could potentially have a different set of build configurations for all kinds of different platforms. Since they don't have a 1-to-1 relationship, it shouldn't be an inheritance relationship.

and may have a custom config file as mentioned above.

I'm assuming, by this, we are referring to the concept of having the Resource be saved as some sort of Python-readable file format (.cfg, .json, whatever) so that it can be visually edited in the Inspector, but available to Python? Or, if it's only ever just having the Resource pass parameters to Python script calls in GDScript, then I don't see any reason to maintain a separate set of config values inside a config file. Might as well just keep it all in the Resource in that case while saving it to a regular .tres format (if it's purely a one-way thing of passing parameters to Python).

Wow huge list, I think we should've to start with little and the increasing the scope, in my mind just compiling code to some languages would be okay, then after some iterations we start to add extra features to personalize builds and stuff like that!

I'm good with that to a certain extent, but the more you think ahead and are able to set up the basic architecture to handle abstraction/customization, the easier it will be to refactor the code later on to meet users' needs. Otherwise, you have to go digging around to separate out the things you need, change relationships (which will inevitably break code and cause bugs), and fix stuff that could have been avoided to begin with.

Of course, you guys are likely the ones to be working on it more than I am, considering my availability, so there's a limit to what I can suggest you should do. XD

Also I got an answer from Godot-rust binding guys godot-rust/godot-rust#637

Well, uh, I have to say @nonunknown, you might not have handled that the best way. I mean, the guy sincerely tried to help you with your problem, explaining what the issue was, and then you told him off for now doing things the way you wanted. But the fact of the matter is, the general practice is for people to independently be responsible for installing the necessary toolchains that allow you to compile for other platforms. Even Godot's official documentation has examples of that, where to compile iOS on Linux, you have to install a bunch of Apple C API toolchain crap.

It's the same story with compiling for any platform where the necessary C API libraries aren't already installed. The compiler tools, or the people maintaining those tools, don't have the responsibility to make sure those exist precisely because the environment in which the compiler tools are executed may change, and there is no global guaranteed online source for finding that content (so no single solution exists). Each platform will acquire those compiler dependencies in their own way, so it's best left to documentation to just detail what things you need and then let users do the work of gathering them. In the same way that cloning Godot doesn't just automatically install Python, scons, and g++/mingw/cl for you. Each platform installs those dependencies in different ways.

With that said, our Python script will probably need to do the work of fetching those dependencies for any given platform. I'm not really sure what the work involved there is though as I've personally never cross-compiled anything to begin with. I'd probably look for guidance by viewing and understanding the source code for Godot's own Python build script with scons and see how they go about using stuff for cross-compilation. It could be that Windows/Linux/MacOS already have necessary desktop C headers pre-installed? If not, then I'm not sure how cross-compiling with scons works out-of-the-box (maybe the scons documentation has more details about that?). In the worst-case-scenario, each language would require different C headers (which may very well be the case), and if so, you'd have to have python scripts in each target language's repo (e.g. godot-rust and godot-nim would have their own) which would be responsible for cloning those C headers from some prescribed place online per-platform (exactly the kind of thing that rust guy didn't want to have to maintain).

So, basically, we need to figure out what can be done from a centralized repository, and whatever ends up needing to be language-specific, we have to then contact language maintainers to see what sort of relationship they suggest having with their codebase. The more we can do from one place, the better, but there'll likely inevitably be things we have to outsource for language-specific actions.

[SIsilicon]

Coming back to this, I'm thinking maybe instead of python, we could use GDScript to build the libraries, plus it removes the python dependency. Not every builder need scons. All the current build script does right now can be easily translated to GDScript. The only downside is that the source code has to be within the editor, but that could be worked around by copying the source code into the resource folder temporarily.

[nonunknown]

@SIsilicon I talked with Vnen, he told me that this plugin will be officially integrated into godot, but for that it must be usind Docker in order to support all platforms compiling, I have little knowledge on that, but we need to consider this case too.

[SIsilicon]

@nonunknown Oh, ok. Is there a link somewhere to the discussion I could go to, or was it private?