Resolves #225.

This implements support for class registration from dependencies (and thus also from different modules).

Disclaimer:

Shipping of whole scene's is not supported with this. Only the class registration is handled with this
IDE plugin is not updated (it has other issues with the 4.0 changes as well. I will resolve all those issues in a separate PR as this one is already large enough)
Documentation is not updated (it needs updating in several areas for the 4.0 adaptation anyways and should be addressed in a separate PR)

Note: This also contains a small fix for reference and variant array type hints

Overview

This PR aims to support the registration of classes provided as a dependency to a godot-kotlin-jvm project. With this, a library (or module) author is able to provide classes and members marked with any of the @Register annotations and the user is able to use them without any wrapping or re-registration out of the box. It also provides some configuration options for both library authors and users on how to provide and use registered classes

Core registration changes

Original approach and it's limitations:

Originally we registered the user scripts through the kt files they were defined in. While easy to use and reason about, this has several drawbacks:

Resource path's need to be unique and thus only one script can be defined and registered from within one kt file
The file location needs to match the package path in order to correctly resolve classes based on the resource path
Registered classes from libraries cannot be attached to nodes, as the required kt files are no longer present. They got replaced by class files in the libraries jar file and thus no files are present to be attached to nodes

Especially the last point made it impossible to support libraries and modules with the current approach.

New approach:

Hence, we decided to ditch the idea of kt files being the resource for scripts. Instead, we generate a "dummy" file per script called "registration file". Inspiration for this idea was the same concept of "dummy" files used in GDNative. The extension for these new registration files is gdj (short for GodotJvm). With this, we're eliminating all of the above drawbacks as resource paths are no longer tied to kt files but the new dummy files we also generate.

This has only one drawback: The user is no longer able to edit scripts from the builtin godot editor.

But given that we never supported that in the first place, it should be no big deal.

New registration procedure

The entry generation now passes multiple processing rounds:

Round 1

The entry file and class registrars for the project's "main" compilation are generated
The registration files for all class registrars of all dependencies are generated

Round 2

The registration files for the previously generated class registrars are generated (aka the registration files for this project)

Round 3

Old/unused registration files are deleted from the project
If there are now empty directories left, they are deleted
The new/updated registration files are copied from the build directory to the designated directory in the project

The main reason for these rounds is simplicity: KSP (and compiler plugins in general) provide this concept of multiple processing rounds out of the box with very little performance penalty (if implemented correctly). With each new round, we get the new/updated files from previous rounds. Hence, in the first round we get all files created by the user and all registrars from all dependencies. In the second round we only get the new registrars we generated in the previous round. And in the third round, we get no new files.

New user project structure

Note: all of the following is configurable through our gradle plugin. See the corresponding section for a table view with the settings and their purpose

Project files

As previously the kt files were the resource's for scripts, the project structure was defined by the directory structure of the kotlin project. Typically this would look something like this:

<projectRoot>
    src/main/kotlin/com/company/project
        - ClassOne.kt
        - subpackage/
            - ClassTwo.kt

Now that we no longer have the constraint that the file location of the scripts resource fine needs to match the package path of the class, the project structure is simplified and more configurable by the user. The user can now provide a base directory to where the registration files are generated to. He can also specify if the classes should be registered with their fqName or their simple name by default and he can also specify if the registration files should be generated "hierarchically" matching the package path, or "flattened" into one dir:

hierarchical:

<projectRoot>
    <scriptRoot>
        com/company/project
            - ClassOne.gdj
            subpackage/
                - ClassTwo.gdj

flattenend:

<projectRoot>
    <scriptRoot>
    - ClassOne.gdj
    - ClassTwo.gdj

Dependency files

The registration files belonging to dependencies are always generated to a directory named dependency/<library-name>:

<projectRoot>
    <scriptRoot>
        dependencies/
            library-one/
                - ClassOneFromLibraryOne.gdj
                - ClassTwoFromLibraryOne.gdj
            library-two/
                - ClassOneFromLibraryTwo.gdj
                - ClassTwoFromLibraryTwo.gdj
    - ClassOne.gdj
    - ClassTwo.gdj

Dependency file structure

The library author can define if the libraries registered classes are exposed in a "hierarchical" or "flattened" manner and if they should be exported with the "fqName" or the "simpleName".

Another option for library authors are classPrefixes. A library author can define a class prefix with which every registered class will be prefixed. This can help with uniqueness when used in bigger projects

New gradle settings

name	default value	purpose
projectName	the gradle project name	defines the folder name in the "dependencies" folder in a consumer project
classPrefix	null	each registered class is prefixed with this prefix. No prefix when null.
registrationFileBaseDir	gdj	defines the root directory to which the registration files should be generated to
isRegistrationFileHierarchyEnabled	true	defines if the registration files should be generated with a folder structure which matches their package path or if they should be generated flattened into the [registrationFileBaseDir]
isFqNameRegistrationEnabled	false	defines if the scripts should be registered with the kotlin fqName by default or with the simpleName by default
isRegistrationFileGenerationEnabled	true	defines if registration files should be generated. Only really useful for library authors to prevent the generation of unneeded files

FAQ:

Q: Are the entry files and class registrars of libraries still needed?

A: Yes, the Bootstrap class now loops over the entry files and registers all classes and members of all dependencies and the main project

Q: How does the Bootstrap then know which entries are there to load?

A: That's a feature of ServiceLoader's. It loads all the entries from the class path using the service files which we generate (they reside under src/main/resources/META-INF/services) and are loaded automatically by the service loader

Q: When building the main.jar we basically create a fat jar. Now that every library has it's own entry file would that not result in conflicts because every entry file has the same fqName godot.entry.Entry?

A: Indeed that would be the case. To circumvent this, we now create a randomized subpackage for each project specifically for the entry file. An example would be godot.entry.wEOwpllYhwghEDgpklHa.Entry. As we generate the service file as well, we can just paste the randomized fqName in there and everything works as before. The ServiceLoader takes care of loading the Entry files with the randomized package name. With that the chance of a conflict is extremely low but of course not 0.

Q: Does this also work with our fat jar?

A: Yes, the "shadowJar" plugin we use can combine all service files from all dependencies into one.

Q: If each library now in essence registers its own registered members through their own entry files, who loads the engine classes which are common to all dependencies?

A: The entry files contain a new property which gets generated called classRegistrarCount. The Bootstrap uses this property to find the entry with the most class registrars from dependencies. The one with the most is always the "main" entry file. There cannot be a case where multiple entry file have the highest dependency class registrar count. The Bootstrap uses this "main" entry to load and register the engine classes

Q: The resource path is now no longer static for dependency projects. If a project is used as a godot project the path would be project/gdj/package/MyClass.gdj, but if it's used as a dependency it's suddenly project/gdj/dependency/dependency-name/package/MyClass.gdj. How does that work? Is any runtime cost involved in resolving that path?

A: No additional runtime cost is introduced. Although not entirely true; the registration is now slightly more expensive. The classes registration code that gets generated no longer generates the "absolute" resource path, but the "relative" resource path. In the example above that would be package/MyClass.gdj. The entry file now has two new properties for resolving the correct resource path called userScriptResourcePathPrefix and projectName. With these properties and with the property classRegistrarCount the registration code now knows which entry file is the "main" entry file through the propertyclassRegistrarCount. Now that it knows which one is the "main" entry file, it also knows which of the userScriptResourcePathPrefix options it should use. With that information each class can be assigned the proper resource path. So for scripts from the "main" project that would be <userScriptResourcePathPrefix>/* and for dependencies that would be <userScriptResourcePathPrefix>/dependencies/<projectName>/*. So the actually registered resource path per script is calculated during the registration of the classes and not during compile time or during "actual" runtime.

Q: There is a new @RegisteredClassMetadata annotation generated which duplicates information from the actual registration code. Why?

A: Yes there is a duplication of generated information now. This is done to be as efficient as possible. You see, KSP can read the values inside the annotations even if the code is compiled. It cannot do this for compiled code. So the values in the actual registration code inside the class registrars cannot be read. Even if it could be, we would need to do so with PSI which we dropped with the default value extraction. So yeah that would be very complex. On the other hand, the Bootstrap can only really read the registration code which we need anyway. If we would read the annotations values, we would need to use reflection which i want to avoid at all cost here. Also as said we need the registration code anyway. So we generate the new metadata annotation for the class registrars so KSP can read it for the dummy file generation and use the regular registration code for the actual registration inside Bootstrap. Hence the duplication of data.

Q: The class registrars are no longer generated with the same package hierarchy as their corresponding class declarations. Also, they now have ugly fqNames (if the fqName registration is enabled) like godot.entry.com_godot_tests_InvocationClassRegistrar. Why?

A: This is for performance reasons. KSP can only resolve classes from compiled dependencies by either fqName or package name. And as we don't know the fqName, we fall back to the highest package name we know for sure, which is godot.entry. We could now recursively resolve everything we see in ksp but this is very expensive and inefficient. Hence we make sure that every class we're interested in, resides in godot.entry. So all class registrars are generated to this package. If one has the fqName registration enabled, we could run into conflicts if classes are named the same in different packages. This now works for the classes but would result in conflicts for the class registrars as they do not have the same package name as the classes. Hence we use the actual registered name of a class as the prefix of a class registrar. So the name is assembled as: <registeredName>ClassRegistrar. Which could look nice for simpleName registration: godot.entry.InvocationClassRegistrar or ugly for fqName registration: godot.entry.com_godot_tests_InvocationClassRegistrar

Q: For what is the content inside the registration files (the gdj files) used?

A: It is completely unused. It's just data which might help the user when godot's autocompletion fails again so one can see the members of a registered kotlin class, and copy within the godot editor for use in gdScript for example. No code complexity is introduced by it as we have the data anyways at the point we generate the registration files. So it does not hurt to include this metadata for the user

What still needs to be done

Handle script creation within the godot editor
- we need to do that still
- should be a separate PR
- should create both the registration file and a kotlin file?
IDE plugin updates
Documentation updates
(at some point) support whole scenes like addons
- we can probably export them within the jar as resources and unpack them at entry generation time, but something to look at when we're actually there

utopia-rise / godot-kotlin-jvm

Generate registration files and add support for library creation #441