Add-on system

[Noordfrees]

Emerged from https://www.widelands.org/forum/topic/4843

Widelands should have an add-on system as it is found in many other games. From inside the game, it should be possible to download new maps, scenarios, starting- and win conditions, immovables, bobs, terrains, wares, workers, buildings, even tribes. And also to upload one's own creations to the server, ideally also from within the game.

This is a pretty big feature. We will need at least:

• [x] It must be possible to load new content without having to add the paths to any init.lua as this would entail merging errors.
  → dynamic tribe loading EDIT: Alternatively, we could first load the official world&tribes, and then load every add-on one after the other, and every add-on has a ~/.widelands/addons/NAME/init.lua script which is then called and may add and/or modify stuff. (Best approach IMHO)
• [x] Also a possibility to selectively enable or disable some add-ons. Especially it will probably be necessary to disable all add-ons in MP matches (or at least enforce that all clients load the same add-ons in the same order). → add-on choice will also have to be stored in savegames and replays
• [x] A main-menu network interface that allows to connect to the server, display the available add-ons, and download and save the desired ones.
• [x] Update system for add-ons to ensure compatibility
• [x] The possibility to upload stuff from the game to the server → in progress
• [x] i18n support for add-ons

See also https://github.com/widelands/widelands-metaserver/issues/71

[Noordfrees]

Just in case someone would like a sneak peek at the current state of this feature ;)

[gunchleoc]

The principal idea of addons is that first, the official units are loaded, and directly afterwards, the add-on scripts are run and modify stuff. I'm providing a lua interface for hacking into existing *Descr objects and allowing to change them. Example script: https://github.com/Noordfrees/wl_addons_server/blob/master/addons/frisians-economy-ultra.wad/init.lua

I had a quick look at the format and here are some thoughts on it:

• We already have init.lua scripts implemented for custom scenarios. The design should be 100% identical and use the same code. This will be easier for code maintenance, documentation and for learning to create an addon.
• tribes:add_custom_worker and the include statements are already obsolete in the tribes filtering branch (Pull request: https://github.com/widelands/widelands/pull/3693) - best build your work on top of that branch.
• I think addon should be replaced by addon.lua or info.lua. We could have more structured information on there in a more human readable and writable fashon, e.g. a list of dependencies and their versions. For example, one might want to create an addon scenario that depends on a tribe addon. The loading order of the dependencies will also have significance.
• category should be replaced by a list of categories, because a big addon could contain tribes, world, campaign scenarios, ...
• The internal data structure of an addon should mirror the data structure of our data folder. This way, we have 1 common specification and code for everything and only need to load stuff on top of each other, but using the same algorithm. As a second though, when we have that, we can skip the categories in the addon info file, because the primary folder names will tell us.

The rest is looking good to me concept wise :)

My branches on the subject:

1. Tribes Filtering (already up for code review) https://github.com/gunchleoc/widelands/tree/tribes-filtering
2. Move tribe-specific info out of unit init.lua files https://github.com/gunchleoc/widelands/tree/pluggable-tribes
3. Move tribe-specific info out of unit helptext.lua files https://github.com/gunchleoc/widelands/tree/pluggable-tribes-helptexts
4. We might also need to revise my design for attribute loading, but we can do that later

[Noordfrees]

For the first two points, I'd better wait then until the tribes-filtering is merged and then adapt everything to the new specs. (I have little time for reviewing large changes ATM though). Will implement the third point. It is already possible to specify any number of requirements and the game will take care of loading order, but yes it would look cleaner to have a lua array for that. We'll have to keep in mind that the server as well as the game needs to be able to parse the addons.lua though – my minimal pseudoserver implementation will have to impose some restrictions on the file until we have a real add-ons server with a full lua interface. Regarding the fourth and fifth point – then these should be several add-ons which depend on each other. IMHO splitting big stuff into several medium-sized modules is more maintainable. Also, for smaller add-ons it would be overkill to have lots of sub-sub-sub-…-directories when there are just one or two files. Here I'd prefer having one main script per add-on that takes care of including/loading/modifying everything.

[gunchleoc]

I think it would be even safer to wait for https://github.com/gunchleoc/widelands/tree/pluggable-tribes because I got rid of the init.lua file too for the tribes; I had forgotten about that. It contains info for multiple tribes at once, and I want to make everything self-contained.

The helptexts thing won't be a problem as long as you keep your examples small enough that merging/adapting won't take too long. This will already be endless fun for the Amazonians... I guess best leave that one to me then, because I'll be most familiar with the new spec.

Parsing Lua on the server is a good point. We should think about which programming language we want to use and whether it can parse Lua before you spend much time on it and we then maybe reverse our decision.

I also think that smaller add-ons are easier to maintain than bigger ones, but we don't know what type of modder we'll get, and somebody might want to make a full conversion mod or something once everything is moddable.

[gunchleoc]

it would be overkill to have lots of sub-sub-sub-…-directories I didn't express myself very well. I mean we could have something like tribes/init.lua and anything tribe-related in the tribes directory etc. It adds 1 directory level.

I have been thinking a bit more about the server implementation.

• What are the security implications? Will we need to manually review the addons?
• We could look at how other games are providing add-ons, e.g. SuperTuxKart (run their own) or 0 A.D. (use mod.io)
• We could create a C++-based binary in the Widelands repo for testing if the addon will load without error. This tool could then also generate JSON for the server, just like we do for map uploads.
• We might let the website code do the uploads, just like for maps. In that case, we should implement versioning for maps first and then reuse that code for other addons. Maps could be just another addon.

[Noordfrees]

Regarding security, I have already implemented a system for that, described here: https://github.com/Noordfrees/widelands/blob/addons/doc/sphinx/source/add-ons.rst#verification So yes manually reviewing them is something we should attempt to do (I don't expect massive usage of this system so the effort should be manageable)

There is also an add-on category "maps" which just adds one or more maps, so we could replace the website's maps section is the long run if we want. Add-ons already have versioning and an upgrade system.

For the server, I can't really comment on that as I have practically no networking experience. I hacked together a simple class that uses the CURL lib to connect to the GitHub repo with the example add-ons and download stuff from there, and I can't even imagine how a more complex approach could look like…

[Noordfrees]

@gunchleoc I have lost the overview a bit – how many branches are left to get merged before I can safely convert the add-ons feature to the new format?

[gunchleoc]

• https://github.com/widelands/widelands/pull/4153 (already approved and waiting for Travis)
• https://github.com/widelands/widelands/pull/4156 (already approved and waiting for Travis)
• https://github.com/gunchleoc/widelands/tree/unify-tribes-world + 1 follow-up to replace wl.Tribes() and wl.World() with wl.Descriptions().

Then there will be a bug fix https://github.com/gunchleoc/widelands/tree/4139-scenario-animations with changes to animation_directory and icon - I don't know if that affects you.

The rest will be getting rid of AI hints, which I think is non-critical for your project.