Library for runtime discovery and binding of mod-dylibs

martinrlilja commented 7 years ago

Part of the path towards mod support. This will involve making a library to support dynamic loading of mods, and some other things outlined below. This are just my general plan, suggestions, help, etc, is of course welcome. I will update this as the implementation comes along.

Packages

Should be a directory containing a toml configuration file. We need a name for this file, probably something like cbmod.toml or just mod.toml. It should have the file extension .toml so that there are no issues with syntax highlighting, default programs, etc. The file could probably imitate Cargo.toml:

[mod]
name = "more-beautification"
pretty-name = "More Beautification"
version = "0.1.0"

[dependencies]
trees = { version = "*", optional = true }
options-manager = "0.6.2"

Questions: Name and pretty name? Archives like zip? Preview images?

Mod interface

Mods will be registered using a macro, so that we can easily hide any implementation details, have less breakage and avoid some hard to track down errors. (Symbol not found, data corruption, etc.)

register_mod! {
    cb_mod: MoreBeautification,
    "1.0" -> "1.1" => (system: &mut ActorSystem) {
        // do stuff to the system
    },
}

We can also add syntax for update strategies, for example. Here I think it's fine to limit so that you can only register one mod per crate.

Each mod will have to implement the CityboundMod trait which has functions for initialisation and state store/restore. We could add functions here for letting the mod handle option menus or something else.

pub trait CityboundMod {
    fn setup() -> Self;
    // generate options menu?
}

Questions: The mods need to hook into something, is kay a good target?

Interface for loading mods

todo

Dynamic software updating

If someone is feeling really adventurous, this might be something to look into. The wikipedia page is probably a good start (https://en.wikipedia.org/wiki/Dynamic_software_updating). There also seems to be a lot of papers on the subject.

aeplay commented 7 years ago

Some comments:

Packages:

I didn't understand the use-case of register in the manifest file
What do you mean by Compression?

Mod interface:

as for hooking in - I would like the mod loading topic to be on-top of kay, not integrated inside it
- it makes sense to pass a reference to the actor system singleton in init, so the mod can register actors and messages (see for example here)
how would you deal with the State type of a new version being different from the State type of the old version (containing new/less fields...) ?
what about state that is stored in actors?

martinrlilja commented 7 years ago

The point of register is to be able to specify some data which will be sent to a dependency whenever the mod is loaded. Say we have a collection of tree models, these could have a register field under a foliage manager, register = { type = "tree", color = "blue", zone = "tropical" }. I don't know how useful it actually is, but it means that the mod creator won't have to write any code to send this data to the foliage manager.

By compression I mean support for putting the mod in a zip file, or something similar, for distribution. Maybe it could be nice to have support for loading those directly without unzipping first. This is similar to how browsers do it with extensions. This is of course low priority.

As for the state, do you think it would be better to store all of the state in actors? And then maybe have kay keep track of which mod created which actor, so that we can hand of this to the mod on restore? I'm very unsure what's the best way to tackle this, but I guess it would be closely related to the saving mechanism. Maybe we could use (or abuse) serde here to keep track of the types of everything? This would have the added benefit of throwing errors if the type representation changes.

aeplay commented 7 years ago

I would suggest to first not have this register feature, thus be forced to manually register with code and see which common patterns emerge. I don't like to "guess" what will be needed there.

Makes sense! I think .zip would be the most accessible (to look into)

Yes, I would prefer to keep all state in actors - and the mod should know itself which actors it created. Just to make you already aware: there won't really be a saving mechanism, just the (already compact) representation of actors in memory will be pages to disk because it will live in a mmap'ed file. So the problem is really how to get, in a paused state, from living actors of the old version to living actors of the new version. I consider this a really hard problem as well - but if we find a solution, we have an infrastructure for ensuring save game compatibility for the main game as well. Inspiration from the masters could be how Erlang handles module or even whole app updates (even while the system is running!) - here is a pretty in-depth tutorial/example

aeplay commented 7 years ago

Note: Erlang has of course the benefit of having dynamic types, making it easier to deal with old/new record definitions. All that I can imagine for rust is that the "transition helper" imports the Type definitions from both old and new mod version (under different aliases) and then does the translation (for example field-by-field) - but it would necessarily "live between versions"

martinrlilja commented 7 years ago

@aeickhoff Unless I'm mistaken, the DUS in Erlang is not safe, and relies on defensive programming in order to not corrupt data, as you said, a dynamic type system is very useful in this case. kitsune on the other hand, a DUS for C, solves the problem by having the programmer write an upgrade path in their own DSL. There is also a lot of interesting stuff going on with upgrading functions. Sadly, many of the already established methods seem to target the specific use case of security updates on running systems.

So I'm thinking that we have to specific use cases:

Hot loading - We don't really care about losing our progress, we only want fast iterations, if something breaks we can just reload more stuff. Although, this has the very important caveat that it must be functionally identical to loading the mod on startup. Code breaking when you restart the game would be pretty crappy.
Mods - Kind of like the Linux kernel is not allowed to break user space, we shouldn't, ever, break a save file. We should probably take a very defensive route here, checkpoints whenever a mod is updated, if we detect any inconsistency in data, we should try to stick to an older version and inform the user.

In order to verify the data, I think we need to create a fingerprint, or just have some metadata about the types in the saves. serde could probably be useful here because of its macros. We could probably abuse it to generate such data.

martinrlilja commented 7 years ago

I've been reading a lot of code from rustc, trying to figure out how to get a hold of dependencies straight from the dylib. But I don't think it's currently possible to do what I want to do, because the decoder for metadata is unfortunately private.

A more general method would be to simply use our own format for dependencies (and maybe a secondary for generated dependencies if the need arises). Doing so would also completely remove the dependency on rustc in weaver.

A note about loading metadata from dylibs and rlibs

I haven't tried this, but I want to write down my finds so far for future reference.

Initialize a rustc_metadata::locator::Context, most fields can be left empty.
Call maybe_load_library_crate on this struct.
You should now have a Library, this has a field of the type MetadataBlob.
MetadataBlob implements rustc_metadata::decoder::Metadata, so that it can be used to read all sorts of interesting information about the library.
Find an interface that actually exposes this information.

aeplay commented 7 years ago

I would strongly prefer custom metadata for dependencies if it allows us to avoid a dependency on rustc!

citybound / citybound