Externally defined systems and scripting

torkleyy commented 6 years ago

Opening an issue because I want to get shred & Specs ready for scripting!

I'm currently working on named resources, there'll be a PR pretty soon. Then we can already write systems in Rust and compile them into a shared library. Next step will probably be exposing the Specs API via C.

cc @Moxinilian @jojolepro

I'm going to work on the PR and link it here once I'm done with it. In case you have any suggestions / want to help please comment ;)

torkleyy commented 6 years ago

This takes a lot less time than a big framework like vnodes, plus in case you're still planning on that this change is fully compatible with that.

Moxinilian commented 6 years ago

Is hot reloading relevant to this issue?

Regarding externally defined components and resources, the design I had in mind (with my lackluster knowledge of the internals of specs) was to declare components, resources and systems as a schema that would simply be codegened into Rust. It would be a different story for hot reloading, but I'm not sure it is relevant here.

Also we're not planning on using vnodes in Amethyst anymore as the overhead it introduces, as small as it is, would be too high to compete performance-wise. We have another design in mind that relies on direct FFI which should have the same advantages.

(cc @kabergstrom)

torkleyy commented 6 years ago

Is hot reloading relevant to this issue?

In that you need to separate into a dynamic library and that only works with this change, yes.

torkleyy commented 6 years ago

declare components, resources and systems as a schema that would simply be codegened into Rust. It would be a different story for hot reloading, but I'm not sure it is relevant here.

Yeah, that's just the same static approach we already have. Hot reloading doesn't work with that (without implementing what I do right now externally).

Moxinilian commented 6 years ago

Okay. Then regarding hot reloading of components, resources and systems, here is what I had in mind:

Systems: Assuming we have something to rebuild dispatchers on the fly (which is not a hard problem), it seems specs already has most of the things needed for that, if I correctly understood what DynamicSystemData is.
Components: Here comes the tricky part.
- Creating, deleting and moving already defined components around is trivial already.
- Creating a new component type at runtime could be handled by having a special non-generic storage type that would store Vec<u8>. Those Vec<u8> would contain the raw data of the new components. Then, those components would be exposed to the scripting runtime through newly generated C headers corresponding to the component schema the user provided. That however will mean that TypeId can no longer be the only thing determining ResourceId, as those storages would share the same type but a different ID. But it does not seem unreasonable to have ResourceId become a (TypeId, InstanceId), at least from my uneducated point of view. Also, at the end of the execution, the dynamic component should be collapsed into a statically generated component to no longer need to store the data in a Vec<u8>. The typical wokflow would be: create your components on the fly, do your testing with it, and if it's too slow then reboot the game. It seems reasonable to me, and besides I don't really expect it to be that slow, especially if you're only using it for logic creation.
- Modifying existing component types on the fly would only work on component types that were created from the scripting environment, but I think it's a reasonable constraint. Modifying a component in dynamic state would simply be a matter of iterating over the Vec<u8> storage and moving the data inside around. Modifying a component in statically-collapsed state however is a bit trickier, but not that hard either. You need to replace all static instances of that component into a dynamic instance of the new component.
- Removing an existing component type at least is easy. Just remove all instances of it and leave it all alone, scared, in a white void up to the next reboot where you end its sufferings.
Resources: Resources can be implemented in a similar fashion as component storages are. I don't think there is much more to say about them.

Most of the aspects I mention above do not have to be handled by specs. This is the design I envision for Amethyst, so everything from schema to static collapsing would be handled by external tooling in the engine.

torkleyy commented 6 years ago

I think we're mostly d'accord here. What I'm currently adding is just a way to add e.g. multiple MaskedStorage<ScriptingComponent> to the resources by allowing an identifier next to the type for the purpose of resource identification.

Moxinilian commented 6 years ago

Ah that's fantastic then. I can't wait to see that in use, not only for scripting.

torkleyy commented 6 years ago

Update: the current SystemData / DynamicSystemData / Accessor / StaticAccessor etc. stuff is already too complex. I want a simpler architecture, and the named resources addition I planned to propose today makes things worse. Thus, I'm changing my priority to creating a simpler, more consistent and more extendable API, also as a result of today's chat on #rust-gamedev.

OvermindDL1 commented 6 years ago

What vnode overhead? It should only have the cost on initial access and loading time, once you store the reified node locally then all access is direct function calls without indirection or virtualization. I've never had an issue with performance about that?

torkleyy commented 5 years ago

Scripting compatibility of Specs is hard to do, and even harder to get right due to many type-safe interfaces. I don't think this is something we can provide in Specs 1.0 and I don't know if Specs can or should do this at all.

I'm closing this for now. If we come up with a solution for this, I don't think it can be in the stable Specs core.

jchitel commented 5 years ago

So there are currently no plans to enable any kind of modding for a game built with specs? Isn't that one of the major benefits of an ECS architecture, that components and systems can be plugged into the engine without impacting existing code?

If I did want to do something like this, how might I proceed?

OvermindDL1 commented 5 years ago

If I did want to do something like this, how might I proceed?

A usual inefficient fallback is make a new component that holds a map of names<->generic_script_data and let the scripting system process over that using whatever language it is.

jchitel commented 5 years ago

I had considered that option, but I was hoping for a way to allow the insertion of native component storages at runtime.

The issue is that the World API depends on monomorphization, which is all resolved at compile time. However, it's just a hash map, so it could just take runtime-defined resource IDs. There's probably something I'm not thinking of, but I was assuming that everything the script might add will only be accessed by that script, so it doesn't necessarily need to be strongly typed.

AnneKitsune commented 5 years ago

We will still have scripting and system support. This issue just mean that it won't be managed by the specs library, but rather by amethyst itself.

jchitel commented 5 years ago

I see. Is there a tracking issue for that or is it still being discussed?

AnneKitsune commented 5 years ago

https://github.com/amethyst/rfcs/pull/1

jchitel commented 5 years ago

Beautiful, thanks!

torkleyy commented 5 years ago

Support for scripting will be worked on for nitric, I've mainly closed this in order to make progress in stabilization.

amethyst / specs

Externally defined systems and scripting #462