Systems

[randy408]

My thoughts so far on how systems should work:

• ecs.system() wraps the flecs function like the existing APIs we have so far

• There has to be a dummy ecs_lua_entrypoint(ecs_iter_t *it) between the VM and flecs

• When the system is called from flecs it pushes each "argument" as an array of components, then copies it backs into flecs when it's finished

• flecs-lua probably shouldn't "own" the lua_State, so it has be kept in mind that the VM can persist across frames and:

• if a system has a Init() we definitely have to persist that VM until DeInit() anyway

Some things that are not so obvious:

1. If there are multiple systems registered from a state, how do we figure out which Lua system is supposed to run?

2. Do we need a lua_State per system?

3. Can we make multithreading work?

4. How does it interact with host scripts that aren't modules/systems?

5. Can we still make "simple" scripts work where sleep() is possible?

6. Reloading script/systems is kinda important, what changes in flecs-lua would that require?

[mo7sen]

What I did so far:

• ecs.system() creates a flecs system that executes the dummy function
• Lua systems are recorded in the Lua Registry and are retrieved by the dummy function to call these functions
• Each LuaSystem's index in the registry is passed to the system through it->param

This kind of guarantees that a LuaSystem imitates a flecs system as much as it could. However, I'm currently trying to find ways to:

• pass an array to Lua without needing to literally copy each element on every system call. Not sure if there is even a way to do that.
• give ecs_lua_entrypoint(..) access to lua_State without making it a global. I tried adding it as a component to each system but this seems to introduce the overhead of calling ecs_get(world, system, LuaState) on each call to the system

  ---

Regarding multithreading, I don't think Lua has support for multithreading. As far as I know, we'd need to have multiple lua_States for this to be achievable

[randy408]

Passing arrays without copying is premature optimization, I think the bigger problem is that if flecs-lua manages all the de/serialization everything will be a dumb table. It has to be aware which components need special metatables, e.g. vec3 + vec3 or scalar * vec3 should work as expected.

If we solve that efficiently we could make the columns special arrays, initially we could push the elements as-is on access and read back the pushed elements at the end.

There are many ways to do it which depend on the use case, which is why I don't want to get into it at this point, we could check for [in] for now.

The API needs some changes for multithreading, I'll have something ready soon.

[mo7sen]

A weird question. Would it be a bit dumb if we deviated a bit from the standard flecs API? What I have in mind is for the systems to work on each entity individually rather than on entire arrays. So the ecs_lua_entrypoint(..) would be something like this:

void ecs_lua_entrypoint(ecs_iter_t *it)
{
  for(int i = 0; i < it->count; ++i)
  {
    lua_rawgeti(L, LUA_REGISTRYINDEX, (long)(it->param));
    // TODO: Pass stuff
    lua_pcall(L, 0, 0, 0);
  }
}

instead of this:

void ecs_lua_entrypoint(ecs_iter_t *it)
{
  lua_rawgeti(L, LUA_REGISTRYINDEX, (long)(it->param));
  // TODO: Pass stuff
  lua_pcall(L, 0, 0, 0);
}

This way the lua system will be something like:

function move(position, velocity)
  position.x = position.x + velocity.x
  position.y = position.y + velocity.y
  return position, velocity
end

instead of having to loop in Lua itself.

[randy408]

To make multithreading work flecs-lua could create states per-system and issue a callback for the host to de/initialize the state.

The registry would look like this:

• Main state

  • ecs_lua - was esc_world, now a userdata of ecx_lua_ctx which now also holds the callback func pointers
  • ecs_lua_systems - lightuserdata array of ecs_lua_system's, we only have access to the main state on deinit so these structs would contain the pointer to the lua_State to be cleaned up on ecs_lua_exit()

• System states

  • ecs_lua - same as in the main state, but as lightuserdata
  • ecs_lua_system - full userdata struct

• ecs_lua_init() doesn't change much aside from taking a ecs_lua_ctx* with more arguments

• ecs_lua_exit() takes the main state and deinits all systems listed in ecs_lua_systems

ecs.system() CFunction:

• Call host for a new lua_State, initialize ecs inside it
• Allocate a ecs_lua_system struct as full userdata, store it in the registry keyed with ecs_lua_system
• Append to ecs_lua_systems
• it->param is the ecs_lua_system*

I'm looking into the module API and how it could interact with scripts.

[randy408]

What I have in mind is for the systems to work on each entity individually rather than on entire arrays.

There are situations where the opposite is preferable or even required, e.g. individual loops over with 2 components out of many one after the other, it's also possible to make API calls inside systems and create additional loops. Because this restricts access to just one cell of each column on each iteration it can't be made the default behavior.

[mo7sen]

What I have in mind is for the systems to work on each entity individually rather than on entire arrays.

There are situations where the opposite is preferable or even required, e.g. individual loops over with 2 components out of many one after the other, it's also possible to make API calls inside systems and create additional loops. Because this restricts access to just one cell of each column on each iteration it can't be made the default behavior.

Yeah, makes sense. I was just thinking about how that would give a new level of parallelism in the systems, but it does seem to be quite limiting.

[randy408]

This is a solved problem at this point. multithreading is tracked by #3.