[FEATURE] Job-System

[genaray]

Unitys new ECS ( part of DOTS ) offers a intergration of Unitys Job-Systems ( https://docs.unity3d.com/Manual/JobSystem.html ), a high performance multithreaded task execution mechanic... there actually multiple ECS out there offering this mechanic.

This feature is perfect for multithreading little isolated units of work that may execute heavy operations. For example multiple little "Jobs" lifting of heavy collision calculation into multiple threads, while the next, chained "Job" uses those calculations to notify the involved entities about their collisions... there plenty of use-cases !

I actually did some research and tried to implement my "own" Job-System, completly depending on entities... ( Remove classes that arent available & events ) https://pastebin.com/0vD5sDTK

It already works, but im not a professional, so someone should take another look at it, polish & very it... furthermore theres still a important feature missing, chaining of jobs from different systems... For example all Jobs from System A should only depend on Jobs from System B... also im not that sure if its "clean" to execute jobs prompty ( onInserted ) instead of waiting till the next frame... this makes us lose flexibility

Edit by Daan van Yperen: Fixed the links

[genaray]

So i did some more work and created a pretty good job system for artemis in java :)

It features single jobs aswell as dependencys between multiple jobs... A newly created Job-Entity gets enlisted into the multithreaded environment as soon as possible :)

I also implemented a interface systems to schedule Jobs from different systems after another. Great if all jobs from System B depend on the finished jobs from System A.

Currently theres no "synchronisation" point, once a job started to run, it runs completly parallel to the mainthread till its finished... some major JobSystems ( like the one in Unity ) offer synchronisation points where we wait for jobs being finished. Often used with movements, collisions or physics.

Each Job-Entity simply requires a job component, and if desired... a job dependency. Hope you enjoy it so far ! I needed several attempts to find a clean and easy solution ;)

Heres a little example and my source code so far :)

public class ExampleJob implements IJob {

    @Override
    public void accept(Integer id) {
       // Do some heavy calculations...
       // You could also modify the entity after the calculations ( be aware of synchronisation/locking ! )
    }
}

@All(YourClass.class)
public class SystemA extends IteratingSystem implements IJobSystem {

    private IntBag scheduledJobs = new IntBag();

    @Override
    protected void process(int i) {}

    @Override
    protected void end() {
        super.end();

        // Creat job somewhere... i just picked the end
        int jobEntity = world.create();

        Job job = new Job(new ExampleJob());
        JobDependency jobDependency = new JobDependency(jobEntity);

        world.edit(jobEntity).add(job);
        world.edit(jobEntity).add(jobDependency);

        // Add job to bag for being visible to other systems
        scheduledJobs.add(jobEntity);
    }

    @Override
    public ComponentMapper<JobDependency> getJobDependencyComponentMapper() { return world.getMapper(JobDependency.class); }

    @Override
    public IntBag getScheduledJobs() { return scheduledJobs; 
}

@All(YourClass.class)
public class SystemB extends IteratingSystem implements IJobSystem {

    private IntBag scheduledJobs = new IntBag();
    private IJobSystem systemA;

    @Override
    protected void process(int i) {}

    @Override
    protected void end() {
        super.end();

        systemA = world.getSystem(SystemA.class);

        // Creat job somewhere... i just picked the end
        int jobEntity = world.create();

        Job job = new Job(new ExampleJob());
        JobDependency jobDependency = new JobDependency(jobEntity);
        systemA.schedule(jobDependency); // Makes all jobs from this system run after those from system B

        world.edit(jobEntity).add(job);
        world.edit(jobEntity).add(jobDependency);

        // Add job to bag for being visible to other systems
        scheduledJobs.add(jobEntity);
    }

    @Override
    public ComponentMapper<JobDependency> getJobDependencyComponentMapper() { return world.getMapper(JobDependency.class); }

    @Override
    public IntBag getScheduledJobs() { return scheduledJobs; 
}

Link to source-code

[genaray]

So i continued my work on the JobSystem and noticed that i missed something. The job system currently has a pretty broad usecase, you can do everything you want in that IJob interface... but if you really want to iterate over a set of entities in order to modify them on another thread, its probably too much boilerplate-code.

Since java does not support unlimited generics i came along this solution...

/**
 * A job which is iterating over a set of {@link com.artemis.Entity}'s in order to process them in another thread.
 */
public interface IJobForEach extends IJob, BiConsumer<Integer, IntBag> {

    default void accept(Integer integer){ this.accept(integer, getEntities()); }

    @Override
    default void accept(Integer integer, IntBag intBag) {

        begin();
        for(int index = 0; index < intBag.size(); index++) process(intBag.get(index));
        end();
    }

    /**
     * Gets called before this job started to process its entities.
     */
    default void begin(){}

    /**
     * Gets called on every {@link com.artemis.Entity} this job processes-
     * @param entityID The entity this job is currently processing
     */
    void process(int entityID);

    /**
     * Gets called after the job processed its entities.
     */
    default void end(){}

    /**
     * Should returns a {@link IntBag} of entities this job is processing
     * @return Should returns a {@link IntBag} of entities this job is processing... a copy would be better in order to prevent modification exceptions
     */
    IntBag getEntities();
}

A little interface that reduces the amount of boilerplate code required to iterate over a set of entities :) Its not perfect, but i didnt found a better solution... actually its pretty similar to the way unity is doing it ;)

Heres an example...

        Job job1 = new Job(new IJobForEach() {

            @Override
            public void begin() {
                // Prepare stuff ? log ? whatever ( not required... because of default )
            }

            @Override
            public void process(int entityID) {
                // Process entities and modificate them
               Velocity velocity = velocityMapper.get(entityID);
               velocity.speed += 0.01f;
            }

            @Override
            public void end() {
                // Do something else... loging ?  ( not required... because of default )
            }

            @Override
            public IntBag getEntities() {
                return this.getEntities(); // return copy of a certain set of entities... would be better to copy them in the constructor on the mainthread.
            }
        });

[DaanVanYperen]

Most of the API is not thread safe (Like Bags and component mappers) so can't be safely called inside a job. You'd have to feed it copies of the data it needs during creation for example, and merge the job results synchronously with the world thread, during system processing.

Beyond that, makes a lot of sense having a mechanism to dispatch asynchronous jobs in a game, just wonder if it is out of scope for the ECS itself, if you can't really use any of the ECS features in the job. Would depend a lot on the use case, if you need to serialize pending jobs for example. Certainly would fit a plugin.

[genaray]

Thats actually something i need to rework, jobs shouldnt be entities. I think im gonna implement this in the typical unity style where its much easier to deploy jobs.

The thread safety is a real problem, creating copies would cause too much garbage. Unity solves this with sync points, for example a system can spawn in multiple jobs, but need to finish before the system ends. This way we could speed up or parallelize the work.

But this is a huge topic... im currently only using it to dispatch database operations easily. In the future i probably gonna let it run behaviour trees and similar stuff.