Resource system 2.0

[einarf]

The resource system so far has been fairly simple only referencing a path to a file. It stores the reference to the resource using path as a key. If the resource is previously loaded, we return the existing one. This however starts to become increasingly complicated as the data file we wish to load gets more complex.

A resource is not necessarily just a single path. It can be multiple paths as shown in shader loading. One file per shader type (vertex, geo, fragment.. etc). When we also add varying properties to this such as shader preprocessors, texture flipping and whatnot.. the responsibility of the resource system gets increasingly complicated.

With the introduction of pluggable loaders for all resources we would have to ask the user to provide some resource metadata class with a custom __hash__ method (and maybe even more) to suggest to the system if a resource is unique or not. This is just too much. The resource loading system should have one responsibility: To load the requested resource. It should not be in the business to caching resources internally. We don't know how our users intend to use the system.

EDIT: This also questions if the resource system should support deferred loading. It might still be reasonable to return a (meta, resource) tuple list when loading a pool.

Why we should still have some kind of registry

Effects and classes such as the TextWriter load resources on initialization. Unless we ask people to supply these resources on creation or somehow pre-load these resources on module import, we would load new resources for each instance.

Another problem is how a single effect can run with runeffect compared to issuing the run command. An effect package might also need a way to separately define its resources. This also needs to work for standalone classes such as the TextWriter since these are not actual effects.

The need for some kind of resource registry is still there.

[einarf]

Project

Since the responsibility of keeping track of resource has been removed from the resource system, this needs to happen elsewhere. The idea is to introduce a project.py module in the root of the project that will do this job.

By default this is referenced in settings:

PROJECT = 'path.to.project.module'

A newly created project structure would then look like this:

myproject
├── settings.py
├── project.py

The project module will be responsible for:

• Deciding what resources should be loaded for the project and assign labels to these resources. This means effect are no longer directly responsible for loading resources. The project will do this and provide a way for effects to fetch resources using a label.
• Create effect instances and assign labels to these. These labels are then used in a separate system for dealing with the project timeline.

Users should be a lot more free to do whatever they want in the project module. They can even assign resources to the effect instance they created directly or through the effects initializer.

This also means the EffectManager class dies. It used to be responsible for creating effect instances. We replace this with a pluggable Timeline system that is only responsible for knowing what effect instance should be drawn at what point in time.

Effect package

And idea so far has been to introduce a resources.py module. This contains resource loading code for this package specifically. When registering the package in settings.Effects these resources are included in the project automatically.

The cubes effect package.

cubes
├── effects.py
├── resources.py
├── <local resource dirs>

This way an effect package is also able to act as standalone unit loading its own resources providing runnable effects thought runeffect. When using the run command the users project will receive the resource list from the the effect packages and chose to use or discard them. When runeffect is issued we create a Project class that just accepts these resources.

We might also need to introduce a runnable flag for effects so the runeffect command knows what to do. This way the author of the package can provide an "entrypoint" to a runnable example. If an effect package depends on effects from other packages we cannot guarantee that runeffect will be successful. For that scenario you need to run a project with run and provide those packages in settings.EFFECTS.

[einarf]

We supply simple classes for describing a resource. This is just a container for arbitrary attributes with a minimal set of required parameters.

Ugly example:

class Texture:
    def __init__(self, label=None, loader='2d', **kwargs):
        self.label = label
        self.loader = loader
        self.kwargs = kwargs

        if not self.label:
            raise ValueError("blah")

Resource lists can then be built with these objects:

resources = [
    Texture(path='wood.jpg', label='wood'),
    Texture(path='tiles.png', label='tiles', loader='array', layers=10),

    Shader(path='cube.glsl', label='cube'),
    Shader(vertex_shader='stuff.vert', fragment_shader='stuff.frag', label='stuff'),
]

Then fetched in effect initializers:

class MyEffect(Effect):
    def __init__(self):
        self.texture = self.get_texture('wood')
        self.shader = self.get_shader('stuff')

The resources.py file could just be a simple list like that. A Project would be forwarded the list from effect packages and handle that as they wish.

This does mean that the resource system could still support deferred loading by collecting these meta objects, but it will do so blindly by consuming a list. The Project class is responsible for the final outcome.

It should still be possible for classes outside the effect system to register resources such as the TextWriter and DeferredRenderer. These were deliberately created this way to make things flexible.

[einarf]

Done in the 2.0 branch