Asset loading/storage and management system investigation.

[agkountis]

Task

Investigate various designs for a asset loading/storage system for crius.

Details

• The system will have to be used as a resource inserted into the main ECS world structure so that the user can retrieve it and use it from within scenes.
• Investigate possiblities for asynchronus loading using tokio or async-std.
• Investigate possibilities for asset streaming.
• Have a look at https://github.com/amethyst/atelier-assets , https://torkleyy.github.io/blog/amethyst-assets/ for getting some ideas

Document possible design ideas in this issue as comments or start a discussion on gitter or discord

[mitso23]

After a bit of a research online and having a quick look on the amethyst.

Asset definition

• Is an asset a combination of primitive resources like, Texture, Sound, Animation, Mesh
• For example a character asset could be a 3DMesh combined with some texture applied to it ?
• If that is the case as a first building block we could implement a resource manager that manages raw resources

Defining the file format for the assets

• We need to document what kind of metadata needs to be stored along the asset itself
  • Asset type
  • Location
    • URL
    • FileSystem Path
  • Asset Size
  • Priority
  • UUID
  • Dependencies
• I think @agkountis mentioned that we could use flatbuffers

Asset Storage/Look up

• Individual Assets should be stored in a container that has a contiguous memory layout
• Indexing should be done either by using a UUID or some sort of descriptive text that can be converted to int at compile time like hash("texture-1.jpeg")
• The fastest lookup can be achieved using an unorder map
  • I think rust implements open address hashing which is very efficient for collision resolution
• So the underlying storage could look like this std::unorder_map<UUID, RefCount<StorageType>>
• Do we need to implement some sort of memory allocator for efficiently managing the underlying memory used by the assets ? Could be done at a second stage maybe

Assets life cycle

• I guess each game will have all the required assets stored in some sort of lightweight database
• How do we decided which assets we pre-load before the game starts for example to speed up the game experience?
• The asset manager should own the underlying memory. When a user requests an asset the assent manager will either return immediately the resource (if this has been already fetched) or schedule a download and notify the user via a callback asynchronously.
  • The main render thread shouldn't be blocked while waiting for the resource to complete ?
  • The asset manager should return a weak pointer to the client requesting the resource
  • if the reference count of the asset drops to 1 then the asset manager can decide to free this resource and load another one.

Caching

• Most recently used resources should be cached in memory
• Need to define on asset replacement policy when max resources are used (similar technique to paging)

Asset loader / Change monitor

• Need to be able to load assets from Disk asynchronously
• Need to be able to load assets over the Network
• Can we start using an asset before it is fully available @agkountis
• Reload assets if the underlying file has changed

Asset dependency management

• we would need to maintain a graph of all the dependecies of all the assets
• If asset A depends on asset B and B has changed, we would need to reload both assets @agkountis

[agkountis]

@mitso23 First of all thank you for investigating the matter.

Now to address some of your points and present my views:

Asset Definition

An asset is a "resource" that can be loaded to be consumed by the engine. E.g:

• a Mesh
• an Animation
• a Model
• a Texture
• a Shader
• an Engine material file (a file that describes a material, if we chose to design that way)
• an Audio file
• etc.

Notice how I am not mentioning configuration files as I think these should be a separate category because their use is mostly singular and that is for initialization and no other systems usually consume them at runtime.

Defining the file format for the assets

Flatbuffers

First let's talk file format. I suggested flatbuffers from google for various reasons.

First I have experience with the format because I am currently using it at work to design a custom file format for the Unity application I am working on. Flatbuffers demonstrate significant increases at deserialization times.

Flatbuffers also allow for separation between the schema definition of the file format and the language of implementation, since it uses a custom DSL description language that is used to generate bindings for multiple languages, one of which is Rust. This makes flatbuffers ideal because there is no need to recompile the code bases for each change on the file format schema.

Flatbuffers is also easy to extend without causing breakages since it follows a deprecation model where you are not allow to completely remove an entry to the schema, but only deprecate it. Additions to the schema are allowed.

Metadata

As far as metadata go we will most likely figure out what is needed when we start experimenting loading them in the engine. But most of the ones you mentioned would make sense but we have to discuss the usefulness of each one of those.

Another point of discussion would be if these metadata should be stored in a separate file or as a header in the asset itself. Both of those have advantages and disadvantages. The problem I see with storing them in a header is that flatbuffers don't seem to have the ability to read "part" of the buffer. They load the entirety (and it is really fast).

Compression

Asset files should be compressed using LZMA for maximum compression and transmission over the network and LZ4 for local storage for fast decompression speeds.

Asset Lifetime

Keep in mind we are not developing a game, but an engine. Crius only provides tools to the developer to manage assets to be used in various engine systems. Asset loading should be managed by each scene the user application has.

• The asset manager should own the underlying memory. When a user requests an asset the assent manager will either return immediately the resource (if this has been already fetched) or schedule a download and notify the user via a callback asynchronously.
  • The main render thread shouldn't be blocked while waiting for the resource to complete ?
  • The asset manager should return a weak pointer to the client requesting the resource if the reference count of the asset drops to 1 then the asset manager can decide to free this resource and load another one.

I agree with the above but we have to consider the following:

Mutability

Assets should most likely be immutable pieces of data. If the user needs to mutate an existing asset they will most likely have to pay the cost of a clone.

RefCounting

You mention reference counting. Reference counting in Rust is done using Rc<T> for single threaded scenarios and Arc<T> for multi-threaded scenarios. Both those types are immutable by default, meaning you cannot mutate the contained type T. If you want to mutate you have to use interior mutability using RefCell<T> so the types would look like Rc<RefCell<T>> which isn't an easy to use API.

If we restrict the lifetime of assets to a per-scene basis we can practically hand out simple immutable references to the assets.

• If assets need to be mutated the assets should implement the Clone trait.
• If assets need to persist to the next scene. The AssetManager should give the user the option to flag the asset and keep it alive during the transition.
• Assets should stay alive for the entire lifetime of a Scene unless the user of the engine instructs the AssetManager otherwise.

Caching

Shouldn't that be a responsibility of the user application to manage which assets they want present in the Asset Manager at any point in their scenes?

For now I don't know if setting limits in asset loading makes sense. The user application should be sensible in their data requirements based on the systems they will be targeting.

Asset loader / Change monitor

• Need to be able to load assets from Disk asynchronously
• Need to be able to load assets over the Network
• Can we start using an asset before it is fully available @agkountis
• Reload assets if the underlying file has changed

I agree will all the above.

• Can we start using an asset before it is fully available @agkountis

This sounds like asset streaming. The only cases I can think of are:

• Texture streaming
• Sound streaming

So yes streaming is definitely a possibility, but let's start simple.

Asset Dependency Management

Can you elaborate a bit more on having assets depending to each other? The only one I can think of right now is maybe Materials referencing Shaders?

[mitso23]

@agkountis thanks for the comments. I guess we can ignore the asset dependency for the time being. We can always add it later on.

So the list of the assets to be downloaded will be part of the configuration file that is going to be YAML format ?

[agkountis]

@mitso23 I don't think asset downloading is in the current scope.

Even if it was, I believe download logic should be handled by the user application.

If the user wants to download assets from a remote source, the engine should provide some network abstraction at most (still out of the current scope).