Multi-stage asset loading

[ndarilek]

What problem does this solve or what need does it fill?

Often, the result of an asset load that I'd like to pass around isn't just raw bytes, or some struct created by another library. For instance, I have a plugin that loads byes from the array and does some calculation--determines how many channels an audio file contains. My plugin then creates some sort of audio context, which in turn will load the bytes and create a buffer which my systems will later use directly.

Right now, my asset system loads the bytes, does any processing, and marks the asset as loaded. Then, a system listens to events, using the context resource and modifying the created struct with the type introduced by the context.

Unfortunately, this doesn't interact well with the asset loading system. Assets are marked as loaded, even though the real data I want hasn't been created yet. The startup system rushes to instantiate everything, and usually succeeds. But sometimes it fails, and while I can avoid a crash, I can't access the asset in the way I'd like. There's no easy way to tell whether an entire group of assets is fully loaded, in the same way I can with regular asset loaders.

What usually ends up happening is that the bytes of my game's background audio fully load, and the asset is marked complete. My startup OpenAL system detects the new asset and creates an OpenAL buffer. This almost always succeeds before the background audio starts playing, but occasionally it doesn't, and the game starts but my background audio doesn't play. I'm currently rolling my own custom Web Audio backend, and I expect this problem to be even worse given the general latency of loading things from the web.

What solution would you like?

Two possibilities:

1. Make resources available to asset loaders somehow, so the loader can instantiate the asset in its final desired form. Alternately, create a different asset loader that has access to resources.
2. Create some sort of partial-loaded state which the asset loader can use to signal that it's done with the asset, but which doesn't report that the asset is fully loaded when groups are queried. Add an event for partial loads, and let systems indicate that they've either partially or completely finished an asset load in a way that integrates with the current load checks. Then the loader marks the asset as partially loaded, and systems can detect PartialLoad events and either finish the load or perform another step and re-emit the event to even more systems for processing.

What alternative(s) have you considered?

I imagine I'll have to emit a custom event when I've loaded all my assets, but this further complicates the already non-straight-forward asset-loading process by requiring me to manually track when my custom processing step finishes. Given that I'm doing this in a plugin, whatever mechanism I create will be specific to that plugin. I'm open to other possibilities that I may be missing.

Thanks.

[alice-i-cecile]

Does #1665 solve your use case?

[ndarilek]

I haven't looked at the entire PR yet--super swamped this week.

Immediately, though, I'd note that asset-loading and switching stages is already somewhat unintuitive. I agree that a non-blocking API is essential, but for smaller games it seems like you just want to load everything and have immediate access, which the current API doesn't make as easy as it might.

But right now, I can at least run a map over a Vec of assets and ensure that they're all loaded. The PR looks like it'd make me catch the initial load completion, then somehow construct a new Vec of assets that were dynamically added by the new creation systems and check those for completion.

In my case, it would not make sense to mark anything in this pipeline as complete before resources are instantiated, because there is no use for the partial asset. Being able to say that they're partially loaded, then kick completion down the road to a new system, seems like it wouldn't complicate things in the same way that getting completions for useless assets would.

Maybe I'm misunderstanding the PR, but it seems like I'd be getting two completions, one for incomplete assets, and others for assets my startup systems wouldn't even know to track. Please correct me if I'm wrong. :)

Thanks.

[alice-i-cecile]

Those are good questions, and beyond what I'm fully confident in my answers for. Tagging in @mockersf <3

[mockersf]

The PR looks like it'd make me catch the initial load completion, then somehow construct a new Vec of assets that were dynamically added by the new creation systems and check those for completion.

You wouldn't need to keep the initial handles and wait on them.

Roughly and not checking it would compile exactly like that (probably need more typing), you could do:

let my_handles = ["file1.ext", "file2.ext", "file3.ext"].iter()
    .map(|file| asset_server.load(file))
    .map(|handle| asset_server.create_from(handle, my_asset_transform_fn))
    .collect::<Vec<_>>();

Then you can wait on that vec of handles of transformed assets. The asset_server.create_from method takes care for you of waiting for the initial handle to be ready before actually creating the new asset from it.

[ndarilek]

Hmm, interesting. A bit sleep-deficient over here, but it seems like that might kick knowledge of having to make these transformations to the asset load, where what I'd like is for that to happen behind the scenes. So instead of just saying "load these sounds, and let me know when they're complete," I'd have to perform a secondary transformation at load time rather than in some plugin's startup system.

If this PR lets a game-specific system start an asset load based on a loader introduced by a plugin, and that plugin can then defer asset-processing to a system that completes the processing without any knowledge of the plugin that started the load, I'm OK with it. I guess I could see a use case for needing to split the steps yourself if you need granular control over the process, but I don't want users needing to know how my plugin works internally just to load supported assets.

[mockersf]

Your plugin can provide its own method that will do asset loading and transform, for example in an extension trait on AssetServer. Then your plugin users will be able to do asset_server.load_my_audio_file(file) and get directly the transformed handle

[zicklag]

I've needed this feature before, for something as simple as just setting the texture filtering mode and to me it looks like @mockersf 's PR will work great for it.

[ndarilek]

OK, if the consensus is that this will meet my needs, then it's fine by me.

One more question before I'm completely satisfied with this solution. Operating under the constraints that I don't want plugin users to use a different asset flow to load my asset type vs. other types, at what point in the cycle does the loader have access to resources? Can the asset server retrieve resources somehow?

[mockersf]

The flow when loading an asset:

1. call asset_server.load(file_path)
2. create an handle
3. start a task in IoTaskPool, giving it a clone of itself, the handle and the path, and detach, this continue in another thread
4. returns the handle

From there, the task:

1. find the asset loader based on file extension
2. read the file as bytes
3. give the bytes to the asset loader
4. the asset loader builds a map of handles to new assets (so that it can load several assets from one file)
5. for each asset in this map, the asset server will send it through a channel specific to the asset type

Finally:

• a system, one for each asset type, is running every frame and listening to new assets coming from the channel, adding them to the assets resource

[ndarilek]

Cool, if that's what the PR in question does then that sounds perfect!

Not sure whether or not to leave this issue open pending closure of the PR. I searched issues before reporting this and didn't find anything, so either I used the wrong keywords or issue search didn't search PRs. :)

[alice-i-cecile]

Not sure whether or not to leave this issue open pending closure of the PR

We can leave it open and let GitHub do the work for us when it's merged. Closed by #1665.

[zicklag]

@mockersf the asset post-processor can't borrow world resources though, can it?

// new texture with one pixel every two removed
let texture_handle_1 =
    asset_server.create_from(texture_handle, |texture: &Texture /* can't put resource here 👈 */| {
      // Can't use resource to create new asset
    });

In a situation I just ran into today I have an audio manager resource it would be very nice to be able to do something like this:

// load sound from disk
let sound_data_handle = asset_server.load("blink.mp3");
let sound_handle =
    asset_server.create_from(sound_data_handle, |sound_data: &SoundData, sound_manager: Res<SoundManager>| {
        sound_manager.add_sound(sound_data_handle);
    });

I have a feeling this would be much more difficult to accomplish and that I am going to need to use a system to watch asset events to accomplish this.

[mockersf]

no, taking a closure with dynamic parameters would be hard... it should be possible to change it to run in a system with world access and give it to the closure? Or you can try and move your ressource from out of the closure to in instead of taking it as a parameter

let sound_manager = ...;
let sound_handle =
    asset_server.create_from(sound_data_handle, move |sound_data: &SoundData| {
        sound_manager.add_sound(sound_data_handle);
    });