Make a table shippable

[freezy]

VPE is currently able to export a table as .vpx which is 100% compatible with Visual Pinball. However, we want to be able to let authors make full use of the Unity game engine, and thus a table file also needs to contain data that isn't compatible with Visual Pinball.

Such data can be shaders, lightmaps, compiled code, the object hierarchy, which components sit on which objects and their data, and so on. Basically the entire scene, potentially with dependencies.

The table file is loaded by the player at runtime, so some data such as shaders and code need to be compiled beforehand. But we also want to keep the table file editable. In summary, we have two hard constraints the table file must fulfill:

1. The table file must be loadable and runnable at runtime.
2. The table file must be loadable in the editor and thus modifiable.

Runtime

Some runtime assets are platform-specific. Shaders and compiled code fall under that category. Best-case scenario: We get Unity to build for multiple platforms and pack all binaries into the table file. Worst case: Separate table files per platform.

I don't think it's possible to compile shaders at runtime, but maybe I'm wrong. We could compile scripts at runtime, but probably in a limited way (i.e. they would only be able to interact with an API defined by VPE, not the entire Unity API).

Editor

Importing the table file into the editor should restore the scene as the original author exported it. If a table uses additional UPM dependencies, they should be pulled in automatically, but if they have to be added manually that would be acceptable as well.

Packaging

The .vpx format acts like a virtual file system, so we can add whatever data we want and the table file would still be loadable in Visual Pinball.

The easy/dumb approach would be:

• Write VP-compatible data into the .vpx as we do now
• Include the scene in a separate stream
• Build the scene as AssetBundle and store it into the another stream

But that would save everything three time. Even worse, Visual Pinball's example table takes up 18M when saved as .vpx and 200M as .unity scene after import. So assets that take up space (meshes, textures and sounds) need to be saved a) only once and b) efficiently.

So the "smart" approach would be:

• Write VP-compatible data into the .vpx as we do now - that includes textures, meshes and sounds.
• Include the scene but the scene would only reference the heavy assets in the .vpx format and not copy them
• Build the scene as AssetBundle but fetch the heavy assets from the .vpx format as well. This should be possible with the Addressables package I think.

Performance

In my opinion, as long as we can export to VP-compatible .vpx files, using a completely new format (.vpe) would be acceptable as well. I would consider this, if:

a) the new format is significantly smaller b) the new format is significantly faster to load at runtime

Compatibility

I don't know how this works exactly, but a table file created with Unity version X should be loadable by a player compiled with Unity version Y.

How?

I'm not yet familiar enough with Unity's package formats and how the scriptable build pipeline works. I'll let you guys comment below :)

[freezy]

So I've done my homework and had a look at how Unity handles assets.

AssetDatabase

Unity doesn't use the original assets but converts them in a runtime-optimized, platform-dependent format. The result is stored in the Asset Database.

The Asset Database comes in two files:

• Library/ArtifactDB for imported assets
• Library/SourceAssetDB for the metadata

An Asset Database can be converted into a Custom package (or Unity package) that can be imported into the editor. It's what VPE currently is: a file structure with package.json, assembly definitions and so on, that can be either added from outside or embedded into the project.

Another way is to convert it into an Asset package, resulting in a .unitypackage file. AssetDatabase.ExportPackage() allows per-file granularity. When exporting, all "dependent assets" can be exported as well, which are the source files (as far as I understood). Asset packages is also what the Asset Store is using. For import, there is an API as well.

The import/export API is only available at editor time (it's part of the UnityEditor namespace). However, the format is known, so we might be able to pull out resources at runtime.

To summarize, .unitypackage files are:

• Cross-platform
• Can be generated and imported via an API
• Can contain source code, shader/material sources, textures, geometry, basically everything in a scene
• Are made for the editor, but we might be able to extract data at runtime
• Support a lot of different formats (e.g. .hdr)

AssetBundles

From the documentation:

An AssetBundle is an archive file that contains platform-specific non-code Assets (such as Models, Textures, Prefabs, Audio clips, and even entire Scenes) that Unity can load at run time.

AssetBundles are the only way of shipping things like shaders. They are also what's used under the hood by Unity's Addressables. They can be created and obviously loaded with an API.

However, AssetBundles can't contain code. For that, we would either need to ship assemblies (for all platforms) and load them manually, or compile the source at runtime using the Roslyn compiler, for example.

In summary, AssetBundles are:

• Platform-specific
• Can be generated and loaded via an API
• Can't contain assemblies, only "data", but in an optimized fashion
• Built for runtime, so very fast to load (no convertion whatsoever, Scene AssetBundles are even optimized for the scene to load)

Options

Let's be more specific what we actually need to ship in a table file:

• Table Media - Textures, meshes, audio, and probably video at some point
• Table Data - Metadata of the game items, collections, layers, etc
• Table Scripts - Source code for the table, either as-is, or additionally compiled for all platforms
• Scene Data - Metadata of the scene, i.e. the object hierarchy and its data including materials, configuration assets, and more. It indirectly includes the relevant parts of Table Data as well.
• Render Data - Compiled Shaders and probably a lot more

We have three methods of storing data:

1. Store the as individual "files", similar to what we do now (GameItems are just binary files)
2. Create one or more .unitypackage files and put them in the table file
3. Create one or more AssetBundles and put them into the table file

To simplify, let's assume for a moment we break backwards-compatibility with the .vpx format and look at how we can store each of these items in some arbitrary file container:

• Table Media we can probably store in any of those. Using "files" or .unitypackage would reduce flexibility in terms of file formats. With .unitypackage we would need to fiddle because there are no official APIs. With an AssetBundle it would be the most performant, but it's not cross-platform (unless Windows/macOS/Linux are compatible).
• Table Data makes only sense as "files" or .unitypackage, since it would be only useful in the editor for exporting to .vpx (at runtime, Scene Data would be used).
• Table Scripts are more complicated. Both source and compiled scripts need to be stored as individual files and handled manually during runtime. We can package them either as "files" or in a .unitypackage, but not with AssetBundles.
• Scene Data needs to be stored in both the .unitypackage (for the editor) and an AssetBundle (for runtime).
• Render Data need to be stored in the AssetBundle, for each platform.

Conclusion

None, yet. @rbxnk, @ecurtz and @Roland09 I'd like you guys to validate the above first, to see whether there's something wrong or missing.

[rbxnk]

Your research is pretty much spot on, but I'd just like to point out a few things that may or may not affect the conversation.

1. I don't see any reason to use .unitypackage as a runtime format, as you mentioned it's basically a glorified zip file, with no official runtime APIs, if we want to ship an arbitrary file structure, we can just use zip, or the ole/vpx stuff. Now that said I like the idea of using a .unitypackage as the fallback to share the "editable" side of a table, but in my opinions it still makes sense to bury that in some other archive, and pull it out via one of our import calls in the editor, since sharing around files with .unitypackage as vpe tables would just feel... weird. (Also if we go this fallback route we should be wary of folks just stripping that part of the package for file size reasons, or to not share with the community... if enforcing sharing is a goal here)
2. Asset bundles/Cross-platform concerns - As mentioned, asset bundles are an archive of "munged" assets, they're built for the target platform. Not a huge deal by itself, we could have the exporter hit all the targets and package everything up. However - platforms evolve over time, new ones are added, file formats are deprecated, etc. Relying on Unity's built exports for archival purposes could very quickly result in a situation where there's .vpe tables floating around that aren't usable unless someone goes through the trouble of updating it. Or worse, say a new platform arrives on the scene, the VPE player gets ported to it, then all of a sudden none of the "legacy" tables will work, since there's no asset bundles built in the archive for it.
3. Scripts - to cut to the chase, there's no way we can support (caveat - cross platform) c# as a "scripting" language for folks' tables. iOS for example, straight up disallows JITing or loading executable code otherwise. We're going to be pretty reliant on Unity's native compilers, or at least their intermediate ones, since il2cpp and burst are going to be the primary targets Unity supports (for example I don't think Unity's mono target can even support x64). Anyway, long winded way of saying, as longs as we just support a scripting interface instead, we'll be fine, just don't count on table authors being able to use c# or directly interface with the VPE code for exported/runtime stuff.

[freezy]

@rbxnk Thanks!

1. I agree that if a .unitypackage is just a glorified .zip file, we should save (but compress) the files directly. It's just that the APIs for grabbing everything and writing to .unitypackage are there, contrarily to grabbing everything and writing to separate files. But I'm sure we'll figure something out.
2. I completely agree that we should do everything to keep the format as long-living as possible. However, there is stuff like compiled shaders nobody can foresee how a future new platform will handle them, so I don't think this is a use case we would be able to support (unless, of course, we restrict the shader usage and ship all shaders with the player). So what's important is that the format provides the assets necessary to automatically recompile the .vpe files once a new platform arrives. About changing formats I don't have any experience. Which types of assets usually change? In any case, we'll try to store as much data as possible in a platform-independent way, so which is the data we can't and has a risk of changing?
3. I'm not sure I'm following. If iOS can't JIT nor load assemblies on runtime, how would it support a scripting interface? What's the benefit of an interface when you can't execute the code implementing it?

Anyway, I think we're closing in on basically a) grab everything from the scene, save it in the most agnostic way possible, use as much as possible at runtime, and b) only compile the strict minimum into platform-dependent bundles. That's pretty much what @ecurtz proposed on Discord if I understood correctly.

There is just one thing I'd like to be clarified. Let's say we push towards an entirely cross-platform format. Everything platform-specific would move into the player app. Which would be the issues? Is it even possible? What I've understood so far:

• Authors could only use a limited set of shaders, no custom shaders like shaders built with Shader Graph
• HDR textures seem to be a problem
• Loading will take (a lot?) longer
• ...what else? The amount of Compiling XYZ.. messages during Unity's build is kinda scary.

[rbxnk]

regarding 2: I think there's a bit of precedent for shaders in in this space, like cross compilers and lately especially the spir-v stuff, but at the end of the day shaders are code, and they're just fundamentally going to be harder to carry forward that binary image formats or mesh layouts. If the goal is to always automatically recompile, we're setting up some future maintainer to write some translator from whatever the unity format we support is to whatever the format of the day is. And unfortunately this is still dependent on Unity eventually allowing us to load shaders from arbitrary source at runtime, as it is somebody would still need to route through the editor.

regarding 3: Sorry I wasn't clear, what I mean establish some standard way that scripts would interact with VPE and the existing hooks etc, and embed some common interpreters, lua, python (esp. for mpf), js (others?)

regarding your last question bullets - imho:

• yes, if we want to be totally agnostic, we'd need to establish a set of shaders (but could expand over time, e.g. vp9 to vpx)
• HDR textures is an issue to solve, but not that big of a problem, we just need to write or find a library to read them, at the end of the day they're just textures, vpx seems to use them as cubemaps generally, not a big deal
• maybe maybe not, depends on what we do. don't see why well optimized and especially native loaders couldn't be plenty fast
• agreed, and the current state of burst/ecs leaves much to be desired. start up times seem pretty atrocious at the moment.

[ecurtz]

I don't feel like the current cross platform load from the VPX file is offensively slow, although obviously the faster the better. There are presumably ways that could be improved such as skipping the intermediate GameObject creation for anything that was going to end up in ECS. It might also be possible to include more complete objects in the main application, especially if the intention with authoring is to allow even more pre-built mechs.

I believe the iOS stuff allows interpreted languages as rbnxk mentioned. The limitations is on compiled code that wasn't included in the original application. Since iOS is such a walled garden anyway I don't think it would ever be possible to release VPE in a way where code changes wouldn't have to be included in main application updates, leaving any DLC as pure content assets.

The "plan" as I understand it for ECS compilation is that they intend to move to an asset bundle format in which that happens prior to packaging rather than at launch. That doesn't really help us in this situation unless they manage to keep that format cross platform or small enough so that it doesn't matter.