search

StrataSource / Portal-2-Community-Edition

Task tracker for Portal 2: Community Edition
https://www.portal2communityedition.com
151 stars 3 forks source link

[RFC] Core Engine Extensibility #421

Open leops opened 4 years ago

leops commented 4 years ago

Summary

This proposes the inclusion in the engine of a WebAssembly runtime as a common target for all scripting and extensibility work.

Motivation

For legal reasons the source code of the engine cannot be distributed. The Source engine is, in theory, a modular system spread out in multiple libraries, however this system relies heavily on implementation details of the C++ language and compiler used to build it, making it near-impossible to write new modules without knowledge of the exact memory layout of all objects that transit between modules: this knowledge being found in the definitions of the data structures shared between the modules, this means that without access to the source code one cannot easily extend the engine with new features and / or modify the existing ones.

In case like this where handing the source code of the engine to the programmers is impractical or impossible, the solution of choice is to turn to a scripting language. Most game engines include a scripting system in some form, and Source is no exception with the VScript system using the Squirrel language. However while scripting engines do allow some level of extensibility, they usually come with a performance cost that makes them impractical to use for "low-level" extensibility, such as adding a new entity that must update itself on every frame or interacting with the rendering system. In these case it is generally necessary to get down to the source code for the implementation.

Interestingly, this is also a problem that has been extensively explored in another class of computer programs: web browsers. For obvious practicality and security reason you cannot expect every web developer to add features specific to their site in the code of all major web browsers, so the JavaScript language was developed to fill the script-shaped hole of the web platform. However as mentioned earlier there are a number of performance issues inherent to the nature of scripting languages, and while JavaScript is nowadays regarded as one of the most performant languages in its category due to modern JS engines being close to state-of-the-art in the field of JIT compilation and optimization, some classes of issues simply cannot be resolved without taking a completely different approach.

WebAssembly is this new approach. As its name implies it's not a new programming language, but instead a low-level target for programming languages to compile to. It exposes a complete set of cross-platform instructions meant to be lowered into actual assembly for the target machine in a very direct and efficient way. And because of the web platform's inherent design of executing remote loaded, untrusted code, it is sandboxed and "secure" by design (of course the actual security also depends on the actual implementation).

So after all of this, this is what this proposal eventually boils down to: integrating a WebAssembly runtime into the Source engine and exposing a hooks to it through a stable API in order to allow developer to add new stuff to the engine through scripting with performances on par with writing actual C++ code.

Guide-level explanation

How this feature would actually be used by the end developers will depend on how different programming languages are supported on top of it. Naively any language that can target WASM is going to be compatible (C / C++, Rust, Python, Go, Java, maybe even JS through AssemblyScript), but it would be interesting to provide bindings to the engine API that expose the different features in a way that is more idiomatic for the source language.

In the end all a programmer would have to do to use this features would be to drop a .wasm or .wat file in the scripts/ directory and restart the engine: for the initial version all WebAssembly modules would be loaded eagerly when starting the engine, though we may later add a way for these to be hot-reloaded dynamically while the engine is running.

As mentioned earlier WASM itself is sandboxed, so a low-level error such as an out-of-bound memory access or triggering a trap instruction should not crash the game but print an error message in the console and try to gracefully recover from the error condition (one could imagine a "module poisoning" system where, once a module has triggered an error, all code related to it is considered "poisoned" and becomes disabled in order to prevent broken invariants being witnessed). That is not to say the system is completely crash-proof: calling into the engine API with invalid parameters still has the potential to crash the game depending on how well these are sanitized (a process we may want to eschew for performance reasons).

Reference-level explanation

First of all, some context: I've actually tried to add this to the Source engine independently before in order to verify the feasibility of the idea. What will follow is an explanation of what I did to make this work and why I did it that way, not necessarily how it should actually be implemented. I'm going to split this in two parts, first on how this would be generally added to the engine, then some technical details on the implementations of some extensibility APIs.

For the general architecture of my PoC, I chose to use the Rust language and the wasmtime runtime. I choose Rust because the some important parts of the webassembly ecosystem (CraneLift, wasmtime, wasmer, wasmi, ...) are written with it, and because there is already enough ways for a 20+ years old C++ codebase like the Source engine to segfault without adding new ones. I went with wasmtime simply because I was the easier to embed, event though I retained wasmi as a secondary option for reasons explained in the drawbacks section. In my version the code is compiled into a fake server.dll to be picked up by the game, of course with actual support from the engine that code would instead be built as a static C library to be embedded directly. Also, while that's a minor thing, this means the codebase for the WASM runtime can be developed and maintained in open source, independently from the rest of the engine.

Now for the actual extensibility, the main thing I've tried out right now is adding a new entity. I find the most efficient way to do that is to expose a function to the runtime to create a new class: this will insert a new entry in the EntityFactoryDictionary, but also allocates a writable copy of the vtable of the selected base class (CBaseEntity, CBaseAnimating or CAI_BaseNPC for instance), finally this vtable is then exposed through the API to be patched with functions declared in the WASM module. This has the advantage of being almost zero-cost performance-wise since it simply redirects calls to virtual methods into the JITed code ("almost" because calls into webassembly need to be wrapped with some code to load the WASM memory location from the entity pointer).

Drawbacks

The main problem with this technical approach right now is that 32 bits Windows is not a supported target for the Cranelift backend (the compiler backend used by many WASM implementations including wasmtime). This is why I used wasmi as a backup solution for now, but since it is an interpreter and not a JIT compiler the performances are not the highest they could be. However this is 1) probably temporary, the i686-windows target will be added to Cranelift eventually, and 2) not that important if we have x86_64 builds of the engine (after all the reason 32 bits is not supported by Cranelift is because the platform is being phased out anyway)

Secondly, the biggest hurdle for developers to get into this new way of programming the engine is that it adds a compilation step when traditionally scripting systems like VScript can load and interpret the script source directly. Eventually maybe this could be alleviated by providing interpreters for scripting languages like Squirrel precompiled in WASM ?

Rationale and alternatives

As exposed in the motivations section, the rationale for selecting WebAssembly is to use a well supported platform as a basis for future extensibility work on the engine. WASM is already largely adopted, has a growing amount of tooling available, and can be targeted by many different languages. It is also designed to be highly performant and have minimal cost over bare-metal execution, a highly desired characteristic for a video game engine.

Using Rust for this is not a technical requirement as wasmtime also exposes a C API, and while I generally think the rationale that led to my use of it in my PoC still applies to a more stable implementation, the cognitive overhead of using a whole new and different language in the engine might not be entirely justified.

Finally, there are other WASM runtimes out there beside wasmtime. I mentioned wasmi as an interpreted alternative, but wasmer also seems like a possible alternative in the Rust ecosystem (though I didn't chose it because it's a lot less well documented). And finally there are non-Rust runtimes available, which I didn't explore because "I didn't want to have to write bindings for them when I already have to write a ton of bindings to the Source engine"

Unresolved questions

Future possibilities

This proposal aims to open a new way to improve and build onto the Source engine, and though the initial version of the API would probably be limited to just adding new entities, other hooks into the engine could be added to it to allow for even more customization.

Secondly since WebAssembly is a general purpose target, this proposal would allow the development of specialized, domain specific languages for developing on the Source engine.

InevitablyDivinity commented 4 years ago

Currently our idea was to use hot-loaded C# like that of S&box while using LLVM's LibTooling to auto-generate bindings for it. This sounds like a more powerful option than C#, but it sounds less portable. Are Linux and maybe Mac supported? We are definitely going to be supporting Linux, and maybe Mac if it becomes a possibility.

skyem123 commented 4 years ago

If I understand correctly, WASM itself is generic and portable, and the implementations of things that can run it fast (JIT?) exist for Windows, Linux, and Mac, because web browsers exist for those platforms..

JJL772 commented 4 years ago

@EchoesForeAndAft WASM would be supported on any platform and most languages can compile to WASM, including C, C++, JS, Rust and a bunch of others.

JJL772 commented 4 years ago

Thanks for the RFC, I'm actually a big fan of this issue format for large feature requests.

I see a few issues with this idea though, and I'd like to address them.

What about debugging?

From a couple quick google searches, it doesn't seem like WASM can have any debugging info embedded into it. Our version of vscript doesn't support debugging, but that's a feature I wanted to add before release (think Source 2's vscript debugging).

Why not Mono or Microsoft's embeddable .NET runtime?

Both of these solutions are cross platform and are similar. This would give people the ability to easily use C#, F# or VB.NET with p2ce. MSIL also supports debugging info out of the box.

Our Current Proposed Solution

As of right now, our current idea is to use Microsoft's embeddable .NET runtime and build a hot-loading C# layer similar to what S&Box has. Script bindings would be automatically generated using a clang-based tool (similar to source 2's schema compiler).

I don't see the benefit of using WASM over a .NET runtime or just normal scripting languages, especially with the lack of debugging info. It would be cool to look into for sure, but this doesn't exactly stand out compared to our current ideas.

leops commented 4 years ago

To address a few of the questions:

Cross Platform support

WASM doesn't generally support operating systems a platform, it's a much lower level abstraction so it instead supports machine architecture. That is to say, WASM doesn't run on Windows or macOS or Linux, it runs on i686 or x86_64 out AArch64. The WebAssembly spec only defines the language and how it should be executed, not what functionalities are available to it (by virtue of being designed for the web platform where things like the fileystem and networking work very differently to how they do in traditional OSes). This is where the WebAssembly System Interface (WASI) project comes in, it defines a general abstraction layer over stuff you would expect an operating system to provide like clocks, filesystem or networking that could be specialised for the Source engine (the filesystem call would go through the VPK archives before hitting the actual filesystem for instance).

Debugging

WebAssembly files do support debugging informations, either by using an external sourcemap as it is traditionally done on the web, or by embedding a DWARF section like what used by most other native languages. What is less clear to me at the moment, and why I left a note about debugging in the "unresolved questions" section, is how debugging would work. According to the wasmtime documentation it looks like it generates debug infos in the JITed assembly that can be picked up by LLDB, but its JIT debugging doesn't work yet on windows and so I haven't tested it.

Mono / MS .NET

I think using the .NET runtime is indeed a valid solution to the general problem of "making source more programmable", and a move in a very good direction as well. However I think WebAssembly is a more open-ended and generic solution to the same problem, and one that opens up the space for programming on the source engine a lot more broadly.

As you might have guessed by now, I program in Rust a lot. I particularly like the concept of "zero-cost abstraction", the idea that you shouldn't pay for something you don't use. This is especially important in video games where performance budgets tend to be very tight. In that regard I think WebAssembly comes a lot closer to that idea: I mentioned it having little to no overhead over actual assembly, but I consider that the overhead doesn't only manifest in terms of performance but also in the abstraction imposed by the runtime as well. For instance, how WASM represent memory is very low level: a module allocate memory by 64KB pages and can access it linearly by loading addresses from 0 to the current size of the memory, up to 4GB. Not only does this make saving and restoring the state of the module very easy because its memory will generally be backed by a single allocation that can be dumped as-is, it also means there is no imposed abstraction over memory management, and scripts can bring their own allocator to fit their use (a module with a predictable memory allocation pattern could use a specialised allocator for instance). WASM imposes no threading model as well, as I noted quickly in my reference section once built WASM modules are nothing more than a collection of C functions that can be called from anywhere (of course since the backing memory store is shared by all functions in a module they should probably be put behind a mutex or semaphore).

Finally and as a quick aside, I'll note that it is technically possible to build the Mono or MS runtime for WebAssembly, and like many other languages there are projects to port those runtimes to WASM. So the two ideas are not fundamentally incompatible and can be built on top of each other (after all, this is what I meant by "explicitely supporting languages"), though of course that might lead to a spread of human resources over many projects instead of one.

JJL772 commented 4 years ago

The main problem with this technical approach right now is that 32 bits Windows is not a supported target for the Cranelift backend (the compiler backend used by many WASM implementations including wasmtime).

@leops I forgot to address this in the original issue. This isn't an issue because P2CE will only be supporting 64-bit platforms.

leops commented 4 years ago

I just realized I initially did not mention V8 in my list of possible engines due to it being a very heavy dependency, but on the CS:GO branch Source is actually already bundled with V8 for Panorama. This would generally solve the debuggability question, as an embedded V8 instance can be remotely debugged from a running Chrome browser with all the devtools usually available on the web.

codefromthecrypt commented 2 years ago

@leops thanks for writing this. I happened upon this browsing github and really enjoyed your sound reasoning about extensibility and how it relates to WebAssembly. Regardless of outcome, what you did, discussing in public in OSS is something worth thanks!