Rust on Consoles

[sanbox-irl]

Hi there,

I'd like to help get Rust onto Consoles more. I have two devkits from the major platform holders (I'm trying to be a little subtle here to avoid breaking any NDAs) and have signed an NDA with the third.

I have tried at work to get others onto using Rust in gamedev, but the hardstop, for the kinds of games I work in, is console support. If you've ever wondered why Godot seems to have so little traction, a lot of it is a lack of console support. Every conversation about Godot, and, unfortunately, Rust, starts with "wow I love this" and ends with "pity I can't easily get it to run on consoles."

I think we would all agree, however, that Rust is excellent for gamedev, and it would be great to code less in C++. However, to even get to that point, we need to build some infrastructure. I think this would be in the purview of the WG and would love to discuss that with some of you. I imagine this would take the form of a private repository managed by the WG which requires an invite, predicated on demonstration of an NDA signed, to join. I would be very willing to help with that. It would be easier to talk about this over voice and in a less written way of course.

Getting that infrastructure down first allows us to tackle the actual code issues necessary for console support.

Let me know what you guys things

[AlexEne]

I think, for now, all we can practically do is voice these wishes / ideas on the dev forums as those are places that filter ppl by NDA agreements already.

[aclysma]

+1 on using the pre-existing first-party-sanctioned means to discuss. (Sorry for speaking with all the charm of a wet blanket! I would love to see progress here too!)

Another concern is that even if every technical issue were solved the title still has to pass cert. You might need a waiver to use non-sanctioned toolchains and I’m not sure how keen first parties will be to grant that.

There is still lots of space for tools and backend services for AAA. I also feel that rust has to prove itself on PC and mobile before first parties will get behind rust in a meaningful way.

Maybe we should try to kick off a conversation on one of them. We may need to coordinate which one, discord private messages might be the best place for that.

[AlexEne]

We could start with the platforms where the tech-approach seems promissing.

Hint-hint: 1) http://llvm.org/devmtg/2013-11/slides/Robinson-PS4Toolchain.pdf 2) https://www.youtube.com/watch?v=9_7exO60EA8 3) windows-pc 😄

The reality is that gamedev & reasearchy things don't blend well together due to the usually hellish delivery constraints that gamedevs have. So maybe a studio that loves Rust can start something more serious in this space (@repi ? 😄).

But as I said, let's take this discussion to the dev forums.

[repi]

Rust console support has not yet been a priority for us here at Embark, been focusing on desktops (windows, Mac, Linux) first. But I think it will be later during this year so likely will do some more experiments with it, but not ship anything with it this year.

We a tracking issue on our page: https://github.com/EmbarkStudios/rust-ecosystem/issues/18

But yes given the nature of the consoles and the NDAs, work on them can't be done in public. But if we do start on it we would want to do it as a private github repo with the platform companies managing access

[Osspial]

I'd definitely support this, if we can figure out a way to navigate the maze of NDAs the console APIs are wrapped up in. From what I understand, getting Xbox One support should be the easiest out of all of the consoles, since they support UWP, but I haven't looked into that too much so I'm not 100% certain on that.

[Lokathor]

The easiest way to get going is (probably) to just shove the whole game into a staticlib, and then you'd have the platform specific stuff be the binary (wouldn't even have to be in rust) that links to the static lib, boots necessary services, and calls game_do_frame(args) every frame.

This way, each platform can have whatever platform specific top level quirks, and the game doesn't really have to know much about it. You'd pass in a few function pointers for the "services" that the game will need to have provided for it (allocator, vulkan instance, how to read a file, etc etc) and then the inside of the game can otherwise be pretty agnostic to the platform in question.

It's not the most ideal perhaps, but every console SDK can doubtless link to a C static lib, so to start just make Rust pretend to be a C static lib to the outside world.

[zicklag]

Hey, with that model that @Lokathor just mentioned, what if we then created a public crate that would define the standardized interface for communicating to the console. That crate would allow you to register callbacks for the console APIs that your game needs and it would provide the extern C functions that would be called from the console SDK.

The crate could be public because all it does is define a C interface for use with the console SDK and a Rust interface used to hook those C calls to your game's Rust code. Then we could, in the open, collaborate on a console agnostic API.

Obviously there are console-specific APIs that you might need to interface with, so maybe those would need to be separate private crates?

Anyway, at that point, behind the "NDA walls" we could develop the glue for each console that links the console SDK to our interface crate's C API.

[Lokathor]

You can't quite do that with functions because the thing with the functions to call would be your own game, not some dependency of the game.

However, in terms of defining a common set of input types that can be used to form the "interchange" between outer OS shell and inner game lib, that's pretty doable.

(Bonus: if you have this "game as a lib" setup working you can generally hotswap the game during development with not much difficulty)

[zicklag]

You can't quite do that with functions because the thing with the functions to call would be your own game, not some dependency of the game.

Ah, OK. Maybe we could make a crate with the standard types, like you said, and also include macros to generate the extern C functions for you? Maybe writing those isn't a big deal and it wouldn't be worth it, but it might help it feel a little more Rusty if you could just set_whatever_callback!(my_normal_rust_fn); or something like that in your lib.rs.

[Lokathor]

I would suggest plain and obvious function declaration, with no macros.

Don't complicate things just for the sake of it being "rusty feeling".

[zicklag]

Fair enough.

[AlexEne]

As much as I am pro rust-on-consoles, discussing it here just isn't practical at this stage.

I know it's anoying but proposing things here leads to strange situations where ppl who know about how console SDKs look like and are capable of can't say anything, while people who didn't sign NDAs are left to speculate without gettting any feedback if any of the sugestions are practical or not (or even if they address an actual issue).

For those of us who have NDAs signed, the forums are already there to facilitate more practical next steps. So, for these reasons I am semi-tempted to suggest parking this public issue for now.

[zicklag]

That whole situation is a bummer, but unfortunately I think that's sound reasoning. It probably isn't best to discuss here.

Oh, well. Their consoles their rules I guess.

[DemiMarie]

@AlexEne I agree, with a caveat: XBox One supports UWP, and UWP is a public API.

[ozkriff]

A recent URLO thread: "Is there a reliable way to support mainstream console(including std)?"

[DemiMarie]

I wonder how LuaJIT got console support. I do know that its console support is 100% open-source. Is it because LuaJIT just uses essentially C standard library functions?

[kvark]

Ideally, we'd have an organization like "RustOnThisParticularConsole", which would have all the repositories in private, would only accept members who signed the NDA, and allow these Rust community members to collaborate on the ports. Otherwise, Embark does one thing, other team does another thing, single developers re-discover all those things and probably waste a lot of time. Did anybody try to confirm with platform holders if such organization/private repos structure is acceptable?

[DemiMarie]

Ideally, we'd have an organization like "RustOnThisParticularConsole", which would have all the repositories in private, would only accept members who signed the NDA, and allow these Rust community members to collaborate on the ports. Otherwise, Embark does one thing, other team does another thing, single developers re-discover all those things and probably waste a lot of time. Did anybody try to confirm with platform holders if such organization/private repos structure is acceptable?

Is UWP an option on XBox?

[ozkriff]

https://github.com/rust-lang/rust/pull/88991 ("Add Nintendo Switch as tier 3 target")

[sanbox-irl]

rust-lang/rust#88991 ("Add Nintendo Switch as tier 3 target")

I feel like I need to find more info there. That PR is uhhh it feels kinda sketchy as hell. I think I need to have an offline conversation with them to find out what they intend haha

[Lokathor]

If you follow the Zulip link it's fairly clear I think.

[sanbox-irl]

Ideally, we'd have an organization like "RustOnThisParticularConsole", which would have all the repositories in private, would only accept members who signed the NDA, and allow these Rust community members to collaborate on the ports. Otherwise, Embark does one thing, other team does another thing, single developers re-discover all those things and probably waste a lot of time. Did anybody try to confirm with platform holders if such organization/private repos structure is acceptable?

I spoke to Ryan C. Gordon, the primary maintainer of SDL, which as you might know, has a Switch port in a private Github repo. I asked them how they came to that arrangement, and they said that once they'd done the work of making the port, they spoke to their developer relations contact at Nintendo and asked if the username/password for the repo could be posted on the forums. They agreed to that. Perhaps we could do something similar for adding the Switch as a target for Rust?

We can also create such a repo via the nda'd forums, though we'd want to talk to them before we post any links to it in the forums.

[heavyrain266]

I'm slowly starting studio "Eukaryote Interactive" focused on porting rustc and needed libraries for major consoles but costs of signing NDAs are high, but first we have to discuss technical side:

• Xbox is not using UWP anymore but instead, uses win32 + gdkx which makes it easiest target to port (except that, devkit requires working game prototype or finished one)
• PlayStation is mostly easy to port, rustc have to use Sony's LLVM fork and then you just copy-paste FreeBSD std implementation and add some console specific libs
• Switch uses OS written from scratch which makes it harder to port std
• Libraries to port: alloc, std, raw-window-handle, wgpu, winit and so on + console specific bindings (wgpu could be upstreamed by using nintendo's public vulkan extensions)

About the costs... let's start from that, after signing NDAs you cannot be a part of any other company (mostly tech ones like some studio, sony, nintendo, microsoft, apple etc.). Not every devkit is free or "easy" to acquire, switch one costs ~$500, xbox requires game prototype or already finished project to be approved and PS5 requires only a company just like others. Costs of maintaining studio in my country (Poland) are ~800€/mo without office place.

Here is the beginning of the real problem, signing NDAs forces to quit your job which slowly makes it even harder to maintain and that I'm currently unable to hire people in order to help with ports of compiler, at least 10 different libraries, prototype etc. which may take much longer. (thus that may require some fundings 😢). If anyone is willing to help, you cannot be employed by any tech company, possibly have own studio or I will talk to console vendors about signing NDAs with them.

EDIT: I forgot to add, code related to any console cannot be hosted as private repo on any public hosting because that NDA violation in some ways. It must be hosted independently on private host without direct access for outsiders (like gitlab/gitea/sourcehut instance, perforce and plastic-scm)

[DemiMarie]

@HeavyRain266 Is this approach an option?

[heavyrain266]

@HeavyRain266 Is this approach an option?

Yes, you need to wrap your game in C (tested on PS5) to sign binaries with Sony's clang and deploy to the store

See this gist: https://gist.github.com/HeavyRain266/824593ae4a961f1c1cf712e1cfc1ffb7

[DemiMarie]

@HeavyRain266 Is this approach an option?

Yes, you need to wrap your game in C (tested on PS5) to sign binaries with Sony's clang and deploy to the store

See this gist: https://gist.github.com/HeavyRain266/824593ae4a961f1c1cf712e1cfc1ffb7

What I was referring to, and what I believe @zicklag intended, is to have an abstraction layer that is independent of any particular console, and which is developed in the open. Specific implementations of this abstraction layer would be behind an NDA.

[heavyrain266]

What I was referring to, and what I believe @zicklag intended, is to have an abstraction layer that is independent of any particular console, and which is developed in the open. Specific implementations of this abstraction layer would be behind an NDA.

Ohh, got it!

Well thats not really possible, there is no need for abstraction layer. What you really and only need is, port of allocator, standard library and then generated bindings used to start porting Bevy Engine by creating console specific backends in libraries such as raw-window-handle, winit, naga and wgpu. Where wgpu port for switch is easiest because you possibly can upstream some public Vulkan extensions which are pre-existing in ash.

[Lokathor]

Everything you just described is the abstraction layer.

[heavyrain266]

All what I meant, is that. None of changes other than possibly Switch extensions in wgpu can be upstreamed as public abstraction layer described above. WGSL to PSSL translation in naga is questionable because specification isn't public and WGSL spec doesn't cover core parts such as mesh shaders and WebGPU doesn't support raytracing just yet.

[zicklag]

Ah, I see. Essentially @HeavyRain266, you're saying that we already have platform abstractions in the form of wgpu, winit, etc. All we need is people to make backends compatible with the platform abstractions we already use.

That makes sense to me.

[heavyrain266]

All we need is people to make backends compatible with the platform abstractions we already use.

Exactly, I will contact Embark about this. Everyone who is willing to help, will have to sign NDA, first. Starting with Sony because PS5 devkit is essentially easiest to acquire, next will buy Switch one and at the end Xbox Series because that one requires game prototype with trailer etc. or a game which is ready to be published according to their registration forms.

[Lokathor]

Nintendo will also ask to see a game prototype, or who you're doing work for that has a game prototype, before bringing you into the Switch dev program. (Unless they've relaxed that in the past few years.)

[heavyrain266]

According to their registration process, there was only company info required

[HackerFoo]

All console developer programs require a game pitch to convince them that your team is capable of shipping something, if not that exact game. If your team has already shipped a game, it makes it a lot easier to get accepted.

It might also be possible to get access to develop middleware, but I'm guessing that it's more difficult, because the business is about making games.

[heavyrain266]

I've seen conversation about single place for working on console ports. We could contact the Rust Foundation and ask about creation of "rust on consoles working group" to connect licensed developers for collaborative work. Something like I do one thing, someone else another thing and then Embark comes with their own implementation some day.

As a side note, a few new small/medium studios were interested in any kind of support (code, fundings, ...) after my series of tweets but I'm not really interested in collaboration which companies focused on implementing NFT games which lives on the blockchain...

[tshepang]

The Rust Foundation is what you would ask for financial support, and the Rust Project (perhaps with the help of wg-gamedev, who are part) is what you would ask for creating such a working group.

[erlend-sh]

We could contact the Rust Foundation and ask about creation of "rust on consoles working group" to connect licensed developers for collaborative work.

This seems like a natural extension of the Rust GameDev WG. You could join this WG and lead a dedicated ‘Rust on Consoles Task Force’. A dedicated channel can be created for it on the foundation’s Discord server.

[heavyrain266]

Sounds good but:

A dedicated channel can be created for it on the foundation’s Discord server

Must be done on self-hosted chat instance, same idea as not being able to work for any tech company after signing NDAs.

[DemiMarie]

Sounds good but:

A dedicated channel can be created for it on the foundation’s Discord server

Must be done on self-hosted chat instance, same idea as not being able to work for any tech company after signing NDAs.

At this point I think the best thing to do is something like this:

• Have the open source tools support developing for open platforms.

• Include something like this in the documentation:

  The only consoles supported by the open source project are:

  • Steam Deck (based on Linux)
  • XBox using UWP only (no non-UWP support)

  Support for other consoles, and for non-UWP XBox, cannot be provided openly, as the console vendors require NDAs. The only legal workaround would be clean room reverse engineering by those who have not signed NDAs, which is impractical. Instead, console ports are provided by the following vendors:

  • Company X
  • Company Y

  The following crates are supported on consoles:

  • core
  • alloc
  • std
  • winit
  • raw-window-handle
  • wgpu
  • etc

  The console ports use the same API as the open-source ports, so if your game only uses crates on the above list (and does not rely on platform-specific code) it will run on consoles without modifications.

[heavyrain266]

I don't agree with this statement:

XBox using UWP only (no non-UWP support)

As mentioned before, UWP has been dropped and what you need is windows-rs and upstream winit patch to silent "unsupported feature" for certain DLLs. Also that, I can't even find any usable UWP libraries for Rust, not to mention that UWP has been deprecated in favor of WinUI which has different goals.

UWP is only available if you are a student which gets "dev mode" on retail console and your game is super sandboxed with limited performance.

EDIT: I don't agree with this one too:

The console ports use the same API as the open-source ports, so if your game only uses crates on the above list (and does not rely on platform-specific code) it will run on consoles without modifications.

You need to provide console specific API extensions which may take some time for developer to learn the new API which allows you to send notfis or interact with console's UI or whatever else

[dlight]

@HeavyRain266 alongside porting the Rust gamedev ecosystem to several consoles, I think there is value in those two approaches:

First, make console-specific functionality entirely self-contained. Instead of forking wgpu to add console-specific bits (and risk having a higher maintainance burden of continually updating your proprietary fork), instead create crates wgpu-switch, wgpu-xbox etc, and publish a stub on crates.io with the exact same API but no implementation (just unimplemented!() on everything is enough). Then, make wgpu depend on it with feature flags, and make people use Cargo's [path] to override the crates.io stub and use your proprietary implementation under NDA.

This should ideally be done with every functionality, apart from std.

Second, identify common abstractions that apply to both consoles, mobile and desktop (like "display a notification") and implement a crate that does this in a platform-agnostic way, selecting the implementation with feature flags. Again, when selecting a console implementation, people should use [patch] to use your NDA crate.

[dlight]

Now, why am I proposing this? I think that there is value in upstreaming console support into foundational crates like wgpu, even if the actual implementation must remain closed. The chief benefit is that we will be able to update dependencies instead of being locked into old versions of wgpu, winit, etc.

[heavyrain266]

So far, only PS5 requires closed source fork of wgpu, switch uses vulkan extensions present in ash and xbox doesn't require any upstream changes.

[dlight]

Oh. Well, my suggestion then is to not fork wgpu even in the PS5 case. This enables us to a) closely follow upstream and b) make our own private modifications to wgpu (and eventually open source them), which wouldn't be possible or would be awkward if wgpu were forked.

[guilhermewerner]

First, make console-specific functionality entirely self-contained. Instead of forking wgpu to add console-specific bits (and risk having a higher maintainance burden of continually updating your proprietary fork), instead create crates wgpu-switch, wgpu-xbox etc, and publish a stub on crates.io with the exact same API but no implementation (just unimplemented!() on everything is enough). Then, make wgpu depend on it with feature flags, and make people use Cargo's [path] to override the crates.io stub and use your proprietary implementation under NDA.

I think this proposal to create a stub crate just to share the api is very interesting. It would be a way to "create" closed source rust crates when necessary, as in the case of consoles.

[heavyrain266]

@dlight That won't work either.

I see that the whole conversation is pointless because some of you don't understand what NDA is and how it works really. You cannot share interface to the code from under NDA, even as closed source crate, that still violates their rules.

As I said many times, only way is to have closed source forks which can only be shared with people which signed NDA for each console separately, otherwise you can end up being sued.

[Ralith]

wgpu is, itself, an interface, so that's clearly false. Perhaps you could share the specific NDA language you're concerned about?

[AlexEne]

I would suggest we keep wgpu over some platform's API X separate from a rust ready to work on a given platform.

wgpu isn't a required bit, and there are pretty good reasons why wgpu may not even be the right abstraction layer. I feel like the rest of the work is way more important.

[heavyrain266]

Wgpu causes too many technical issues for playstation but works fine for xbox and switch (dx + vk). I will just continue working on custom renderer for Bevy and only silver, gold, platinum sponsors and people which signed publisher contract will be able to benefit from it.

[dlight]

@HeavyRain266 thanks for the clarification!

But I think it's very productive to have this conversation in the open. Many game developers want to port to consoles but didn't sign an NDA yet.