Ability to interact with `WebSurface` via code

[Remscar]

For?

S&Box

What can't you do?

I'm unable run JS code or have JS call C# functions on websites loaded through a WebSurface

Without these abilities it's impossible to interact with WebSurface 's unless it's with the mouse or keyboard or programmatically respond to the website.

How would you like it to work?

For the consume API I would like the following functions:

void AddCallback<T>(string name, Action<T> callback) Which will add a function of name which can be called JavaScript to trigger a C# function callback. Ideally this can support 0-n parameters (n >= 1 ideally). Perhaps the parameter to callback is just a JSON object which the user has to deal with.

void Call(string js) Which will execute a string as JavaScript code on the website. (https://partner.steamgames.com/doc/api/ISteamHTMLSurface#ExecuteJavascript)

virtual void OnConsoleLog(string msg) Overridable function where we can catch console messages logged by the website.

What have you tried?

I don't see any way around this besides creating some sort of session based middle man proxy-like website that you can separately issue POST commands to interact with the website with.

Additional context

No response

[peter-r-g]

What's the use case? Couldn't you implement the page with game UI and then use either Websockets or HTTP requests if you need to contact an external server?

[Remscar]

What's the use case? Couldn't you implement the page with game UI and then use either Websockets or HTTP requests if you need to contact an external server?

YouTube player that lets players queue videos on a shared queue, then plays the videos in a synchronized way for all players on the server.

Using websockets/http means I have some sort of middle man web service.

[peter-r-g]

YouTube player that lets players queue videos on a shared queue, then plays the videos in a synchronized way for all players on the server. Using websockets/http means I have some sort of middle man web service.

This is already possible through in-game means.

For example, a Mediaplayer addon already exists. To my knowledge, the source is not public though sadly. You would have to reach out to Octaver#3807 on Discord for that or create your own implementation.

If you are looking to only support YouTube, you could also utilize the GET parameters of YouTube to keep someone in sync. Track the current time of the video and if someone needs to be re-synced you can just re-set the panel URL. For example: WebPanel.Surface.Url = $"https://youtube.com/watch?v={videoId}&t={currentTime}"

Past that, you could create an in-game queue that is on the server which is then replicated to clients to keep everyone in sync across videos.

[f37ch]

Its a cool and useful feature anyways. But as i remember, ISteamHTMLSurface had security issues with steam login token. Guess if devs gonna add javascript calls, they need to do some restricting

[xezno]

I don't think letting people inject arbitrary javascript into websites is a good idea

[Remscar]

There are safe ways of doing this, whether SteamHTMLSurface will allow it to be safe is a whole other question. There's a lot of things this will enable/make significantly easier, even Garry's Mod had (and still has?) support for this kind of thing too, so I find it weird that this would be something that S&Box doesn't ever support.

[peter-r-g]

There are safe ways of doing this, whether SteamHTMLSurface will allow it to be safe is a whole other question. There's a lot of things this will enable/make significantly easier, even Garry's Mod had (and still has?) support for this kind of thing too, so I find it weird that this would be something that S&Box doesn't ever support.

Just because Garry's Mod has something does not make it expected in S&box. Making a new code access list for JavaScript to support such a minor feature in the grand scheme of things sounds like an enormous waste of time. You should speak more about what this feature would enable and make easier. Right now the only use case given is one that has already been made without this feature.

At the end of the day, I find it will be a very hard sell to allow games to run JavaScript on pages. Your idea of a callback that a website can call of its own volition sounds plausible but again you would need to provide a valid use case for it.

[Remscar]

There are safe ways of doing this, whether SteamHTMLSurface will allow it to be safe is a whole other question. There's a lot of things this will enable/make significantly easier, even Garry's Mod had (and still has?) support for this kind of thing too, so I find it weird that this would be something that S&Box doesn't ever support.

Just because Garry's Mod has something does not make it expected in S&box. Making a new code access list for JavaScript to support such a minor feature in the grand scheme of things sounds like an enormous waste of time. You should speak more about what this feature would enable and make easier. Right now the only use case given is one that has already been made without this feature.

At the end of the day, I find it will be a very hard sell to allow games to run JavaScript on pages. Your idea of a callback that a website can call of its own volition sounds plausible but again you would need to provide a valid use case for it.

There's a lot this feature would enable, and certainly make easier. For the use case I gave, it can be hacked in through a brittle method that is in my opinion, far more complicated than it needs to be.

I took a look at the source code of the media player you mentioned. It relies upon a 3rd party hosted API. The API stores an API key to the YouTube API which it uses to query the YouTube API for the duration, thumbnail, and title of a YouTube video. This is sent back to S&Box so it knows how long a video is when it's being played.

It then guesses that after X seconds of playing the video, it's actually done playing on all clients, before moving onto the next video. It's a guess though, because it has no way of knowing for sure if the video is done playing or not.

What's worse is that to run a (youtube) player, we've now introduced a requirement of a hosted API to do something we could easily do if we could run a simple JavaScript snippet player.getDuration() to get the duration of the currently playing video from the website. If this was just our server, no issue, but if we were to release this to thousands of players, now we're paying a huge bill to host that API, not to mention running the risk of it being attacked by a malicious actor.

We should also consider how we synchronize a video. The only way we can interact with the website is by changing the URL which causes a full page load (could take a long time). We can't do something like call player.playVideoAt(number) which would set the playtime to exactly where we want it to be. So we would have to factor in how long it will take for the page to load, the player to load, and the playback to actually start. Except we can't because we don't have access to the console logs or details about page loads. Nor could we create an iframe through JavaScript and time it's load in the web browser. So we're not really synchronized, just hopefully close enough cause the best we can do is set the URL and pray.

As for why a callback from the website -> C# would be useful; that is the only way a website would be able to communicate with the game. Reading console logs would be a primitive form of this (and would actually enable callbacks since you could serialize and send via console logs.) In the previous example where I mentioned calling player.getDuration, I would first register a call back to get the result, and then I would run some JS code which would call player.getDuration and then call my callback with the result so that the game could use it.

Sure, you can do a lot with HTTP requests, server coding, etc. But there are some things that will be shrimply impossible to do without being able to call JavaScript functions on a website. These are issues I've encountered with just my use case, but with some imagination it's not hard to see how other use cases could also run into these same problems.

I'm quite surprised to see you so passionately attacking a feature request. I can only speak to this depth in the use case I've encountered and worked through, if the devs at Facepunch would like me to document other use cases I'd be happy to assist, but in the meantime I hope this explanation shed some light on why this feature would be helpful.

[xezno]

No guarantees that this will get looked at, but if it does get implemented I think the best way would be to use the web messaging API, that way both sides would be aware of it happening & we'd be able to reduce security concerns.

[xezno]

Had another look into this and even postMessage isn't really safe (1, 2).

Using a websurface isn't the best way to go about playing videos though, I think the best solution for your use case would be to implement something like YoutubeExplode into your application, or just make a request to an Invidious API instance (like this) and grab the video streams from that.

There's a slim possibility that we might look into this again in the future, but the security risks involved make it unlikely we'll implement it right now.