Scripting language?

[PJB3005]

Which scripting language to go with?

There's a consideration of whether we need sandboxing so the client can safely download code from the server and run it.

Python is an option, and somebody can probably write a PyPy <-> C# thing though I've got no idea how P/Invoke works. Can't sandbox it though (PyPy does support sandboxing but it kills performance)

For C# it's easier to get to work but again, no sandboxing it without killing performance. Though you might be able to wrap the entire installation in an AppDomain but then you'll be restricted with file access. Gonna need looking.

Lua is another option but personally I think it's disgusting and not something that can handle SS13.

Then finally there's Javascript (maybe with TypeScript), I found a V8 binding called ClearScript which can probably work, and I assume it'd be sandbox able.

So really, if we drop sandboxing then having the client load a C# assembly from the server (or C# script files, that exists) would definitely be best.

[ComicIronic]

If you want to use an AppDomain, it's possible to use IronPython, which is apparently trivial to hook up thanks to .NET. Microsoft seems to support sandboxing the interpreter:

https://stackoverflow.com/questions/4393153/sandbox-ironpython

[PJB3005]

Issue is iron python is unmaintained and stuck in the Python 2 stone age IIRC.

[PJB3005]

Also, if we were going to use AppDomains I don't see a reason to not go all out C# for code since the sandbox rules are the same. But there's still the potential performance issue of AppDomain bound crossing.

[ComicIronic]

The main reason would be that C# strictness and the syntax differences will make new contributions harder. A lighter language is better for the community, and certainly Python is similar enough to DM to be able to push for a switchover without killing contributors.

The version problem does suck for IronPython.

[PJB3005]

We'd still want to go hard on running mypy over the Python code to make it more maintainable if we did use Python though, so honestly I think C# might be better than Python either way.

[psykzz]

Why not just run a linter like flake8 and perform integration tests? Having mypy is destructive if your not familiar with type checking python3, as it migrates your code.

[PJB3005]

You need type checking to be able to maintain a project the size of SS13 though. I cannot imagine how painful refactors would be if all you had to go by were AttributeErrors

[ComicIronic]

Is statically typed Python so much worse? We can hook it up for automation, and it looks about as easy as regular Python to write. The main issue I have with C# is that's it's much more verbose than a content language really needs to be, and the community is going to be much more unfamiliar with it. Good typing is important, but Java-levels of syntax is going to trip people up a lot.

[PJB3005]

I don't think C# would be that bad for content language. The thing is that type defs like that xenomeat pancake recipe urist mcnewbie wants to add would go into an XML or JSON file so he won't have to touch the C# code. There's already code for entity loading and stuff too.

Also, speaking from personal experience why Python type checking becomes ass: circular references.

In regular Python, if you have a circular reference between 2 files like this:

# file A
from B import b

class a:
    def b():
        return b()

# file B
from A import a

class b:
    def a():
        return a()

you can generally resolve this by putting the imports into the function bodies:

class a:
    def b():
        from B import b
        return b()

Problem solved

Issue is that with type checking. It's not so simple, because the signature would look more like this:

from B import b
class a:
    def b() -> b:
        return b()

So you need to have the import outside the function body, but then you get circular references so you've got a problem!

How is this fixed?

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from B import b

class a:
    def b() -> "b":
        from B import b
        return b()

yuck.

[PJB3005]

@ComicIronic well apparently there's this: https://github.com/IronLanguages/ironpython3 but it's definitely not ready for production use yet.

[ComicIronic]

Let's not go the route of static external definitions - it'll complicate contributing to have to deal with multiple formats, and you won't give people the power they want in contribution, so Urist McPancake +5 to Mining either bloats food code with special cases or it just gets added manually anyways.

Just spitballing, but one way to solve that typechecking import would be to make a macro with something like https://github.com/lihaoyi/macropy to simplify it. It is a problem, though.

[psykzz]

For the circular reference issue.

instead of

class a:
    def b():
        from B import b
        return b()

Could you not just

import shared

# file A
class a:
    def b():
        return shared.b()

# file B
class b:
    def a():
        return shared.a()

[PJB3005]

Then you still have a circular reference.

[PJB3005]

Or, no, never mind.

Though I fail to see how that solved the problem.

[psykzz]

By restructuring your module you just dont have the reference issue.

That only happens because of shared or duplicate code, so just bring it out into a helper or another module and you fix the issue,

[ComicIronic]

You can't make shared.a() and b.a() well-typed, at the same time, though. I'm surprised there isn't Python support for a dependency graph.

[psykzz]

The problem here is forcing them into a Type. Just make it a generic type in this case.

[ComicIronic]

Well we'd like to be able to type them, since it would make the language a lot more manageable - compare to DM now, where everything is terrible.

[PJB3005]

Even DM is more manageable for SS13 than Python with no types at all.

I mean DM's typing isn't that bad because there's some at least. Can't say that about most scripting languages.

[psykzz]

So really the criteria here is:

• Strong type checking
• Familiarity
• Sandboxing

[ComicIronic]

Frankly, I'm only pushing for familiarity as long as it's a possibility. The further we get from Python, the less it will be possible, and the more the other two will be important in the final decision.

[ComicIronic]

One other possibility (which might be good for readability anyways) would be to forbid the from x import y syntax for circular references. It lets mypy work properly, and cleans up stuff like the example to be much more readable.

[psykzz]

We should think beyond the circular imports, they are easy to detect with linters and won't be a blocker as long as we have a good CI pipeline.

[PJB3005]

@ComicIronic disallowing from x import y doesn't solve circular references. A specific import like that still loads the module in the exact same way, just the loading module's locals get modified differently.

[PJB3005]

@psykzz the issue with circular references isn't that they're hard to fix (they're not), the issue is that it's incredibly ugly and cumbersome to fix them in a type checked python environment.

[ComicIronic]

It doesn't solve the circular reference, it allows mypy to resolve the names correctly.

file A.py

import B;

class a:
    def b(self) -> 'B.b':
        return B.b()

file B.py

import A;

class b:
    def a(self) -> 'A.a':
        return A.a()

works totally fine.

[PJB3005]

Wait that works?

Huh learn something new every day I guess.

[ComicIronic]

Looking elsewhere than Python for a moment, I guess we're throwing Lua out immediately for lack of typing.

We could use Guile - it's meant to have good script embedding support, though I don't know how it is for C#. It might be possible to just use Scheme instead. It satisfies a typing need, and the functional aspect isn't so strong that it prevents simple useful code. The main trouble would be new syntax and brackets.

[greym0uth]

What are your thoughts on using TypeScript, it has:

1. Typing
2. Its super simple to learn and has great asynchronous capability.
3. Sandboxing when compiled to Javascript and used with Jurassic

Just wanted to see what you guys thought about a language that is growing very rapidly that isn't python.

[ComicIronic]

Looks good, but if we take it I would not use Jurassic - ES6 support hasn't progressed far enough to give classes, which would be a strong requirement for the language in my eyes.

[greym0uth]

It doesn't look like its really active but Javascript.NET is a binding to the V8 Engine that has ES6 support.

[ZoldorfTheWizard]

I think LUA should be considered too.

[PJB3005]

@Jaden-Giordano there's also "ClearScript" for a .NET V8 bonding.

@Silvertorch5 no typing, ugly syntax (IMO), no direct classes... It's kind of a mess.

[psykzz]

ES6 support hasn't progressed far enough to give classes

That's all just sugar anyway under the hood there is just the same old class system.

Also as well be compiling we can use polyfills to fix all the missing features if that's something we care about.

[PJB3005]

Would Jurassic have the advantage of low overhead sandboxing though? Also it even says it's not intended for end users in the Readme on GitHub.

Also I want to remind you that Javascript is just a crap language in general that's band aided so badly there's literally more band aids than the actual language itself, just to make it bearable out of necessity.

[BitCortex]

V8 doesn't support in-process sandboxing. If a runaway script exceeds V8's memory limits (some of which aren't configurable), V8 considers itself compromised and immediately crashes the process. So if you need to run untrusted script code safely, you should probably avoid the V8-based options. The only way to sandbox V8 is to run it in a separate process.

[PJB3005]

@BitCortex does that actually just mean it hard crashes or does untrusted code actively create security hazards by allowing access to the filesystem?

If it's just crashing it doesn't matter. The reason we need sandboxing is so that we can have client side code while still preventing servers from giving people malware by still connecting. There's not really any value to a malicious server to crash people.

[greym0uth]

Yeah the Javascript/Typescript option might not be best option.

[BitCortex]

It just crashes the process in a way that embedders can't trap.

By default, V8 only sets up the JS intrinsics (Object, String, Math, etc.). The embedder or binding layer then sets up access to other things, usually under the application's control. So unless your application deliberately sets up a JS file system API, script code shouldn't be able to mess with the file system.

On the other hand, V8 like most script runtimes is susceptible to sophisticated techniques such as heap spraying, potentially giving script code the ability to exploit application or OS vulnerabilities.

[PJB3005]

@Jaden-Giordano of course not, though so far it's the only really easily sandboxable other than Lua.

I wonder if doing C# server side and AppDomain sandboxed C# client side would be good, keeping client code to a minimum for responsive UIs and stuff, though PyPy could probably work too.

[greym0uth]

Going to lua though, with a few libraries, OOP and type(checking) can be added through two libraries MiddleClass and typecheck, and lua is really light weight, I agree with Silver on considering it.

[PJB3005]

It also makes me want to gauge my eyes out, has tons of the same issues as Javascript, and makes every variable declaration a trap.

[PJB3005]

@Jaden-Giordano both of those libraries you linked are terrible, terrible hacks on top of an all around crappy syntax and I'd rather drop sandboxing and go all out C# than have to work in that crap.

[greym0uth]

Yeah, I don't have too much knowledge on them, but I've used them once it was awhile ago. That makes sense.

[PJB3005]

So after a quick chat with N3X15 on Discord, it turns out AppDomains aren't even needed for sandboxing C# (see KSP). If that's the case it's probably the best option to go with I'd say.

It's really easy to implement (just load an assembly with System.Reflection), makes working on both sides of the game straight forward, it's powerful, fast.

Personally I think C# is the best option.

[PJB3005]

Merging this discussion with #80:

I think personally something like this could work:

Define object prototypes in a .yml file, something like this:

prototypes:
- name: hamburger
  components:
  - type: sprite
    icon: hamburger.png
  - type: food
    calories: 300

Side note: one issue with YAML is that it's really error prone if you fuck it up.

Then it could become more advanced, like this:

prototypes:
- name: welder
  class: Prototypes.Tools.Welder
  components:
  - type: sprite
    icon: wrench.png
  - type: tool
    tool_type: weld

And then an accompanying C# file:

namespace SS14.Shared.Content.Prototypes.Tools
{
    public class Welder : EntityPrototype
    {
        public override void AttackHand(Entity User)
        {
            var component = GetComponent<ToolComponent>();
            component.Enabled = !component.Enabled;
        }
    }
}

May be a little verbose but I think it can work out. What do you guys think?

[ComicIronic]

one issue with YAML is that it's really error prone if you fuck it up.

Very true. We can always encourage a client tool if we decide to use it.

My worry about the proposed system is the disconnect between the YAML and the code might make mistakes easier if you write a type wrong and then you don't find out until runtime. Would it be possible to check it in a test-compile somehow?

[ZoldorfTheWizard]

closing as this has moved to #211 #131