Execute programs in a separate content process

[remi6397]

Currently everything is contained within a single process, so for example when you create an infinite loop in your code, it would hang the entire application:

def TIC
 while true
 end
end

If we fork runners into separate processes (like it's done in most applications that execute untrusted code, such as browsers), we gain ability to:

1. Simply halt hung programs without killing TIC itself
2. Sandbox programs (using Capsicum, pledge, sandboxd, seccomp-bpf (?), etc.) to prevent arbitrary code execution

I'd already begun work on this offline, but I wanted to know community's opinions on that…

[nesbox]

Sounds good, after all these additions I only care about performance and whether WASM allows code to run in a separate process, I hope so :)

[joshgoebel]

Well, it's not a separate process... WASM3 is just a WASM virtual machine... you can definitely still mess everything up with infinite loops... but this is an annoying problem also shared by WASM-4 so hopefully there is a way to deal with this inside the WASM3 engine...

@remi6397 Can you name a case where any of our runtimes currently allow "arbitrary code execution"?

[remi6397]

Well, it's not a separate process... WASM3 is just a WASM virtual machine... you can definitely still mess everything up with infinite loops... but this is an annoying problem also shared by WASM-4 so hopefully there is a way to deal with this inside the WASM3 engine...

That's not what we meant here. Forking allows us to sandbox sensitive parts, which in this case means the code runners. Some code to visualize this idea:

void tic_runner_fork(tic_core *core, tic_script_config *config) {
  assert(core->runnerPid != 0);

#if !defined(TIC_RUNNER) ||                                                    \
    !(defined(WIN32) || defined(_WIN32) || defined(__MINGW32__))
  core->runnerPid = -1;
#else
  int coreToRunner[2];
  int runnerToCore[2];

  if (pipe2(coreToRunner, O_NONBLOCK) < 0) {
    fprintf(stderr, "runner: could not create coreToRunner pipe: errno %d\n",
            errno);
    abort();
  }

  if (pipe2(runnerToCore, O_NONBLOCK) < 0) {
    fprintf(stderr, "runner: could not create runnerToCore pipe: errno %d\n",
            errno);
    abort();
  }

  core->runnerPid = fork();

  if (core->runnerPid < 0) {
    fprintf(stderr, "runner: could not spawn a runner: errno %d\n", errno);
    // We _probably_ want a core dump for that...
    abort();
  } else if (core->runnerPid == 0) {
    close(runnerToCore[0]);
    close(coreToRunner[1]);

    core->writeFd = runnerToCore[1];
    core->readFd = coreToRunner[0];

    tic_runner_disarm(core);
  } else {
    close(coreToRunner[0]);
    close(runnerToCore[1]);

    core->writeFd = coreToRunner[1];
    core->readFd = runnerToCore[0];

    config->init = tic_runner_script_init;
    config->close = tic_runner_script_close;
  }
#endif
}

static void tic_runner_disarm(tic_core *core) {
  // Only ran in the child process
  assert(core->runnerPid == 0);

  // Close redundant file descriptors inherited from parent
  // readFd (coreToRunner) should be lower!
  close_range(3, core->readFd, 0);

#if defined(TIC_RUNNER_SANDBOX) && defined(__FreeBSD__)
  caph_cache_catpages();
  caph_cache_tzdata();

  if (caph_limit_stdout() < 0 || caph_limit_stderr() < 0) {
    fprintf(stderr, "runner: could not limit stdio capabilities!\n");
    abort();
  }

  if (caph_enter() < 0) {
    fprintf(stderr, "runner: could not enter capability mode!\n");
    abort();
  }
#endif
}

@remi6397 Can you name a case where any of our runtimes currently allow "arbitrary code execution"?

Running untrusted code is always a risk, especially when we have so many kinds of VM's maintained by different vendors. It's only a matter of time till a vulnerability shows up in one of them. Kernel-enforced sandbox is a form of mitigating those possible threats. I mean, you can run a browser unsandboxed and as long as you're always up to date the risk is very low, but not inexistent, so sandbox there is.

As a digression, WASM is sandboxed by design under most implementations, so no need for forking there.

As for the current status, I'm a bit out of time lately, but if I manage to find some time, I'm probably going to simplyfy the IPC somehow, which in my prototype looks like this (ugh):

typedef struct {
  tic_dispatch d;
  u8 color;
  char text[];
} tic_dispatch_trace;

void tic_api_trace(tic_mem* memory, const char* text, u8 color)
{
    tic_core* core = (tic_core*)memory;

    if (core->runnerPid != 0)
    {
        core->data->trace(core->data->data, text ? text : "nil", color);
    }
#if defined(TIC_RUNNER) && TIC_RUNNER == 0                                     \
    && !(defined(WIN32) || defined(_WIN32) || defined(__MINGW32__))
    else
    {
        tic_dispatch_trace msg = {
            d = {
                t = TIC_DISPATCH_TYPE_TRACE,
                s = sizeof(tic_dispatch_trace) + strlen(text) + 1,
            },
            color = color,
        };
        strcpy(&msg.text, text);
        write(core->writeFd, &msg, t.d.s);
    }
#endif
}

Of course systems that don't support forking are accommodated for. Threads can also be added as a third option to allow halting of hung programs on some of those systems.

And of course I have to implement kernel-enforced sandboxing on macOS (sandbox_init) and modern Linux (seccomp-bpf) if I want this thing to have any meaning for anyone other than myself (probably the only FreeBSD user here :stuck_out_tongue:).

[joshgoebel]

As a digression, WASM is sandboxed by design, so no need for forking there.

WASM is a specification - not an implementation, so how do you mean it's sandboxed by design? Is WASM3 not just a VM running WASM byte code like our Lua is a VM running Lua bytecode? What is the additional sandboxing you're referring to?

[remi6397]

On 1/2/22 02:37, Josh Goebel wrote:

As a digression, WASM is sandboxed by design, so no need for
forking there.

WASM is a specification - not an implementation, so how do you mean it's sandboxed by design?

I stand corrected then, but even if sandboxing is not explicitly mandated by the specification, WASM was designed so that you can easily contain it transparently, not by capabilities.

Is WASM3 not just a VM running WASM byte code like our Lua is a VM running Lua bytecode? What is the additional sandboxing you're referring to?

As I said, WASM doesn't [need to] support capability-based security, so the primary benefit from isolating the runner into a separate process would be that of being able to soft-reset an infinitely looped program. Though I'm not sure whether WASM supports forking, threads could be used there instead.

So desktop platforms that support kernel sandboxing (Linux, macOS, et al) would benefit the most from those changes security-wise.

I'm aware that the way I presented this idea is a bit non-intuitive. I'm mostly basing my thoughts on these documents https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/design/sandbox.md.

[joshgoebel]

I think you'll need to be more explicit. Most of our runtimes are already sandboxed to the extent they do not have any access to the network, or filesystem, or the host operating system, etc... TIC-80 does not itself run with escalated privileges. So I'm not sure how running say a Lua VM in a second unprivileged process is much safer than just running it in the unprivileged TIC-80 process? Though perhaps you are talking/hinting about actual restrictions the OS itself can provide... like denying the process network access - even if it somehow "escaped the VM"...

I also worry about the cost of going across this boundary... I was just reading up on decisions made by WASM3 about this (sorry was a day or two ago and don't have link handy - I think it's in their docs though) and they purposely have used a Fiber model in several applications to avoid the expensive costs of real multiple threads/processes. There is a real cost associated with every time the kernel has to switch from one process to another.

I agree that while(true) {} freezing TIC-80 is bad - and would be nice to fix, but it seems there are quite a few different options to solve that vs complete process isolation. :-)

WASM3 (by design) doesn't have a "run loop" like many other VMS, so it's not quite as simple as a callback or counter there...

[remi6397]

Though perhaps you are talking/hinting about actual restrictions the OS itself can provide... like denying the process network access - even if it somehow "escaped the VM"...

That's exactly what I mean, kernel-enforced sandboxing can provide a maintainable and predictable layer of additional security.

There is a real cost associated with every time the kernel has to switch from one process to another.

Writing to or reading from a pipe does not cause a context switch, if that's what you mean.

I agree that while(true) {} freezing TIC-80 is bad - and would be nice to fix, but it seems there are quite a few different options to solve that vs complete process isolation. :-)

As I've already said, we can still use threads where we can't fork().

WASM3 (by design) doesn't have a "run loop" like many other VMS, so it's not quite as simple as a callback or counter there...

@joshgoebel, I'm lost, are you referring to WASM as a runtime for TIC (as it currently is), or as a possible future language binding?

[joshgoebel]

we can still use threads where we can't fork().

I was referring to the possibility of neither. :-) Evidently you can fix such things with lightweight single-process fibers also... though they may not provide the other types of isolation you're speaking about. Although I assume if a game was distributed as an executable (and didn't need network access, etc) that it could just boot right into that security context somehow? The real issue would be the IDE where you want the admin mode to have permissions that the games don't have, correct?

or as a possible future language binding?

https://github.com/nesbox/TIC-80/pull/1772

I'm referring to the "language binding" that's already in PR and just needs the final touches. I call the language bindings runtimes also but now that you've pointed out the confusion there I'll have to see if I can rejigger my thinking, LOL.

[remi6397]

we can still use threads where we can't fork().

I was referring to the possibility of neither. :-) Evidently you can fix such things with lightweight single-process fibers also...

You cannot use fibers for that, as they are not preemptive.

though they may not provide the other types of isolation you're speaking about. Although I assume if a game was distributed as an executable (and didn't need network access, etc) that it could just boot right into that security context somehow? The real issue would be the IDE where you want the admin mode to have permissions that the games don't have, correct?

The problem is that sandboxing GUI's usually involves black magic (also would break drag-n-drop in the studio), but if we get that to work, it'd be even better.

or as a possible future language binding?

1772

Cool, I saw it in a glimpse; I wonder if the binary part could be adapted to mruby bytecode too…?

I'm referring to the "language binding" that's already in PR and just needs the final touches. I call the language bindings runtimes also but now that you've pointed out the confusion there I'll have to see if I can rejigger my thinking, LOL.

Same applies to me, I too freely use these terms interchangeably in this discussion. xD

[joshgoebel]

You cannot use fibers for that, as they are not preemptive.

They don't have to be preemptive, you yield once in a blue moon inside loop constructs. As I said I read a writeup where the WASM3 people talked about using this successfully in another project for exactly this purpose (breaking infinite loops). :-)

Wren has amazing Fibers as well and you can use them in tons of amazing ways, also not preemptive. :-)

Cool, I saw it in a glimpse; I wonder if the binary part could be adapted to mruby bytecode too…?

Sure, the mruby language binding would just need to know what to do with it... and if you have BINARY + source then things get confusing... that's why script: wasm has no "source", just text... the BINARY chunk is the canonical part of the cartridge... BINARY is just a variable sized chunk of data, up to 4 banks (256kb)... you can store anything there.

[remi6397]

You cannot use fibers for that, as they are not preemptive.

They don't have to be preemptive, you yield once in a blue moon inside loop constructs. As I said I read a writeup where the WASM3 people talked about using this successfully in another project for exactly this purpose (breaking infinite loops). :-)

Well, I'm no kernel engineer, but from what I understand, when you externally make one task yield to another task, that's preemption. :upside_down_face:

Erlang has preemptive "fibers", there is a scheduler, etc., but honestly I have no idea how would one implement that, sounds hard. :stuck_out_tongue:

[joshgoebel]

, when you externally make one task yield to another task,

Preemption is when a task is forcefully interrupted. Although I guess if you're using the work "make" there in a strong sense you're not wrong. :) But you can get a lot done with co-operative multitasking (what we had back in the day). It actually works decent IF your software yields in a timely fashion and doesn't have major bugs.

but honestly I have no idea how would one implement that

There are existing fiber libraries for C of course... WASM3 used one IIRC...

[remi6397]

, when you externally make one task yield to another task,

IF your software yields in a timely fashion and doesn't have major bugs.

Well, that's what we want to mitigate after all: infinite loops.

Also, correct me if I'm wrong, but isn't the main advantage of fibers that they are cheaper to spawn? There would only be at most one content process at a time responsible for the currently loaded VM after all, so I think it still makes sense.

[joshgoebel]

Well, that's what we want to mitigate after all: infinite loops.

But in this case the "your software" was referring to TIC-80 and WASM3... assuming WE have no major bugs then coop-multitasking can work just fine between TIC-80 and the VMs runtimes. We're talking about infinite loops in the users software - which the runtimes could catch and report without being preempted.

I mean what we do right now is coop multi-tasking essentially and it works just fine 99% of the time with the only real exception being hung loops. :-) And I just fixed Lua getting hung with a 15 line patch to the Lua VM...

Also, correct me if I'm wrong, but isn't the main advantage of fibers that they are cheaper to spawn?

That's certainly true, but I dunno about main or not... even in a sep process we'd still need a way for the runtimes to realize they are stuck and report debugging info etc... just crashing (without any debug information) is not useful. Perhaps you can do that with signals though (not sure). In any case I'm thinking possibly WASM3 could be handled the same way I just wrote a patch for Lua... with a countdown timer... and then the VM just stops itself when it runs out of CPU "ticks".

More complete isolation is an interesting idea, but I'm convinced it isn't necessary to prevent hung VMs from infinite loops... that can be solved MUCH simpler.

[remi6397]

We're talking about infinite loops in the users software - which the runtimes could catch and report without being preempted.

How could the runtimes catch anything on the same thread if the loop in the user's code blocks that thread in the first place (with the WASM runtime being the only exception)?

even in a sep process we'd still need a way for the runtimes to realize they are stuck and report debugging info etc... just crashing (without any debug information) is not useful. Perhaps you can do that with signals though (not sure).

Are you talking about when the runtime VM itself crashes, or when the user's program has to be terminated? Both can be handled gracefully, without crashing the UI thread and also without much effort.

And I just fixed Lua getting hung with a 15 line patch to the Lua VM...

Someone is going to have to maintain downstream patches for each of our VM's, and if they fail to do that, then, well, vulnerabilities may creep in. :stuck_out_tongue_winking_eye:

More complete isolation is an interesting idea, but I'm convinced it isn't necessary to prevent hung VMs from infinite loops... that can be solved MUCH simpler.

Using threads would be easier and more portable, which is why I started this discussion in the first place.

But personally, I'm not interested in anything but sandboxing. :grin:

[joshgoebel]

Someone is going to have to maintain downstream patches for each of our VM's,

It's a very trivial patch, and not all of the VMs even require patching. I'm not imagining it's much effort to upkeep. And again I think this is true separate process or not. I'm not talking about crashes. A VM crashing should almost NEVER, EVER happen... while it'd be nice if that didn't bring down the UI, that's not a thing that happens really anyways.

How could the runtimes catch anything on the same thread if the loop in the user's code blocks that thread in the first place (with the WASM runtime being the only exception)?

The users code isn't "running" really - it's not controlling anything at least. The VM is. The VM runtimes are in control of their loops, these are all virtual machines. They can detect they've taken to long and return. It's trivial. Now if there is a bug in the runtime itself, that's another thing, but that's not what [I'm] talking about here at least.

Using threads would be easier and more portable

Feel free to show us some code and the benefits. :-) It's outside my ability so I'm going to focus on improving the existing architecture until someone comes along and writes a shiney new one. :=)

[joshgoebel]

FYI: This isn't an argument against sandboxing other than to say "it's not needed to handle VM infinite loops well". That's a very big hammer for a tiny spinner of a nail. But sandboxing has other benefits.

[remi6397]

FYI: This isn't an argument against sandboxing other than to say "it's not needed to handle VM infinite loops well". That's a very big hammer for a tiny spinner of a nail. But sandboxing has other benefits.

You're focusing on the hanging part, but I'm mainly interested in sandboxing as a security feature here. The rest is a byproduct for me, but that very big hammer may come in handy later, when it comes to maintainability.

It's a very trivial patch, and not all of the VMs even require patching. I'm not imagining it's much effort to upkeep.

Up until the VM internals change significantly, I suppose.

How could the runtimes catch anything on the same thread if the loop in the user's code blocks that thread in the first place (with the WASM runtime being the only exception)?

The users code isn't "running" really - it's not controlling anything at least. The VM is. The VM runtimes are in control of their loops, these are all virtual machines. They can detect they've taken to long and return. It's trivial. Now if there is a bug in the runtime itself, that's another thing, but that's not what [I'm] talking about here at least.

Depends on how the VM is designed.

Using threads would be easier and more portable

Feel free to show us some code and the benefits. :-) It's outside my ability so I'm going to focus on improving the existing architecture until someone comes along and writes a shiney new one. :=)

You do it, I'm here for the forks. :D

[joshgoebel]

I'm mainly interested in sandboxing as a security feature here.

I hear you. I don't really see that happening unless someone comes along whose interested in the coding, not just the discussing of it. I'm not sure how likely that is to happen.

Up until the VM internals change significantly, I suppose.

I think we're pretty safe in that regard (doubly so with Lua). It literally took 5-10 minutes to patch (with zero familiarity with the Lua codebase)... so I'm not worried about Lua at least - and Lua is super popular (since it was the first language) so even fixing just it is a big win.

Depends on how the VM is designed.

True, but I'm passingly familiar with Lua (having just patched it), Wren, Duktape, and WASM-3 and think all of them can do this fairly easily. We had it working for almost every language before but it was ripped out because then Lua hooks were too slow... we've never had it for Wren or MRuby if I'm not mistaken.

But I'll try to drop off here and you can do back to discussing multi-threading. :-)

[remi6397]

I don't really see that happening unless someone comes along whose interested in the coding, not just the discussing of it. I'm not sure how likely that is to happen.

I would've coded more if I had time and if not for the current rapid development with for example #1777 putting question mark over my current implementation utilizing shm for cost-effective access to tic_mem. 😤

Lua is super popular (since it was the first language) so even fixing just it is a big win.

I disagree, fixing Lua alone would further push other languages towards being second-class citizens.

True, but I'm passingly familiar with Lua (having just patched it), Wren, Duktape, and WASM-3 and think all of them can do this fairly easily. We had it working for almost every language before but it was ripped out because then Lua hooks were too slow... we've never had it for Wren or MRuby if I'm not mistaken.

It's usually best to leave upstream intact whenever possible. Or maybe send them a patch, so at least they can comment.

But I'll try to drop off here and you can do back to discussing multi-threading. :-)

Hurr durr, it's not threads, it's processes :grin:

---

Nonetheless, threads would be the canonical way to fix this problem (and improve responsiveness in general), if we didn't care for kernel sandboxing [on Unixes (because Windows doesn't have any of that [yet])].

[joshgoebel]

I disagree, fixing Lua alone would further push other languages towards being second-class citizens.

I wasn't saying only fix it, just that we can fix them individually.

[peterhil]

I am also a FreeBSD user @remi6397 who would like to use TIC-80 on FreeBSD. Have you managed to compile TIC-80 from sources?

[peterhil]

I actually managed to build TIC-80 from sources, and opened pull request #1941 for it. @remi6397