add "SES-adapter" package?

[warner]

@michaelfig and I spent the morning talking about how libraries and applications could/should get access to SES features like harden, Compartment, and the closely-coupled HandledPromise in diverse SES/non-SES environments. We have a proposal for a new package, provisionally named ses-adapter, and plans for how it should be used.

The only API of this package is:

import { harden, Compartment, HandledPromise } from 'ses-adapter';

Internally, this would use a global of the same name if one already exists, else it would use a shimmed version. It would also have a way to coordinate with separately-bundled copies of the package, to avoid identity discontinuities.

The goal is to support library and application authors who want to use these features (hardening, Compartment isolation/evaluation, and handled promises), independently of the final application's runtime environment. The operational library code that these authors write should be able to say import { harden } from 'ses-adapter' and then use harden(obj) in all the appropriate places. They should not have to load/install SES to acquire harden, nor should their desire to use harden impose a burden on downstream libraries or applications which import their harden-using code.

The decision about using SES is up to the final application. We imagine three environments in which this might run:

• non-SES: not ocap-secure, but harden enables good programming discipline, and the others are probably helpful too
• yes-SES: configured by the application startup code, ocap secure
• SwingSet

Applications that wish to run under SES should create a local module named install-SES or similar. This module should import and execute any vetted shims (which need to modify globals), after which it should import { lockdown } from 'SES' and call lockdown():

// install-SES.js
import { installShim1 } from 'shim1';
import { installShim2 } from 'shim2';
import { lockdown } from 'SES';
installShim1();
installShim2();
lockdown();

The application's main.js should then import from './install-SES' on its very first line. install-SES is imported solely for its side effects, which are to modify the globals and then freeze/tame them. All subsequent imports will see the SES "Start Compartment" which still has the usual platform authorities (fs, document, etc) as well as a mutable global object. As long as those modules don't attempt to modify the primordials in-place, they should not be significantly impacted by running under SES.

The idea of importing install-SES before anything else is to avoid relying upon the behavior of everything else: those second-run modules cannot modify the primordials. However, since the start compartment's global is still mutable, they could still replace the primordials with something surprising. Proper isolation between dependent libraries is a job for the upcoming module loader framework. But, we might consider recommending install-SES.js also do a harden(global).. I'm not sure. Proper isolation is easier to think about when you create a new Compartment for the untrusted code (and requires running under SES in any case).

When a library using harden/etc wishes to test outside of SES, they don't have to do anything special: ses-adapter will load a shimmed/insecure version of the functions they want to use. If they wish to test inside SES as well, the library should declare a devDependency upon SES, and the individual test-foo.js files that want to use SES (which are like mini-applications) should import a similar ../install-SES module before doing anything else. In particular, they must import from '../install-SES' before they import the code under test, so that it gets the real SES exports rather than the insecure shimmed versions.

The agoric-sdk bundle-source tool will declare SES-adapter as an "external" (an exit from the module graph), and thus will not incorporate SES-adapter or its dependencies into the bundled source object. The SwingSet require endowment will honor a require('SES-adapter') by providing a stub that reflects the SwingSet's host's SES properties. This provides a secure implementation of harden/etc for code loaded into a SwingSet environment (specifically code that gets used in a Vat, or in a contract).

harden, Compartment, and HandledPromise are all on the standards track. As JS engines start implementing them natively, libraries that use SES-adapter (in applications which did not somehow install-SES) will automatically switch over to the native+secure versions.

Applications which call lockdown will probably get native versions too, since I think our plan is for lockdown() to look for existing globals and use them if available.

When running under XS, SES-adapter will prefer the native versions. We might avoid importing SES and calling lockdown() there, or we might continue to import it and rely upon lockdown() noticing the existing Compartment/etc and doing nothing.

At some point in the future, both lockdown and SES-adapter will effectively be no-ops, since all runtime environments will have these features and/or be SES environments. At that point, library code might choose to delete the import { harden } from 'SES-adapter' lines from their files, but they will not have to change anything else.

I don't yet know what application code will do in this future utopia: their continued use of install-SES depends upon what the least-authority module loader looks like. We've talked about running programs under a SES-aware loader (perhaps ses main.js or ses app-manifest.json instead of node main.js). In that case, the construction/configuration of the SES environment would be the responsibility of the loader/app-launcher, rather than the first few lines of the application being loaded.

[warner]

Our initial sketch of SES-adapter looks like this:

import { harden as maybeHarden } from '@agoric/harden';
import { Compartment as maybeCompartment } from 'compartment-shim';
import { HandledPromise as maybeHandledPromise } from '@agoric/eventual-send';

// TODO: add magic to obtain 'globalThis'

// remember, under SES we cannot modify globalThis
const installed = globalThis.__SESAdapterInstalled || {};

let newHarden;
if (installed.harden) {
   newHarden = installed.harden;
} else if (typeof harden !== 'undefined') {
   newHarden = harden;
   installed.harden = harden;
} else {
   newHarden = maybeHarden;
   installed.harden = newHarden;
}

let newCompartment;
if (installed.Compartment) {
   newCompartment = installed.Compartment;
} else if (typeof Compartment !== 'undefined') {
   newCompartment = Compartment;
   installed.Compartment = Compartment;
} else {
   newCompartment = maybeCompartment;
   installed.Compartment = newCompartment;
}

let newHandledPromise;
if (installed.HandledPromise) {
   newHandledPromise = installed.HandledPromise;
} else if (typeof HandledPromise !== 'undefined') {
   newHandledPromise = HandledPromise;
   installed.HandledPromise = HandledPromise;
} else {
   newHandledPromise = maybeHandledPromise();
   installed.HandledPromise = newHandledPromise;
}

try {
  // outside of SES, notify any separately-bundled copies of SES-adapter of
  // our choices
  globalThis.__SESAdapterInstalled = installed;
} catch (e) {
  // inside SES, we ignore the failed attempt to modify the global
}

export { newHarden as harden,
         newCompartment as Compartment,
         newHandledPromise as HandledPromise,
       };

It depends upon exposing the Compartment shim as a separate NPM-published package (currently it is hidden inside the ses package).

One unfortunate downside of this approach is that we do the work of creating e.g. a harden instance (which has to walk the whitelist and add everything to the "fringe") all the time, even though under SES environments it won't get used. We could perhaps arrange for SES-adapter to import make-hardener instead, and only build a harden if necessary, but that sounds like more work. The performance hit only occurs once per application load, so perhaps it's not a big deal.

The larger impact might be if the shims cannot be run under SES. For example the compartment shim probably requires access to sloppy-mode to build the evaluator, which wouldn't work under SES. We won't ever call the compartment shim when under SES, but we must make sure that it can still be loaded.

[warner]

@erights @kriskowal: we'd love your thoughts on this. Also anyone else (@katelynsills ? @Chris-Hibbert ) who could think about this from the library author's point of view: if we want these features to be widely used, does this provide a gentle-enough progressive-deployment pathway?

[warner]

BTW this would replace imports of @agoric/harden and @agoric/evaluate in agoric-sdk code.

For basic string evaluation (with endowments), you'd use the Compartment API directly:

import { Compartment } from 'SES-adapter';

const c = new Compartment();
return c.evaluate(source, { endowments: { .. } });

For importing entire bundles (specifically the output of bundle-source), you'd use a new package we'd write named importBundle, which would import SES-adapter internally.

[erights]

How would such code work when run on the XS SES-only machine?

(Not that I see a problem. Just want to ensure that case is thought through.)

[warner]

I think SES-adapter would notice the presence of harden/Compartment/etc on the global, and return those values.

I'm not yet sure what the SES module or it's lockdown should do. Either the application knows it's been built into an XS binary and doesn't import it (SES wouldn't even appear in the XS module manifest), or it does import SES but lockdown knows that it's inside XS and does nothing. The former is probably easier but means the application has to be more aware of its environment, which to be honest it probably does anyways, especially given XS's other differences (more frozen, modules must be known at build time, having a "build time" at all).

[Chris-Hibbert]

This looks very comfortable to me. I think both the Shim and the longer term target would be quite useable, and it sounds like a fine transition plan. I like having a single evaluation pattern that works for multiple contexts.

[warner]

Oh, also, @michaelfig thought this could be related to a notional "Jessie standard library": what are the globals expected to be available to a normal Jessie program? This might justify why HandledPromise wants to be present, even though it's somewhat orthogonal to SES.

(We want SES-adapter to provide HandledPromise because it's an easy way to make sure everybody gets the same one)

[warner]

I've started work on this in https://github.com/agoric-labs/ses-adapter . I don't know that it wants to be in a standalone repo.. maybe it should live here in SES-shim, or maybe even in agoric-sdk (since it depends upon @agoric/eventual-send for HandledPromise).. ideas welcome.

[Chris-Hibbert]

I think within agoric-sdk isn't a great choice. Since we want people using SES outside SwingSet, I think it deserves a separate repo. Maybe moving eventual-send out with it to a separate repo is the right choice? How many external dependencies to @agoric code do we have from agoric-sdk now?

[warner]

I've added this to the agoric-sdk monorepo for now, we can move it to a better place later.

[warner]

published as https://www.npmjs.com/package/ses-adapter/v/0.0.1

[kriskowal]

We have since changed the nature of the SES-shim, which now provides its API as globals, like a traditional shim. I believe this conversation has concluded.