SwingSet on xs: exploratory prototypes

dckc commented 5 years ago

A key to Agoric's smart contract platform is compatibility between widespread understanding of best practices in JavaScript development with object capabilities and fail-stop deterministic execution. Initial development of cosmic-swingset is based on the popular and feature-rich node.js platform, but node.js represents an uncomfortably large amount of code to include in a trusted computing base as well as a substantial risk to deterministic execution. The xs engine in the Moddable SDK is designed for constrained microcontroller environments. It has some limitations, but the supported feature set seems to be an excellent match for the Agoric platform.

Issues (for reference)

NOTE: In more recent designs, only vat workers run on XS, not all of SwingSet. See #2107 for details.

Toward cosmic-swingset ag-solo on xs OBSOLETE

Current work is on ag-solo on xs. In particular, an xs-lite branch where I'm integrating lessons from earlier "hack and slash" mode work (xs-platform branch) without breaking the main build.

The pixel demo works thru getting a pixel balance but then loses its marbles with "key not found" stuff. I suspect / hope this is due to the promise queue priority issue #45.

Bootstrap documented in github actions

[x] github action workflow to test 2 packages
- [x] script to generate manifest.json, main.js (see #370)
- [x] github action: ag-solo-xs.yml
  - for example: build log for 817d413
    - [x] moddable SDK with x-cli-lin
    - [x] mainloop design #118
    - [x] moddable-linux-sdk.tgz release ag08 ff41561
    - build-release.yaml
    - see #118 for end-game
- [x] PR to add xs build to regular testing (see #373)

SwingSet unit testing on xs

A large part of the SwingSet test suite has been ported to xs:

platform	branch	date	rev	scripts	tests	passing
node.js	master	Jan 17	f833610831e6	20	755	755
xs	xs-build-lite	Jan 18	5df242c2	12	594	594

2 scripts are skipped due to the queue priority issue (#45)
1 is skipped because the build-bundle.js tool I cribbed together doesn't grok ~. yet
5 are skipped because pre-bundling is incomplete

For more details, see Jan 18 comment below.

tape testing work-alike: xs is ESM-only

One of main differences between node.js and xs is modules: xs supports only ES5 modules. Attempting to port tape showed extensive CommonJS dependencies. Incremental development of a work-alike, https://github.com/dckc/tape-xs , has worked reasonably well so far.

[x] tape initial port: https://github.com/dckc/tape-xs
- [x] basics: equal, is, ok, notOk
- [x] deepEqual, async
- [x] count assertions
[ ] tape-xs issue: #370
[ ] #49 test scripts -> modules? actually, no...

Using xs deep freeze for `harden()`

The @agoric/harden and @agoric/make-hardener code do extensive patching in a browser or node context to reach a state that seems to be essentially the default in the xs platform. xs has a deep freezing feature: Object.freeze(x, true) has an extra argument that, when set to true, seems to do the equivalent of harden(x). Using deep freezing passes all the @agoric/harden tests but one.

[ ] https://github.com/Agoric/harden/issues/54 why complain about prototype not in roots?
[ ] https://github.com/Moddable-OpenSource/moddable/issues/303 non-determinism of %GeneratorFunction%[[Prototype]]
- https://github.com/Agoric/harden/issues/47

dckc commented 5 years ago

Sharing import path specifiers

main issue: https://github.com/Agoric/agoric-sdk/issues/57

In trying to demonstrate that sharing import paths was no just difficult (https://github.com/Moddable-OpenSource/moddable/issues/251) but impossible, I discovered it's not.

IOU details, but this looks promising. The main impact is that test scripts should become modules (https://github.com/Agoric/agoric-sdk/issues/49 e.g. 8bf4116679d5)

Also: the test-xs-manifest.json and test-xs-main.js files are in the top SwingSet directory.

test-xs: 14/14 PASS from test-marshal https://github.com/dckc/SwingSet/tree/xs-platform 40ee4a0

Earlier approach

earlier status: 12 / 16 PASS - 4 FAIL. description: https://github.com/dckc/SwingSet/tree/test-kernel-xs/test/test-kernel-xs 2784e5b

dckc commented 5 years ago

All 174 tests from test-marshal and test-kernel are passing with no change to the src/ code and small tweaks to test/ code.

https://github.com/Agoric/SwingSet/commit/643f37dd7c856f074ee24406be96cc6acfb4bf82 since rebased to 5e6c533f4b2d9f7df6439bfbe38d91e2497df881

Some dependencies are tweaked:

npm link @agoric/harden https://github.com/dckc/harden/tree/deepFreeze-xs 7dbc995
npm link @agoric/eventual-send https://github.com/dckc/eventual-send/tree/delay-weakmap-xs bd2be3f (merged)
npm link @agoric/tape-xs https://github.com/dckc/tape-xs 038d62d

Noteable changes

0023bb2 2019-09-02 test-marshal: declare Promise.makeHandled dependency
2f61a35 2019-09-02 turn test-marshal script into module
c0df6e7 2019-09-02 xs-platform work-alike for console.log
3c82696 2019-09-02 xs platform work-alike for setImmediate etc.
26b90d5 2019-09-02 test-xs: 14/14 PASS from test-marshal
74eb464 2019-09-02 xs: push harden, eventual-send changes down
4ed2437 2019-09-02 turn test-kernel into module
495a7ee 2019-09-02 xs: add kernel subdirectories to manifest
5e6c533 2019-09-02 xs: run test-kernel too (174 passing out of 174)

Moddable SDK: Getting Started

Prerequisite to this work is Host environment setup for linux (or for macOS).

At this point, one can run the hello-world example along with the xsb debugger:

~/projects/moddable$ cd examples/helloworld/
~/projects/moddable/examples/helloworld$ mcconfig -d -m -p lin
~/projects/moddable/examples/helloworld$ # or: mcconfig -d -m -p mac
# xsc Resource.xsb
...
# ld mc.so

xsb-hello

eventual send operator syntax

i.e. acornInjectPlugins: [infixBang()] (though infixBang is now a misnomer... IOU pointer to discussion of target~.msg(Args))

Should I ask the moddable / xs guys about supporting that? p.s. done: asked for advice on how to go about it:

[ ] https://github.com/Moddable-OpenSource/moddable/issues/261

bundling

Getting Babel to run in xs would have various grungy issues, but moreover, do we really want that in the TCB?

I'd be surprised if xs has a runtime module loader... They have an example using import(), but it does't parse js at run time; it loads ahead-of-time compiled stuff.

A quick-n-dirty rollup work-alike shouldn't be too hard, if my understanding is correct... all the imports have to go at the top of the file and export has to be a top-level token.

p.s. @dtribble points out @michaelfig 's work on https://github.com/Agoric/transform-module , which looks like just the thing!

[ ] SES on xs https://github.com/Agoric/SES/issues/16
[ ] __dirname, fs access https://github.com/Agoric/agoric-sdk/issues/66
- [ ] https://github.com/Agoric/agoric-sdk/issues/15 bundleSource is ambient authority
  - [ ] https://github.com/Agoric/agoric-sdk/pull/371

dckc commented 5 years ago

test-controller: 26 / 32 PASS; 6 FAIL

so the total, with test-marshal and test-kernel is 199 / 205 PASS; 6 FAIL

All six failures are the withSES === true cases.

The current withSES === true takes the kernel source, loads it as a function, and then stringifies it again, an then uses eval to turn it back into a function inside an SES Realm. In xs, the string form of the kernel isn't handy when you load it as a function. Straightforward options are:

take the kernel source and eval it in an SES Realm, without loading it in the outer realm
just import the kernel module into an SES Realm

The withSES === false code just imports the kernel module. It imports it into the outer realm, so I should add a new Compartment() around it. I'd have more confidence in doing that right if we had a few tests that shows why the withSES === false case is insecure (#151). (Same applies for eventual send operator: Agoric/SwingSet#150)

If it's important, I should be able to finish up the SES stubs to get withSES === true to run, if we skip loading the kernel source code into the outer realm and just deal with it in string form until it gets evaled in the SES realm.

I cheated and used ahead-of-time compilation for the vat code too. I aim to sync with @michaelfig on transform-module to address that.

Details of the test run: https://gist.github.com/dckc/dc7e560bbb2b6b989653b62445f38acf

as noted above, it involves npm link in 2 or 3 cases. It also involves a symlink, since the xs loader requires the .js extension:

~/projects/agoric/SwingSet$ ls -l test/vat-controller-1.js 
lrwxrwxrwx 1 connolly connolly 16 Sep  8 15:42 test/vat-controller-1.js -> vat-controller-1

Changelog (edited)

a3668d8 2019-09-08 require(), require.resolve() KLUDGEs
ea55849 2019-09-08 __dirname KLUDGE
f730661 2019-09-08 path stubs: isAbsolute, resolve
1a3adda 2019-09-08 fs shim: readdirSync, statSync
5e831ab 2019-09-08 SES stub: throw
705935e 2019-09-08 evaluate-options stub: default function
f0170f5 2019-09-08 manifest entries for controller modules
a3ee7f7 2019-09-08 require shim: preload @agoric/harden; filter user modules
0192471 2019-09-08 test-controller: turn off SES in 'load empty' (WIP)
586ce02 2019-09-08 noop module for making compartments

dckc commented 5 years ago

module loading, xs platforms, and `fxFindModule`

While the xs js-level Component API seems to handle only ahead-of-time compiled modules, I see the moddable SDK makes it pretty straightforward to define a platform including a C level fxFindModule hook: "Finding modules can involve looking for various kinds of files, using a set of preferred locations, etc. But on microcontrollers, all modules modules are prepared".

I'm not sure fxFindModule would allow async module lookup, though.

[ ] https://github.com/Agoric/agoric-sdk/issues/57 struggling with import specifiers

dckc commented 4 years ago

Struggling to reproduce my own results away from my desktop: 163 / 173 PASS

As noted above, at my desktop, I have 195/201 tests passing. After remembering a couple more work-arounds below, I'm up to 163/173. If I run just one of the test modules at a time, I get:

testMarshal: 63/63
testKernel: 106/106
testController: 7/17

linux platform crept into npm script

The test-xs script specified -p lin. On a mac, it should be -p mac.

875ab0f split test-xs by platform

xs only recognizes files as javascript modules if they end in `.js`

work-around:

SwingSet/test$ ln -s vat-controller-1 vat-controller-1.js

(IOU issue, docs, and/or fix)

failed to open directory

next challenge:

# bootstrap with SES
= loading config from basedir ./test/basedir-controller-2
.../projects/moddable/modules/files/file/mac/modFile.c (245) # Exception: Iterator: failed to open directory!

See https://github.com/Agoric/SwingSet/issues/180

tape / harden integration

harden$ git diff
diff --git a/package.json b/package.json
index 4754ead..0ddb906 100644
--- a/package.json
+++ b/package.json
@@ -36,7 +36,6 @@
     "prettier": "1.16.4",
     "rollup": "^1.3.2",
     "rollup-plugin-node-resolve": "^4.0.1",
-    "@agoric/tape-xs": "^0.1.0",
     "tape": "^4.9.2"
   },
   "directories": {

harden$ ls -l package.json 
-rw-r--r--  1  1338 Oct 26 15:04 package.json

dckc commented 4 years ago

@warner did some xs in C work back in Dec 2018 https://github.com/agoric-labs/moddable/tree/warner-linux-cli ddf4a43

Since then, moddable seems to have made an independent git history.

I re-sync'd BW's work: https://github.com/Moddable-OpenSource/moddable/compare/public...dckc:ag-linux-cli 3346e1a

(what happened to my notes from last night??? oh: https://github.com/Agoric/SES/issues/16#issuecomment-557375740 )

dckc commented 4 years ago

I got the kernel working inside the little xs cli program.

I'm not sure where to put stuff yet, so for now, it's a gist:

https://gist.github.com/dckc/8b0655faa70226e741b8f09a938840f4

dckc commented 4 years ago

@warner you asked how SES-like xs is. I think https://github.com/Moddable-OpenSource/moddable/blob/public/documentation/xs/preload.md#automatic-freezing-of-built-ins is the answer. Is there some place (else) I should document it?

dckc commented 4 years ago

ag-solo builds a withSES=true kernel, starts to load devices

it trips over import ... in device source. Gotta bundle those up.

At least this much is checked in...

https://github.com/dckc/cosmic-swingset/tree/xs-platform 5f07a15 https://github.com/dckc/moddable/tree/ag-linux-cli 1c04cb8 https://github.com/dckc/SwingSet/tree/xs-platform 6479fde

dckc commented 4 years ago

ag-solo runs thru `run() done.`; does it work?

@warner @dtribble How close is this to working?

https://gist.github.com/dckc/dee8ca7e527e018bd3d03ec5e540bbbd

cosmic-swingset: 1ea6a96 SwingSet: 616104c moddable: 1539a49 https://github.com/dckc/bundle-source/tree/allow-ocap 7946ff3 moddable 1539a492

and perhaps a few other local kludges. (in particular, all the vats and devices are rollup'd with filename conventions)

dckc commented 4 years ago

ag-solo on xs caught up with monorepo etc.

https://github.com/dckc/agoric-sdk/tree/xs-platform 757c337

run log: https://gist.github.com/dckc/a5adf370abea5ab88a86a912e98e625d

I automated "all the vats and devices are rollup'd with filename conventions" in ag-solo-xs.mk

dckc commented 4 years ago

toward github action for ag-solo on xs

My desktop gets more and more full of kludges. I haven't managed to coordinate with anyone else to reproduce my results, so let's try getting a robot to do it.

I got as far as one of the manual steps in my makefile: check out master and make scenario3-setup

https://github.com/dckc/agoric-sdk/commit/d211823d7c20897aec2dbb60b95fd52c80edf18f/checks?check_suite_id=370981243

https://github.com/dckc/agoric-sdk/blob/ag-solo-xs-action/.github/workflows/ag-solo-xs.yml d211823

hanging at zoe vat

Ugh. After addressing various symptoms that I have seen locally, it's doing something I don't recall seeing before. I'm kinda stumped about where to go from here.

https://github.com/dckc/agoric-sdk/commit/d69140dc480199afab7eee857e2320b47b949a19/checks?check_suite_id=371898943

heisenbug: `Cannot pass non-promise thenables`

After I filled in the makeEvaluators part of my ses port (9ce30d7), I started to get:

doProcess: vat[v15][o+0].bootstrap dispatch failed: Error: Cannot pass non-promise thenables {}

I worked on debug logging enough to learn that it was from the copy of marshal in bootstrap, and then set things aside yesterday. Now today I can't reproduce the problem. (Meanwhile, it looks like hp.resolve became HandledPromise.resolve in marshal 0.1.2, but that version isn't referenced in all the relevant places. (yet?))

Hence the motivation to get this stuff running somewhere other than my desktop.

erights commented 4 years ago

Hi @dckc the two are related. A bug in HandledPromise was causing handled promises not to be recognized as genuine promises, and therefore misclassified as non-promise thenables. Marshal, to avoid passing things with problematic behaviors, forbid non-promise thenables.

dckc commented 4 years ago

github action for xs-lite

I'm trying a github action again with less invasive edits to moddable and agoric-sdk

So far: my version of moddable builds. I hope it got cached...

https://github.com/dckc/agoric-sdk/commit/88e8a532d321c35acd43dca7811c1e9c8a68d595/checks?check_suite_id=374897939

I got the eventual-send tests built; I think it runs, but since it never exits, the output doesn't seem to make it to the logs. Maybe I should make a separate issue for this event loop stuff... https://github.com/dckc/agoric-sdk/runs/364252413

p.s.

github action for 2 packages on xs (eventual-send, marshal)

Now that I got some moddable SDK stuff worked out ( https://github.com/Agoric/agoric-sdk/issues/118#issuecomment-569178998 ) the tests for these two packages run pretty cleanly:

https://github.com/dckc/agoric-sdk/blob/xs-lite/.github/workflows/ag-solo-xs.yml https://github.com/dckc/agoric-sdk/commit/817d413c65488ca4543f3fb192fa3ef73a935044/checks?check_suite_id=375331754

[ ] report exceptions from main() in lin_xs_cli.c

dckc commented 4 years ago

@dtribble hot off the press: https://github.com/dckc/agoric-sdk/tree/xs-build-lite

ag-solo works correctly on xs in the case of ERROR: . doesn't appear to be an ag-solo base directory 🙂

dckc commented 4 years ago

ag-solo start seems to work on xs modulo web stuff

baa8348baa8348

run log: https://gist.github.com/dckc/5f2b521687f4fd022bc3dae30ec1c565

express, morgan

8e9b9f4 has express (and serve-static), morgan, and http

next: WebSockets

dckc commented 4 years ago

Got a screen full of colored pixels!

It actually works sometimes, in the x-cli-lin build; I haven't seen it work in the debug setup.

next: async http route methods

The moddable http Server API seems to require responses to be computed synchronously. I got away with that for static files, but the put('/vat') handler is async, so I'm just getting lucky somehow when it works.

websocket handshake working

I get ws.open! in the browser console.

Hmmm... now I'm getting: main.js:30 WebSocket connection to 'ws://localhost:8000/' failed: A server must not mask any frames that it sends to the client.

e7b49bcce751

dckc commented 4 years ago

SwingSet unit tests up to 594 / 594

CI build as of 5df242c: https://github.com/dckc/agoric-sdk/commit/5df242c22e894259b1caf65830d2b0030b69bf20/checks?check_suite_id=408373914

[x] test-comms {"pass":4,"fail":0,"total":4}
[x] test-controller {"pass":65,"fail":0,"total":65}
[ ] test-demos-comms SKIP: build-bundle.js does not grok ~. yet
[ ] test-demos SKIP: test bundling WIP
[ ] test-devices SKIP: test bundling WIP
[x] test-evaluate {"pass":69,"fail":2,"total":71}
[x] test-kernel {"pass":179,"fail":2,"total":181}
[x] test-liveslots {"pass":188,"fail":2,"total":190}
[x] test-marshal {"pass":202,"fail":2,"total":204}
[ ] test-message-patterns SKIP: test bundling WIP
[ ] test-node-version SKIP: see queue priority issue https://github.com/Agoric/agoric-sdk/issues/45
[x] test-promises {"pass":263,"fail":2,"total":265}
[ ] test-queue-priority SKIP: see queue priority issue https://github.com/Agoric/agoric-sdk/issues/45
[x] test-state {"pass":275,"fail":2,"total":277}
[x] test-timer-device {"pass":478,"fail":2,"total":480}
[x] test-transcript-light {"pass":537,"fail":2,"total":539}
[x] test-transcript {"pass":581,"fail":2,"total":583}
[x] test-vat-imports {"pass":588,"fail":0,"total":588}
[ ] test-vattp SKIP: test bundling WIP
[ ] timer-device/test-device SKIP: test bundling WIP

my local test log (with evaluate fix 3cdd017 and ses DEBUG turned on): https://gist.github.com/dckc/f600ecbc714b5b30c13924a2b0466ca1

dckc commented 4 years ago

My work on this issue is mostly standing by while the new SES / Compartment API are integrated into SwingSet; see #477

I have done a bit of work to keep my branch(es) up to date with master and I have worked with @michaelfig a bit on test debugging.

dckc commented 4 years ago

made a little progress on new-SES:

7ac25880 * toward SwingSet/test on xs: build, link
53cb51ab * SwingSet/test: test script compartment dependencies
21f01a3d * xs-npm as of 11dd443 Jan 18, 2020
350835b8 * SwingSet/test: toward test script Compartment API map (WIP)

https://github.com/dckc/agoric-sdk/tree/xs-new-ses

dckc commented 4 years ago

I sat down to resume work on this today given recent progress on new-SES etc. but re-discovered ambient file access in SwingSet/src/controller.js and lost momentum more or less right away. Sigh.

dckc commented 3 years ago

@dtribble @warner I'm inclined to change the scope of this to exploratory prototypes and close it in favor of #2107 and related stuff. OK?

I just reviewed the whole thing and I'm confident that there is no outstanding work represented only in this ticket.

dckc commented 3 years ago

@warner agreed with closing this.

Agoric / agoric-sdk