Implement modules

[bookmoons]

Proposing a model for modules #5. Foundation for an implementation of publication of checkpoints to a blockchain #1.

The code uses the StandardJS style, since it was in the dependencies. If there's no objection I'm documenting with JSDoc.

General usage is like this:

const { Module } = require('openrealrecord/module')

const stream = createStream()
const module = new Module(stream, async function executor (controller) {
  for (const item of items) {
    await controller.write(item)
    if (controller.stop) return
  }
  controller.done()
})
module.start() // Module starts feeding

You provide an async executor procedure. It receives a controller that provides async 2 way communication back to the module. The executor signals when its done. The module can signal a stop request which the executor can watch for. A module can be stopped and restarted any number of times, since that seemed to be the intent in the wip code.

Subclasses can define their own internal executors that handle a certain type of data. The IterableModule implements for [Iterable]()s.

const { IterableModule } = require('openrealrecord/module')

const stream = createStream()
const data = [
  Buffer.from('1234', 'hex'),
  Buffer.from('5678', 'hex'),
  Buffer.from('ddff', 'hex')
]
const module = new IterableModule(stream, data)
module.start() // Feeds to stream until data is exhausted

My hope is to provide a solid foundation for defining new module types.

There are a lot of commits again. If you prefer I can squash them.

[bookmoons]

Proposing this as a from scratch replacement of the current timestamp code. I have an upcoming PR to implement hash publication on this model, if we can get this piece to something acceptable.

Something like this:

const { BitcoinPublishCheckpointModule } = require('openrealrecord/module')

const stream = createStream()
const bitcoin = createBitcoinConnector() // Carries node details, wallet details
const module = new BitcoinPublishCheckpointModule(stream, bitcoin)
module.start() // Watches for each checkpoint, publishes hash to chain, writes txid to feed

The publication proofs would end up written to an ancillary feed of some kind, as mentioned in comments.

[bookmoons]

The organization is like this.

Module <-- Relay <--> Controller --> Executor

The relay and controller are correctly entangled so that messages and capabilities can be exposed to the executor procedure without exposing unsafe members.

Some justification of a pattern here, since it can seem a little strange at first.

Revealing Constructor

This is a means of achieving protected members in JavaScript.

The construtor accepts an exposer procedure. It invokes the exposer with protected data and procedures. The caller stores them somewhere private. No other code has access to them.

This pattern is used in the Promise constructor. reference

[bookmoons]

It would be nice if each commit represented a conceptual change

Are you treating commits like parts of a patch set on the GNU lists? Maybe I can reorganize the commits into that model.

[xloem]

It would be nice if each commit represented a conceptual change

Are you treating commits like parts of a patch set on the GNU lists? Maybe I can reorganize the commits into that model.

Yes, something like that would be great.

@bookmoons it's so exciting to see the work you are doing here. Since you're doing a lot of work on this codebase, I was wondering if you had any comments on the other PR #10 . It changes the styling all around; would it get in the way of your work if I merged it in?

[bookmoons]

Yes, something like that would be great.

OK, will look at how to do that.

@bookmoons it's so exciting to see the work you are doing here. Since you're doing a lot of work on this codebase, I was wondering if you had any comments on the other PR #10 . It changes the styling all around; would it get in the way of your work if I merged it in?

Will look that over as soon as I can. Maybe I'll try merging and see what happens.

[xloem]

Yes, something like that would be great.

OK, will look at how to do that.

I find the easiest way to reorganize my commits to be git rebase -i

[xloem]

@bookmoons i'm still looking through things, to keep you in the loop a number of changes were going on in the wip branch that I'll need to make sure can be made in your work too:

• bugfixes: I foolishly integrated bugfixes into the wip branch and would need to cherrypick them out. it would be helpful if you could work off wip
• upgrading the node vm target and using newer language features for the library: I see you did this too. it would be nice for the whole project to adopt one style; I'll want to grok yours, and grok my old one, and maybe grok the other PR, and figure out what the plan is going forward
• additional module-related functionality, such as metadata: I guess I'm going to need to review my old work and make sure any concerns I had won't be lost when I adopt your new work. I was focused on a specific final end-product integrated into the module concept.

Your work is very compartmentalized from the rest of the library, and it partly makes me want to split the behaviors off into two different projects, which I understand is the preferred approach in the node.js ecosystem; but that's a bit much for me at the moment.

[xloem]

(@bookmoons but if you'd rather work on the cryptohashes bounty please do so even if I haven't merged this in; I'd still be happy to close it even if my concerns above are still pending)

[bookmoons]

Updated these commits to something cleaner. Tests pass after every commit.

Will go through all these comments soon.

[bookmoons]

(@bookmoons but if you'd rather work on the cryptohashes bounty please do so even if I haven't merged this in; I'd still be happy to close it even if my concerns above are still pending)

That is generous. If that works for you, I'll make it my first stop. Get to something functional on this model. Think I'm about halfway through it. Then we can pursue settling all these points for clean integration. I'm still interested after that. Happy to keep discussing in parallel.

it would be helpful if you could work off wip

OK, made a note to make this switch.

[bookmoons]

Your work is very compartmentalized from the rest of the library, and it partly makes me want to split the behaviors off into two different projects, which I understand is the preferred approach in the node.js ecosystem

I'd be glad to do it. If you wanted to open a new repository(ies), I could make pull requests there.

npm not long ago added scoped packages. @user/package. Large projects have been moving to this scheme. Babel has tons of scoped packages. Maybe it would work for modules.

@openrealrecord/module - Base module system. @openrealrecord/syslog @openrealrecord/mp3

Or if it's possible other groups of packages would be defined:

@openrealrecord/module-mp4 @openrealrecord/module-sensor @openrealrecord/module-message

[bookmoons]

it would be nice for the whole project to adopt one style

To describe what was used here. Still need to look over that TypeScript patch.

Style:

• StandardJS style. standard has good editor support. I have Atom linting while I type.
• Wrapped to 80 characters. Prefer vertical expansion to horizontal.
• Always name functions. This is an error reporting thing. If you have anonymous functions you end up with stack traces -- anon called anon called anon. It's difficult to see where the thing started. Named functions, even for function expressions eg as callbacks, keeps stack traces clear.
• Turn callbacks into promises at the earliest possible point. Since I went with async functions I transmuted to that as soon as possible.

Features:

The Babel plugins page happens to be a good list of features by JavaScript version.

Saw you had used async functions, so I went with that as the upper bound. They show up in ES2017. Probably worth confirming, but I think everything preceding was available well before async functions released.

[xloem]

@bookmoons, just wanted to note that it is kind of difficult for me to review the work because there are so many individual commits and so many files changed. anything that can be done to consolidate the changes into easy-to-understand chunks is helpful. very excited by what you have built !!

[bookmoons]

@xloem Thanks for looking at it. Is that about the piece for the hashes issue? I expect there may be some changes when I start running testnet trials, so I've been holding off. Will get things organized once that churn has settled down. And write up a technical description.