[META] Intent to Publish, Document and Extract Infrastructure Packages

[wycats]

At the moment, there's a lot of nice infrastructure inside of the workspace directory, and much of it would be broadly useful to people not working on Starbeam.

In an effort to streamline the repository and make it more approachable and sustainable, I intend to begin publish, document and extract useful infrastructure packages into standalone projects. I intend to do all of the work on this extraction before the new year.

In general, there are four basic strategies that could apply to workspace infrastructure:

1. Leave it alone. For example, I think this makes sense for our in-repo eslint plugin.
2. Migrate to existing ecosystem infrastructure. We did this early on with pnpm and jsonc-parser, and more recently have started to use Turbo for more of what we were doing in bespoke ways before.
3. Extract the package into its own standalone project (either inside of the Starbeam org or outside). This means: a. Establishing clear API boundaries around the package. b. Writing tests for the package. c. Documenting the package, at least via a readable, standard README. d. Publish the package to npm and begin depending on the published package. e. Extract the package into its own repository.
4. Decide that even though the infrastructure is nice, we're gonna cut it loose so we can avoid losing focus.

To be honest, I intend to do some extractions because I am happy with the code (and frequently types) and want to use it in other projects.

A really good example of this is the paths package, which provides an API for interacting with paths in a well-typed way (differentiating between regular files, directories and globs). I also did a lot of work on the debug display of the paths.

Is this strictly important for Starbeam progress right now? Not directly. But since I want to use this package elsewhere, I intend to the admittedly larger amount of work to do the package extraction (instead of opting for Option (4) and using a path library without the nice type and debugging bells and whistles).

After the 2024 New Year, I intend to break these ties in favor of Option 4 (i.e. cutting things loose).

With the pontificating out of the way, here's the list of infrastructure I'm going to be burning through:

• [ ] @starbeam-workspace/paths - A typesafe, debuggable path library (needs tests and documentation, but is fairly complete and nice)
• [ ] Display, Diagnostics and friends - This small library makes it easy to provide nice debugging output in node (see Display and Diagnostics below).
• [ ] @starbeam-workspace/edit-json - As nice as jsonc-parser is (and edit-json depends on jsonc-parser), it's very difficult to effectively work with jsonc-parser to do normal code changes ("replace this key or array element", etc.). And there are bugs in the high-level APIs that frequently fail to correctly replace the right ranges. edit-json is a higher level library that's built on jsonc-parser and provides interfaces for modifying jsonc-parser nodes. As with paths, a lot of the work of edit-json has been around nice debugging. This library has some tests, but needs documentation.
• [ ] @starbeam-workspace/package and @starbeam-workspace/workspace - these are higher-level packages for interacting with workspaces and packages, their paths, scripts, etc. It builds on pnpm's CLI, adding type safety and ergonomics. These packages need tests and documentation, and workspace has accumulated a lot of unrelated infrastructure that would need to be pruned out.
• [ ] Logging and Reporting - these packages make it ergonomic to shell out to npm scripts and report on what's going on in a way that groups appropriate things together. It has support for tables, lists and other conveniences, and automatically handles error reporting and no-output cases (e.g. when you create a table, you can specify what should happen if there are no rows -- when you log the output of an external command, you can specify what should happen if the external command didn't output anything, etc.).
• [ ] Development and Package Querying Commands - This is the infrastructure that makes it easy to build script commands that work across all of the packages in the repository, automatically supporting package scoping for every command. It builds on all of the infrastructure above. Personally, I like the idea of having some control over development commands without having to fully buy in to an entire monorepo framework. It works well with Turbo, and I think that if we extracted this into its own project, it would be something marketed as complementary to Turbo, not a competitor.
• [ ] Templating facilities - This is the infrastructure that makes it easy to template out new packages in the monorepo and quickly roll out changes to package.json, .eslintrc.json, tsconfig.json, etc. This makes it plausible to have repo-wide policies that (unfortunately) depend on having a snippet of code in every package.json etc.

I'll update this over the next few weeks as I inventory the infrastructure.

There's a lot there, and I have no illusions that the answer to every piece of infrastructure is going to be "carefully pull it out into its own project." My primary goal with this short-term effort is to reduce the codebase's reliance on me personally that is driven by all of this workspace code.

I'm also very interested in getting help, once I have a clearer sense of what we plan to do with all of the individual pieces. If you're interested in a particular piece of infrastructure and want to help make it work as its own project, comment here or reach out! If there are folks interested in specific pieces of infrastructure, that makes it a lot easier to decide to invest in extraction.          

Display and Diagnostics

The basic idea is that it should be easy to create rich console.log output for library classes using the regular Symbol.for("nodejs.util.inspect.custom") extension point.

This example was printed by passing an instance using Display in a custom inspect method to a normal console.log:

An involved of DisplayStruct in the Preact codebase is:

class InternalComponent {
  // ...
  [Symbol.for("nodejs.util.inspect.custom")](): object {
    const propsFields: Partial<{ props: unknown }> = {};
    const stateFields: Partial<{ state: unknown }> = {};

    if (objectHasKeys(this.#component.props)) {
      propsFields.props = this.#component.props;
    }

    if (objectHasKeys(this.#component.state)) {
      stateFields.state = this.#component.state;
    }

    return DisplayStruct(
      "InternalComponent",
      {
        vnode: this.vnode,
        context: this.context,
        ...propsFields,
        ...stateFields,
      },
      { description: String(this.id) },
    );
  }
}

Diagnostics make it easy to present annotated snippets of source code. You can mix diagnostics with Display features as seen above.

[wycats]

Here's an example of what I mean when I say I've invested a lot in the debugging output of the path library:

This is the implementation of the display:

And all I did to make the nice output was regular console.log. Since I happened to have already given the objects nice debugging output, it just worked.

(The path library doesn't require you to make particular use of roots or workspace roots, but if you build your paths with roots in mind, it's really easy to get the right relative paths when you need them in code, and also really easy to get nice display output that is automatically relative to the user's context).

[sandstrom]

You are a god-level programmer Yehuda! 😄 (I'm a huge admirer since Merb!)

But there hasn't been any updates on Starbeam (from Embers perspective) for a while. I check this repo from time to time (and follow you on Twitter), to track progress.

This extraction sounds clever, but doesn't seem immediately related to the progress of Starbeam, or Ember's chances to survive as a viable framework.

My understanding is that the strategic plan was to build a renderer to share with other frameworks, so that Ember doesn't have to carry the burden of maintaining it's own (similar to how Embroider is trying to adopt common JS tools for building/bundling). A good move!

If there is a roadmap or plan for Starbeam (in general, i.e. for other frameworks, and as a building block for Ember), I think me and others are eager to hear them.

[wycats]

@sandstrom The primary purpose of this meta-issue was to reduce the bus factor, not simply to do something interesting for the sake of it. I've since announced that I joined Heroku, which is a bit of context for that goal.

136 finishes the work of stripping out any remaining custom infrastructure that wasn't paying its way.

For reference, while Glimmer is Ember's rendering layer, Starbeam is intended to be an extraction of Ember's reactivity system that works with other frameworks.

A roadmap is a great idea!

Briefly (and just my personal opinion), I think we should start by scoping Starbeam 1.0 to the reactive data aspects and get it shipped.

Starbeam resources are nearly done and make sense as a fast-follow to Starbeam 1.0. That said, getting them properly aligned and ergonomic across the frameworks we intend to support has slowed down the reactive data part of the design, which has been done for over a year and isn't significantly changing.

[sandstrom]

@wycats Congrats on the new job!

Sounds awesome that you'll be able to continue working on Ember (at least partly) in your new role. 🎉