Refactor!

[dabbott]

The main goal of this refactor is to make it easier to iterate on the compiler.

The 2 main changes:

• Logic: I rewrote the Logic compilation phases in a more typescript friendly structure. Since these were previously ported from Reason, there was a bunch of stuff that didn't really make sense in TypeScript (e.g. massive switch statements in 1000-line files), so I focused on splitting things up. A few patterns I use I might change, but I'm overall happy with how it turned out.
• Testing: It was hard to keep everything in sync while making changes, so I moved some of the "out-of-band" operations into tests, e.g. generating example workspace snapshots, and generating types for the standard library. Tests now rely on memfs, which helps us easily isolate workspaces (e.g. we can spin up a separate workspace per fixture file) and test output paths.

Some other key changes:

• I split node traversal into its own library: https://github.com/dabbott/tree-visit
• The FS wrapper is now just an object that implements the fs interface. While the wrapper is probably smart long-term since it's small, the standard API is more convenient, since it does everything I want and I already have it memorized
• Most FS operations happen through https://github.com/dabbott/buffs, which supports both fs and memfs
• Anything internal that was async but didn't have huge perf consequences I made sync, for simplicity
• I only had to optimize 1 thing: the ShallowMap class (now MultiMap). On big workspaces, doing tons of deep comparisons was taking minutes! Everything else is plenty fast for now I think.
• I removed some any types in favor of unknown
• I removed some of the things in the helpers object, instead importing them directly for now.
• The Logic files we bundle in static are now in the .tokens code format instead of JSON... I can't remember if we need to ship them in JSON format, but if we do, let's include that in addition to the code version.

The plugins are mostly unchanged. For now I did "cheat" a bit in the plugins, relying on evaluating the standard library instead of having hardcoded code generation - but can revisit another time.

I think this is in a good spot for merging. There are still a couple things I still have to add back in, but nothing we currently rely on I think. The inputs, outputs, and options should be functionally the same.

Next, there's some stuff happening in @lona/serialization that I want to take a look at...

[mathieudutour]

Looking good overall

I moved some of the "out-of-band" operations into tests, e.g. generating example workspace snapshots, and generating types for the standard library

generating types for the standard library isn't a test tho

I removed some of the things in the helpers object, instead importing them directly for now.

Does that mean that external plugins needs to import them as well? If so it will probably create dependency conflicts?

I can't remember if we need to ship them in JSON format, but if we do, let's include that in addition to the code version.

I think it's just faster because we don't need to parse it at run time

For now I did "cheat" a bit in the plugins, relying on evaluating the standard library instead of having hardcoded code generation

So that's breaking code generation as soon as you use the standard library, no? I don't think we have polyfill for it

[dabbott]

generating types for the standard library isn't a test tho

Right, now we just test if they're implemented, rather than generate types. https://github.com/Lona/compiler/pull/12/files#diff-c5526ff31b163e2871ad37985cf545a1R4-R14

It's a little less strict, but much easier to work with. Plus we should be testing each anyway.

Does that mean that external plugins needs to import them as well? If so it will probably create dependency conflicts?

The main thing I removed was AST helpers. I think those should be top-level exports, and the helpers should be mostly data. I added the whole ModuleContext which includes the parsed workspace.

I did a quick grep of the android repo, and here's what I import directly (some just in tests):

// Nodes
import { FunctionDeclaration } from '@lona/compiler/lib/logic/nodes/FunctionDeclaration'
import { FunctionCallExpression } from '@lona/compiler/lib/logic/nodes/FunctionCallExpression'
import { IdentifierExpression } from '@lona/compiler/lib/logic/nodes/IdentifierExpression'
import { IExpression } from '@lona/compiler/lib/logic/nodes/interfaces'
import { ArrayLiteral } from '@lona/compiler/lib/logic/nodes/literals'
import { LiteralExpression } from '@lona/compiler/lib/logic/nodes/LiteralExpression'
import { FunctionParameter } from '@lona/compiler/lib/logic/nodes/FunctionParameter'

// Misc
import { findNode } from '@lona/compiler/lib/logic/traversal'
import { EvaluationContext } from '@lona/compiler/lib/logic/evaluation'
import { Value, Decode } from '@lona/compiler/lib/logic/runtime/value'
import { silentReporter } from '@lona/compiler/lib/utils/reporter'
import { createModule, ModuleContext } from '@lona/compiler/lib/logic/module'
import { UUID } from '@lona/compiler/lib/logic/namespace'
import { StaticType } from '@lona/compiler/lib/logic/staticType'
import { assertNever } from '@lona/compiler/lib/utils/typeHelpers'
import { Token, ColorValue, TextStyleValue, ShadowValue } from '@lona/compiler/lib/plugins/tokens/tokensAst'
import Tokens from '@lona/compiler/lib/plugins/tokens'
import { Plugin } from '@lona/compiler/lib/plugins'
import { Doc, indent, group, print, builders } from '@lona/compiler/lib/utils/printer'

// We use config and reporter - we'll use the new fs and module too
import { Helpers, createHelpers } from '@lona/compiler/lib/helpers'

Most of these things look like good top-level exports. If we run into a circular deps issue... maybe we consider what can come from its own package.

I think it's just faster because we don't need to parse it at run time

Ah ok. Let's go with source files then. The Swift projects have caching, and that's not a bottleneck in the compiler/plugins yet.

So that's breaking code generation as soon as you use the standard library, no? I don't think we have polyfill for it

I handle the evaluated Color, Shadow, and TextStyle specially: https://github.com/Lona/compiler/pull/12/files#diff-d59f3948b199a132191744340baf6454R443-R466.

Beyond that, we don't have anything in the standard library yet that can't be evaluated I think. Well, we need to revisit it, but I don't think it will break our stuff - the snapshots match pretty closely at least.