[RFC/discussion] tools API design

[43081j]

Copying most of this from my notes.

Motivation

We should provide the core functionality already duplicated across the various CEM projects in the wild.

Proposed design

It seems we need a few areas of utilities:

• Type guards (e.g. isClassDeclaration)
• Traversal functions (various find-like functions mostly)
• Assertion functions (e.g. exportHasCustomElementExport)
• Getters (e.g. getAllExportsOfKind)

Type guards

• isJavaScriptModule
• isJavaScriptExport
• isCustomElementExport
• isDeclaration
• isCustomElementDeclaration
• isClassDeclaration
• isFunctionDeclaration
• isMixinDeclaration
• isVariableDeclaration
• isCustomElement
• isCustomElementDeclaration
• isCustomElementMixinDeclaration

Traversal functions

• findParent(manifests, node, assertion)
  • Finds the first parent of the specified node which matches the assertion function
• find(manifests, node, assertion)
  • Finds the first child which matches the assertion function
• findAll(manifests, node, assertion)
  • Finds all children matching the assertion function

TBD

These traversal functions would be paired often with the type guards and assertion functions.

For example, find(package, null, (n) => isJavaScriptExport(n)).

Assertion functions

TBD

Getters

TBD

Prior art

CEM analyzer

• getAllExportsOfKind
  • Filters the exports by a particular kind
• getAllDeclarationsOfKind
  • Filters the declarations by a particular kind
• getInheritanceTree
  • Gets the inheritance hierarchy of a given class name, as an array of class names
• getModuleFromManifests
  • Finds a particular module by path in a set of manifests
• getModuleForClassLike
  • Finds a particular module's path which contains a class or mixin with the specified name
• getClassMemberDoc
  • Finds a class by name within a given module, then finds a member by name within that class.

CE language server

• findClassForTagName
  • Finds the class from a set of manifests which has the specified tag name associated with it
• findCustomElementDeclarationFromModule
  • Finds the first custom element class declared in a given module
• findDeclarationForTagName
  • Finds the custom element declaration from a set of manifests which has the specified tag name associated with it
• findCustomElementTagLike
  • Finds a registered tag which contains the specified string
• getCustomElementTags
  • Finds all the registered tags from a set of manifests
• findCustomElementDefinitionModule
  • Finds the module from a set of manifests which registered a given tag name
• findTagNameForClass
  • Finds the tag name associated with the specified class name from a set of manifests
• moduleHasCustomElementExport
  • Determines if a module exports a custom element
• exportHasCustomElementExport
  • Determines if an export is a custom element export
• moduleHasCustomElementExportByName
  • Determines if a module exports the specified tag name
• exportHascustomElementExportByName
  • Determines if an export is a custom element definition with a particular name
• findModuleByPath
  • Finds a module from a set of manifests which has the specified path
• modulePathEquals
  • Determines if a module's path matches a given path
• isCustomElementDeclaration
  • Determines if a declaration is a custom element declaration

webcomponents.org

• getCustomElements
  • Finds all custom elements exported by a given package, returning their export and declaration amongst other things
• getModule
  • Finds the module in a given package which has a path matching the one specified
• isClassDeclaration
  • Determines if a declaration is a class declaration
• isFunctionDeclaration
  • Determines if a declaration is a function declaration
• isMixinDeclaration
  • Determines if a declaration is a mixin declaration
• isVariableDeclaration
  • Determines if a declaration is a variable declaration
• isCustomElement
  • Determines if a declaration is a custom element declaration or a custom element mixin declaration
• isCustomElementDeclaration
  • Determines if a declaration is a custom element declaration
• isCustomElementMixinDeclaration
  • Determines if a declaration is a custom element mixin declaration
• resolveReference
  • Resolves a Reference to an actual Module from a set of manifests

Open questions

Type guards

• do we want to do what typescript did here and basically provide an is* function for every possibly type? e.g. we'd also have isClassMethod, isAttribute, etc etc.

Traversals

• do we want these functions to consume (manifest, nodeOrCallback, callback)? im not such a fan or these messy overloads. but (manifest, node|null, callback) isn't much better. as in, you could traverse a manifest for something, or you could traverse a node within a manifest for something
• of course everything can be simplified to find/findAll, but do we also want to provide some higher level helpers? like findExports, findDeclarations, findElement, etc etc. we probably do otherwise this package won't reduce boilerplate/duplication much

cc @justinfagnani @matsuuu @rictic

[justinfagnani]

Thanks @43081j !

I think a first and high-level question we need to answer is what kind of package organization we want.

I can think of a few broad categories:

• Utilities for using raw manifest data (getElements(), etc)
• Validation
• Manifest loader interfaces
• fetch-based loaders (ie, from npm registry + unpkg)
• Local filesystem-based loaders (uses require.resolve() or similar)
• Other I/O based loaders?
• Sites?*

The first three categories may differer in scope/purpose, but should be environment agnostic. They could be three packages, or a single one. The lines could blur a bit: resolving references across manifests might require a loader. The validator could cleanly be a separate package, though it's not strictly necessary.

The next three would differ in dependencies and environments: a local loader might have Node-specific API requirements.

What I put up in #1 and #3 are separate environment-agnostic utilities package and a validator package, taken from the webcomponents.org work, but these could be re-arranged as we like...

• I could imagine something like a manifest validator site, but that might not need to be in this monorepo.

[43081j]

i'd be tempted to do it the same way babel did

babel has @babel/traverse, @babel/types, @babel/parser, etc etc.

so in our case we'd have:

• traverse (utilities for find/get/etc of a manifest, incl traversing across multiple manifests)
• type guards/assertions (utilities for asserting what type an arbitrary node is, and if it has particular properties, etc)
• validation
• manifest loaders (one package each)

i guess my original post here is really the first two of those

[Matsuuu]

i'd be tempted to do it the same way babel did

babel has @babel/traverse, @babel/types, @babel/parser, etc etc.

so in our case we'd have:

• traverse (utilities for find/get/etc of a manifest, incl traversing across multiple manifests)
• type guards/assertions (utilities for asserting what type an arbitrary node is, and if it has particular properties, etc)
• validation
• manifest loaders (one package each)

I agree that this approach could work the best. A clear split between the packages for different use cases.

Wether the loader should be 2 packages or one, I don't know yet.

Having an API like typescript where you can create your own IO layer could work but at the same time, we could just write those 2 implementations ourselves since they will differ quite a lot

[justinfagnani]

I'm not sure what we really gain from separating traverse* from type guards. I don't see a downside to keeping them in on e package, but there is a downside to making too many packages - and of need future utilities that don't fit cleanly into traverse or types, requiring another one.

• I'd call it something different because getting all elements, or a module by name isn't really traversing, and this package wouldn't be an AST visitor.

With the goal of separate loader packages being to divide the be environment requirements and dependencies, I think we need a spot for just the loader interface. That could go in its own package, or the general tools package.

Having an API like typescript where you can create your own IO layer could work but at the same time, we could just write those 2 implementations ourselves since they will differ quite a lot

This is the idea with the loader interface. Something like the WC catalog needs to load manifests from an online registry, a VS Code plugin needs to load it from disk.

[43081j]

you're right i think, so we should have a package that contains all the traversal/getters/assertions/etc.

which i guess is the tools package you created in the other PR.

the interface of that package is what im trying to define in the original post

[justinfagnani]

I just merged #1 and #3 that make separate validator and tools packages (though the validator package is empty), but I wonder if I should merge those and add the loader interface there, then I/O-specific packages can come next.

[43081j]

i feel like the validator package should stay separate, just to keep it focused.

though maybe "tools" might end up too generic? bit of a grey area as that package could easily end up bloated with random utils if we're not careful

[bennypowers]

I'd like an object oriented API that I could deliver to 11ty etempaltes:

<template webc:nokeep
          webc:for="declaration of 
                      cemPackageNameMap
                        .get(package)
                        .getDeclarationsForModule(module)">
  <cem-class webc:if="declaration.kind === 'class'" :@declaration="declaration"></cem-class>
  <cem-mixin webc:if="declaration.kind === 'mixin'" :@declaration="declaration"></cem-mixin>
  <cem-function webc:if="declaration.kind === 'function'" :@declaration="declaration"></cem-function>
  <cem-variable webc:if="declaration.kind === 'variable'" :@declaration="declaration"></cem-variable>
</template>