Editor support

[hpx7]

Hey, cool project! I was looking for typesafe templating solutions and came across your project via google. I find it crazy that there isn't more in this space...

I'm working on a project where I'm using templating as a form of code generation. As an example, see this file which generates an auth module based on some user configuration: https://github.com/hpx7/rtag/blob/be5b3c365b415aa7888152f4fc4f4472959ef022/templates/base/auth.ts.hbs

I've been thinking about how it would be nice to have type-safe templating so I could do things like validate exhaustive checking of enums/unions etc. I first thought of switching to using template literals but quickly realized they don't work great for this use case (precise text formatting, lots of control flow).

ETS seems like a great option, but obviously the huge advantage of template literals is the editor support -- you get in your IDE (1) compile errors and (2) autocomplete. I've been brainstorming ideas for how to achieve the best of both worlds, just dropping this issue in case you had any thoughts on this topic.

[tatethurston]

Thanks @hpx7! I was pretty surprised too that there aren't more options in the space.

That file looks exactly like the problem I was working on that prompted me to write this: templating for code generation. I also tried template literals and came to the same conclusion as you.

Completely agree on the IDE integration. I almost tried tackling that in December, but ultimately held off to see if anyone else would be interested in this project. I'd love to hear your thoughts.

A TypeScript language server plugin might work? If not, writing a language server is another option. It looks like https://github.com/lifeart/els-addon-typed-templates achieves this in the type system: https://github.com/lifeart/els-addon-typed-templates/pull/11#issue-351282903.

[hpx7]

Completely agree on the IDE integration. I almost tried tackling that in December, but ultimately held off to see if anyone else would be interested in this project. I'd love to hear your thoughts.

If this project had good VSCode integration and we posted it around a bit, I think it would immediately get popular.

I don't have any experience with language servers or ts language server plugins, but my first thought is that this may be difficult to implement as a plugin because we're fundamentally operating on text files, not typescript files. That els-addon-typed-templates looks like exactly what we want to achieve -- but looks like it's building on top of https://github.com/ember-tooling/ember-language-server.

Have you seen https://code.visualstudio.com/api/language-extensions/embedded-languages?

[tatethurston]

my first thought is that this may be difficult to implement as a plugin because we're fundamentally operating on text files, not typescript files.

Yeah I think so too.

Have you seen https://code.visualstudio.com/api/language-extensions/embedded-languages?

Awesome, thanks for sharing this. I like that the language service implementation would enable integrations with other editors as well.

[hpx7]

Hi @tatethurston just curious if you had any more recent thoughts on this?

[tatethurston]

Hey @hpx7 I've been heads down at work and haven't had a chance to build this yet. Have you seen any open source implementations of a visual studio language extension you like? I poked around a bit but I couldn't find anything concrete -- lots of packages wrapping packages.

[hpx7]

The Vue one is quite good: https://github.com/vuejs/vetur

[tatethurston]

Another good reference: https://github.com/bash-lsp/bash-language-server

[tatethurston]

I released a very MVP of visual studio code editor support: https://marketplace.visualstudio.com/items?itemName=embedded-typescript.embedded-typescript-vsc

Right now, only errors from the ets compiler appear in editor, and there isn't any completion. Read: none of DX from TypeScript yet, but the scaffolding is at least in place.

Next steps will be:

• Integrate with TS. Completions and errors.
• Think about fault tolerance. Right now the compiler halts on the first error encountered.

Related to the first point above, the compiler should probably use the TS compiler as a peer dependency and type check templates so that users do not encounter TS errors in the generated files.

[hpx7]

This is awesome! As soon as there is typescript integration I'm going to replace handlebars with this in my project.

How hard would it be to skip the yarn ets step? Handlebars exposes a compile method which lets me compile my .hbs templates programmatically without a separate cli step.

[tatethurston]

How hard would it be to skip the yarn ets step? Handlebars exposes a compile method which lets me compile my .hbs templates programmatically without a separate cli step.

My understanding is that this is not possible with the current TypeScript compiler API. The compiler would need to expose a plugin API that enabled provided handlers to run and return a TS AST, similar to webpack loaders https://webpack.js.org/contribute/writing-a-loader. A Webpack loader would get close, but AFAICT we wouldn't have the necessary type information available in the importing file (because TS wouldn't know the contents).

Would a watch mode achieve sufficient parity with the DX you are envisioning?

[hpx7]

Oh I didn't realize you needed to act on the AST, so then I suppose tools like esbuild wouldn't help either?

A watch mode sounds pretty good, but the way I use handlebars right now is to compile files like foo.ts.hbs to foo.ts. So ideally it would be possible to go from foo.extension.ets directly to foo.extension without the .ets.ts intermediary.

[tatethurston]

I may be misunderstanding, but right now foo.ets compiles to foo.ets.ts. The later is what is imported by application code (TS implicitly adds the ts extension when you import from 'foo.ets'.

foo.ets => foo.ets.ts.

I opted for appending an extension (.ts) similar to how many graphql compilers will generate a .graphql.ts from a .graphql file. That said, I don't feel particularly strongly about this approach. I'm open to the handlebars output file naming pattern where instead of appending an extension, an extension is dropped:

foo.extension.ets => foo.extension.

The later may be clearer, and would clean up some funniness with jest that I don't love right now -- jest has it's own module resolution system that doesn't follow TS semantics above where '.ts' is appended to all import paths.

[tatethurston]

It would be pretty straightforward to export compile from embedded-typescript so that the source file compilation can be invoked programmatically instead of through the cli invocation -- is that desirable?

For clarity, this wouldn't be the same as var template = Handlebars.compile("Handlebars <b>{{doesWhat}}</b>"); -- we can't do runtime evaluation of the template file because TypeScript won't have any of the type information from the source file. In the Handlebars case the return value of Handlebars.compile is effectively a static (...args: any[]): string;.

Before your application ran you would need compile to have been invoked so the compiled files are generated.

Could you point me to some code where you're programmatically invoking handlebars compiler to generate foo.ts from foo.ts.hbs? I want to make sure I'm understanding correctly.

[hpx7]

This is the code where I do codegen using handlebars templates: https://github.com/hpx7/rtag/blob/67d907187e2adaac2134b85a0839bc781be03fc2/generate.ts#L199-L200

I guess if you expose a compile function, I could generate the .ets.ts file, import this generated file dynamically, call render with my data context and save the output to a file, and then delete the intermediary .ets.ts file afterwards?

[tatethurston]

This is the code where I do codegen using handlebars templates: https://github.com/hpx7/rtag/blob/67d907187e2adaac2134b85a0839bc781be03fc2/generate.ts#L199-L200

I guess if you expose a compile function, I could generate the .ets.ts file, import this generated file dynamically, call render with my data context and save the output to a file, and then delete the intermediary .ets.ts file afterwards?

Are clients supplying templates to your codegen tool? Or does your tool define the templates and clients supply the values?

[hpx7]

The templates are bundled with my tool (see https://github.com/hpx7/rtag/tree/develop/templates/base/server/.rtag for example). Users have to write a yml file which ultimately ends up supplying the values.

[tatethurston]

The general workflow I have in mind is:

1. Write your .ets template file. Check this into version control, because it's your source code.
2. Generate the .ets.ts files from your templates via yarn ets. Check this into version control as well, so that collaborators only need to compile when they change a template file. Generally I don't advocate for version controlling generated files, but in this case these generated files are imported by your source code, so I would for the DX improvement.
3. Build your TS project for distribution: either via tsc, esbuild, or another tool. Your distribution can either be a single js file or multiple files. Either way, the .ets template files won't be present in the distribution, only the compiled .ets.ts contents will be present.

If I'm understanding correctly, in your case it looks like compile is ran on the consuming client as part of invoking generate? If so, I would move compilation into your pre-distribution step, and only execute the generated template contents on the consuming client.

Instead of bundling the uncompiled templates with your tool, you would bundle the compiled templates with your tool. Then you can drop handlebars / ets as a package dependency (they would become a devDependency).

Does that make sense? Let me know if you have a constraint that I'm missing.

[hpx7]

Thanks for walking me through that, yes that would work!

The only real downside I see is for folks who are using a locally checked out version of my tool rather than the packaged one in npm. I would have to publish additional instructions for them (as well as for potential contributors) on the workflow, but not the end of the world.

[tatethurston]

Awesome!

The only real downside I see is for folks who are using a locally checked out version of my tool rather than the packaged one in npm.

If you commit the generated ets.ts files from your templates into git, then only users who change the templates locally would need to know about the ets compiler. Same for contributors -- only those who change the template files need to know about the compiler. For the later group, having a build:watch script or similar could help mitigate any onboarding issues:

// package.json
scripts: {
   build:watch: "yarn ets --watch & tsc --watch" 
}

This would require the addition of a watch mode to the ets compiler