\\n
\\n
\\n \\n
jimafisk
commented
4 years ago Closed jimafisk closed 4 years ago
jimafisk
commented
4 years ago jimafisk
commented
4 years ago jimafisk
commented
4 years ago jimafisk
commented
4 years ago jimafisk
commented
4 years ago Based on the reddit conversation, I'm thinking through how to run a V8 sandbox to compile svelte. The most current + supported project seems to be https://github.com/rogchap/v8go (also doesn't require us to build v8 ourselves).
Even though the actual script might not be faster than running node directly, I think it would speed up the build because we could cut out the exec.Command and the bundling (in Go) and unpacking (in JS) of templates we're currently doing. It would also make installation easier which would resolve issues like https://github.com/plentico/plenti/issues/32 and we would no longer need to execute the system nodejs which should fix the snapcraft issue https://github.com/plentico/plenti/issues/31.
V8go can't require/import other scripts (see https://github.com/rogchap/v8go/issues/22) so we'll have to rethink a couple of things:
jimafisk
commented
4 years ago Watching for a Go API on esbuild: https://github.com/evanw/esbuild/issues/152
jimafisk
commented
4 years ago Esbuild Go API docs: https://github.com/evanw/esbuild/blob/master/docs/go-api.md
jimafisk
commented
4 years ago Proposal:
plenti new site my-site) plenti will write these dependencies to /node_modules/ folder inside the project.plenti new site command. So plenti build and plenti serve must check for the existence of a /node_modules/ folder and if it's not there, write the defaults included with the binary to the filesystem. This check is necessary since the /node_modules/ folder is not typically tracked by git. Things to keep in mind:
/node_modules/, either to extend your site with additional functionality or to simply change the version of a core dependency, this should be reflected in your package-lock.json file. It is now your responsibility to communicate with your team that using NPM like normal is required (e.g. when pulling an existing project, if you simply plenti serve you are going to get the defaults, you should instead npm install first)./node_modules/ folder will not inspect versions or individual packages inside the /node_modules/ folder, it will simply check that the /node_modules/ folder exists - so if you've added this folder and removed a core package, it will break the build./node_modules/ in any way (extended or updated), you will need to account for this in the CI that builds and deploys your site. Basically you will need to add an npm install step. Currently the official plenti image (https://hub.docker.com/r/plentico/plenti) does not have nodejs or npm installed on it, so you will need to take care of setting up that. In the future we'd like to support a second container with NPM on it already to make this easier, but we'll keep it separate to avoid bloat on the official image for folks who want a faster experience when using plenti core without modification.jimafisk
commented
4 years ago We can remove require-relative by just pulling the part we need into build.js, however it still requires referencing the module nodejs package:
import Module from 'module';
let root = new Module();
let htmlWrapper = '/home/jimafisk/Desktop/my-site/layout/global/html.svelte';
let component = root.require(htmlWrapper).default;Where component is equal to { render: [Function: render], '$$render': [Function: $$render] } which allows us to create static HTML and CSS with
let { html, css } = component.render(props);The current implementation using v8go is working for the client build, but it might be worth looking into QuickJS for speed improvements and module support:
Another conversation recommending QuickJS: https://www.reddit.com/r/golang/comments/cd5gja/does_anyone_have_experience_with_parsing/
List of embeddable javascript interpreters: https://gist.github.com/maxogden/c61a58498c1933ece598
Examples:
Tried otto but it took almost 4 minutes to compile the client SPA, vs about 300ms with v8go:
jimafisk
commented
4 years ago Tried some rough benchmarking of QuickJS against v8go:
| Attempt # | QuickJS | v8go |
|---|---|---|
| 1 | 649.606009ms | 53.061618ms |
| 2 | 686.947327ms | 61.719715ms |
| 3 | 673.800305ms | 60.363899ms |
| 4 | 670.392486ms | 57.559065ms |
| 5 | 673.398664ms | 57.768148ms |
| 6 | 683.323302ms | 60.482449ms |
jimafisk
commented
4 years ago Wrapping my head around process for creating static html in Svelte.
What's actually happening behind the scenes is it looks like node_modules/svelte/register.js actually imports node_modules/svelte/compiler.js then runs a svelte.compile() passing the generate: 'ssr' option. It then wraps the js.code outputted in a CJS module using module._compile().
Manually running the svelte compiler with SSR as a test in plenti's cmd/build/client.go:
ctx.RunScript("var { js, css } = svelte.compile(`"+componentStr+"`, {generate: 'ssr'});", "compile_svelte")
jsCode, _ := ctx.RunScript("js.code;", "compile_svelte")
fmt.Println(jsCode)${paragraph}
`)}${intro.slogan}
${paragraph}
`)}\\n layout/content/{type}.svelte\\n \\n \\nlayout/content/${escape(type)}.svelte
Notably each starts with an import { create_ssr_component } from "svelte/internal"; ESM.
If you look at node_modules/svelte/internal/index.js it defines the create_ssr_component() function on line 1317, which is where you get the $$render functions. This ties back to the first snippet in this comment where:
let component = root.require(htmlWrapper).default;
console.log(component);Yields the following value:
{ render: [Function: render], '$$render': [Function: $$render] }Using v8go instead of Node is getting a lot closer, but there are persistent issues. One of those is importing components with custom names. Right now we add SSR compiled JS to the ctx for each component. We don't have a smart way of doing this so we're simply naming based on the filename (e.g. nav.svelte becomes Nav, pages.svelte becomes Pages, etc). This does not account for importing a component with a custom name, for instance the default starter does something like this: import Uses from "../components/template.svelte";.
One solution is when reading the layout file, check for imports and when found assign the aliased name to the base component in the ctx. So using the above example:
SSRctx.RunScript("var Uses = Template", "create_ssr")The downside is you would have a naming collision if different pages used the same alias for different components.
A more robust solution might be isolating the ctx to the current component and its specific imports using a custom gopack-like lookup. Would need to consider the performance implications of setting something like this up. Currently the v8go implementation is roughly as fast as the node implementation. I'll do some benchmarking once this is a little more stable.
jimafisk
commented
4 years ago New thought: give each component a signature based on its path and use that as the variable name where it's declared and everywhere it's referenced, e.g. layout/components/grid.svelte would become var layout_components_grid;. That will avoid any trouble with two components with the same name in different folders, which is actually a different naming collision issue than what was mentioned in the previous comment.
When reading the file we will have to analyze relative paths (e.g. ../components/grid.svelte) to get the full path from root. Then change whatever it is imported as to the signature. Taking a different example:
${paragraph}
`)}${paragraph}
`)}Note we comment out the import/export statements in the SSR'd JS because v8go can't handle it. We essentially bundle the dependencies together manually in the vm ctx.
For the same example above, here is the Template component that is being imported:
\\n layout/content/{type}.svelte\\n \\n \\nlayout/content/${escape(type)}.svelte
\\n layout/content/{type}.svelte\\n \\n \\nlayout/content/${escape(type)}.svelte
When client and static builds were separate (https://github.com/plentico/plenti/commit/c40c1d55d0f8e1751580fa41062387aa1b83a5eb):
The go build compiled components and static html individually:
jimafisk
commented
4 years ago Benchmarking:
| Attempt # | plenti build --nodejs=true (With Node) | plenti build (No Node) |
|---|---|---|
| 1 | 509.542643ms | 607.597406ms |
| 2 | 493.929817ms | 449.384268ms |
| 3 | 490.312936ms | 446.878569ms |
| 4 | 477.554289ms | 441.949252ms |
| 5 | 484.30959ms | 440.094771ms |
| 6 | 492.960634ms | 454.619161ms |
| 7 | 488.414887ms | 447.365403ms |
| 8 | 488.466514ms | 440.468207ms |
| 9 | 481.036876ms | 452.058979ms |
| 10 | 480.749496ms | 440.164956ms |
| average | 488.7277682ms | 462.0580972ms |
jimafisk
commented
3 years ago In this hacker news thread leaveyou wrote:
Please someone make a Svelte compiler in Go.. I can't stand Node.js and I can't install it on my machine without feeling dirty.
batisteo responded with:
There’s a decent JS to JS compiler (replacement of Babel) in Rust which understand TypeScript: https://swc-project.github.io
I remember looking into SWC early on, but doesn't seem like it made it onto this thread for some reason. I have some concerns about the amount of processing we're having to do to run everything in v8, but probably not going to make any switches in the short term. If there was a Svelte compiler written in Go with an API it would make things faster and more consistent for this project. It would also be nice not to require CGO so we could start cross compiling for Windows again.
zaydek
commented
3 years ago @jimafisk I just caught your talk on creating a Go SSG for Svelte: https://www.youtube.com/watch?v=4tD3Jz7JUfk, I’m about halfway through. I thought I’d comment here because you and I seem to be interested in the same problem space, I even tried making a SSG for React and just gave up and decided Svelte is the future. This is as far as I got: https://github.com/zaydek/retro.
I figured out how to solve for these problems:
All that being said, I realized React is actually very problematic as the frontend target because it’s so reliant on Babel / ESLint, etc. that even if you solve the backend problem, you still have the whole DX problem which is significant and I don’t care enough about to solve simultaneously.
So I thought I’d leave a comment here to get your attention and see if you want to connect. 🙂 Feel free to email me at zaydekdotcom [at] gmail or DM me on Twitter @username_ZAYDEK. I couldn’t easily find your email so I thought I’d just comment here. 😬
Solid work you got going on here.
jimafisk
commented
3 years ago Retro looks cool @zaydek! I'd love to connect, just followed you on twitter and sent you an email :)
jimafisk
commented
3 years ago How the svelte compiler works: https://dev.to/joshnuss/svelte-compiler-under-the-hood-4j20 + simplified example: https://github.com/joshnuss/micro-svelte-compiler
jimafisk
commented
3 years ago I took another look at a pure Golang javascript interpreter to see if it was feasible to drop the complexities that come with a cgo dependency. Currently it does seem like that would come at a significant performance cost.
Rough benchmarking of Goja against v8go:
| Attempt # | Goja | v8go |
|---|---|---|
| 1 | 3.222790268s | 56.900828ms |
| 2 | 3.16795169s | 56.840583ms |
| 3 | 3.228237879s | 57.287548ms |
| 4 | 3.262925706s | 57.178266ms |
| 5 | 3.188262189s | 56.869266ms |
From Goja's README:
Although it's faster than many scripting language implementations in Go I have seen (for example it's 6-7 times faster than otto on average) it is not a replacement for V8 or SpiderMonkey or any other general-purpose JavaScript engine.
Running a factorial like the example used for the benchmark is a heavy computation:
If most of the work is done in javascript (for example crypto or any other heavy calculations) you are definitely better off with V8.
The typical Plenti site might be lighter on computational requirements. If down the road we start moving Svelte compiling features into Go, there's a chance the performance between these projects would even out a bit:
If you need a scripting language that drives an engine written in Go so that you need to make frequent calls between Go and javascript passing complex data structures then the cgo overhead may outweigh the benefits of having a faster javascript engine.
Another consideration would be full ES6 compatibility: https://github.com/dop251/goja/milestone/1?closed=1
jimafisk
commented
3 years ago Also took a quick look at https://github.com/gojisvm/gojis, but the docs are still WIP (https://gojisvm.github.io/api.html) so it would need to mature before evaluating more seriously.
Tried running the Svelte compiler.js inside Goja with es6 support: go get github.com/dop251/goja@es6. Unfortunately it does not seem to have support for block scoped variables like const and let: https://github.com/dop251/goja/issues/167#issuecomment-660967321
In case it's useful, here's what you currently get out of the box with Plenti for build times:
jimafisk
commented
2 years ago It seems like Goja has come a long way since I last tested it. The ES6 milestone has been completed and I ran some performance tests against v8go again and the results were very different from last time:
| Attempt # | Script Size | Goja | V8go |
|---|---|---|---|
| 1 | Big | 163ns :pinching_hand: | 161ns :heavy_check_mark: |
| 1 | Small | 167ns :heavy_check_mark: | 180ns |
| 2 | Big | 174ns | 146ns :heavy_check_mark: |
| 2 | Small | 250ns | 175ns :heavy_check_mark: |
| 3 | Big | 245ns | 182ns :heavy_check_mark: |
| 3 | Small | 179ns :pinching_hand: | 175ns :heavy_check_mark: |
| 4 | Big | 179ns | 157ns :heavy_check_mark: |
| 4 | Small | 234ns | 175ns :heavy_check_mark: |
| 5 | Big | 172ns | 148ns :heavy_check_mark: |
| 5 | Small | 252ns | 170ns :heavy_check_mark: |
| 6 | Big | 235ns | 164ns :heavy_check_mark: |
| 6 | Small | 171ns :heavy_check_mark: | 182ns |
| 7 | Big | 168ns :heavy_check_mark: | 173ns :pinching_hand: |
| 7 | Small | 149ns | 131ns :heavy_check_mark: |
| 8 | Big | 152ns :heavy_check_mark: | 163ns |
| 8 | Small | 168ns | 138ns :heavy_check_mark: |
| 9 | Big | 141ns :pinching_hand: | 139ns :heavy_check_mark: |
| 9 | Small | 169ns :heavy_check_mark: | 239ns |
| 10 | Big | 158ns | 151ns :heavy_check_mark: |
| 10 | Small | 154ns :heavy_check_mark: | 159ns :pinching_hand: |
:pinching_hand: = within 5ns
So the numbers are much closer, in some cases goja even comes out ahead. My next step is to try using it to actually compile some svelte components and see how that goes.
https://github.com/zeit/pkg
Should:
npm install)web_modulesfor ESM