inline rollup plugin 'shaderChunks' not contributing as much as it could

[epreston]

The rollup plugin, shaderChunks, is defined inline within rollup.config.js.

It is not contributing as much as it could to the build process. This could result in a small amount of bloat in build products.

Current limitations.

It filters files to '**/*.vert.js', and '**/*.frag.js'. The repository does not have any files that currently match this and the 'glsl' files it could match are defined as '.js'. Effectively this disables the plugin.

This can be resolved by changing the filter.

It will only process one /* glsl */ template literal per file if it was used. The matching regular expression is looking for the longest match. This creates a unique errors where javascript between the first and second /* glsl */ template literal in a single file would get added to the shader code of the first, and the second template literal would come out as "undefined" in the build tools.

This can be resolved by using the shortest match that is a complete template literal.

If developers add an extra space or line between the inline language keyword comment 'glsl' things stop working. If there is an extra space or line separator between the inline language keyword comment and the template literal, things stop working. Most tools allow a degree of flexibility here.

This can be resolved by allowing optional spaces in the 'glsl' language comment, and also allowing spaces and line returns between the comment and template literal.

Resolution

I propose changing the regular expression to the following and adjusting the filter to allow this plugin to contribute to the build process.

/\/\* *glsl *\*\/\s*\`(.*?)\`/gs

The filter would be simplified to the following so it "just works":

`'**/*.js'`

This will provide slightly more forgiving syntax in source code, which matches the tools used for syntax highlighting and editing.

// the original
export default /* glsl */` void main() {}`;

export default /* glsl */ ` void main() {}`;

export default /*glsl*/ ` void main() {}`;

export default /*glsl*/ 
`
void main() {}
`;

export default /* glsl */ 
` void main() {} `;

It will also allow developers to define (and work with) tightly bound fragment and vertex shaders in the same file. Each one will be optimised and treeshakeable. This might be a graceful interaction model in tools like editors.

// shader.js

export const vertex = /* glsl */ `
void main() {
    gl_Position = mvp * vec4( position, 1.0 );
}
`;

export const fragment = /* glsl */ `
void main() {
    gl_FragColor = vec4( 1.0, 0.0, 0.0, 1.0 );
}
`;

The pair would be imported under a common namespace and used as follows.

// render.js

// import the pair of shader exports
import * as myShader from './shader.js';

// we can now use myShader.vertex and myShader.fragment
forwardRender(geometry, myShader.vertex, myShader.fragment);

[willeastcott]

Wow, yeah, we should address this. Hopefully this will shave a few KB off the build size.

[epreston]

This is a robust implementation that may suite. This adds:

• the recommended include and exclude filter support
• a default set of files developers use to hide shaders

import { createFilter } from '@rollup/pluginutils';

/**
 * @type {readonly RegExp[]}
 */
const DEFAULT_SHADERS = Object.freeze(['**/*.js']);

/**
 * @param {PluginOptions} options Plugin config object
 * @returns {Plugin} The plugin that converts shader code.
 */
export function shaderChunks({
  include = DEFAULT_SHADERS,
  exclude = undefined,
  enabled = true
} = {}) {
  const filter = createFilter(include, exclude);

  return {
    transform(source, shader) {
      if (!enabled || !filter(shader)) return;

      source = source.replace(/\/\* *glsl *\*\/\s*`(.*?)`/gs, function (match, glsl) {
        return JSON.stringify(
          glsl
            .trim() // trim whitespace
            .replace(/\r/g, '') // Remove carriage returns
            .replace(/ {4}/g, '\t') // 4 spaces to tabs
            .replace(/[ \t]*\/\/.*\n/g, '') // remove single line comments
            .replace(/[ \t]*\/\*[\s\S]*?\*\//g, '') // remove multi line comments
            .concat('\n') // ensure final new line
            .replace(/\n{2,}/g, '\n') // condense 2 or more empty lines to 1
        );
      });

      return {
        code: source,
        map: null
      };
    }
  };
}

I'll finish testings and post results on build size. I'm hoping for a kb or two.

[epreston]

Here are the results. It's a modest 10 to 30 KB.

Build	Previous	Current	Savings
playcanvas-extras.mjs	59.9 KB	59.9 KB	-
playcanvas.dbg.mjs	15.5 MB	15.5 MB	-
playcanvas.min.mjs	1.61 MB	1.58 MB	30 KB
playcanvas.mjs	2.47 MB	2.46 MB	10 KB
playcanvas.prf.mjs	2.48 MB	2.46 MB	20 KB
playcanvas-extras.js	73.4 KB	73.4 KB	-
playcanvas.d.ts	1.34 MB	1.34 MB	-
playcanvas.dbg.js	15.4 MB	15.4 MB	-
playcanvas.js	2.56 MB	2.53 MB	30 KB
playcanvas.min.js	1.45 MB	1.42 MB	30 KB
playcanvas.prf.js	2.56 MB	2.53 MB	30 KB

[willeastcott]

Nice - modest, but still well worth having. 👏

[willeastcott]

The only think I'm wondering is whether that filter will slow down the build process since it now has to run that regex in the entire codebase.

[epreston]

I'll build a chart. It's milliseconds.

[epreston]

Important: shaders processing code requires a final new line for each chunk.

[epreston]

As far as impacting build times, its small. There are far bigger culprits. Other tools build this project in 1/10th the time.

This is the average of 5 runs over two different types of machines. Just focusing on build/playcanvas.min.js which is the most processed build product created. The additional build time is estimated at 200ms but it's also within the run to run variance of the original...

Build Machine	Time / Variance	Current
MacBook - M1 Pro	10s (+- 300ms)	within run variance
Windows - i9-12900K	11s (+- 200ms)	within run variance

[epreston]

For future reference, plugin performance and other build timings can be displayed by adding --perf to the rollup build command. Example: rollup --perf -c

For the most processed file, build/playcanvas.min.js this will display:

src/index.js → build/playcanvas.min.js...
created build/playcanvas.min.js in 11.2s
# BUILD: 7977ms, 272 MB / 689 MB
## initialize: 0ms, 5.25 kB / 417 MB
- plugin 2 (engineLayerImportValidation) - buildStart: 0ms, 1.39 kB / 417 MB
## generate module graph: 7117ms, 214 MB / 632 MB
- plugin 2 (engineLayerImportValidation) - resolveId: 3ms, 532 kB / 1.22 GB
- plugin 4 (babel) - resolveId: 2ms, 409 kB / 1.22 GB
- plugin 6 (stdin) - resolveId: 2ms, 375 kB / 1.22 GB
- plugin 0 (jscc) - load: 17ms, 2.49 MB / 1.22 GB
- plugin 1 (shaderChunks) - transform: 15ms, 5.52 MB / 1.22 GB
- plugin 3 (strip) - transform: 335ms, 57.6 MB / 1.22 GB
- plugin 4 (babel) - transform: 110ms, 16.5 MB / 1.22 GB
- plugin 5 (spacesToTabs) - transform: 28ms, 17.2 MB / 1.22 GB
generate ast: 574ms, 227 MB / 1.22 GB
analyze ast: 374ms, 120 MB / 1.22 GB
- plugin 4 (babel) - load: 61ms, 13.6 MB / 1.18 GB
## sort and bind modules: 57ms, 9.15 MB / 641 MB
## mark included statements: 802ms, 48.7 MB / 689 MB
treeshaking pass 1: 202ms, 25.4 MB / 666 MB
treeshaking pass 2: 136ms, 21 MB / 687 MB
treeshaking pass 3: 77ms, -2.37 MB / 685 MB
treeshaking pass 4: 53ms, 2.56 MB / 688 MB
treeshaking pass 5: 43ms, -28.1 kB / 687 MB
treeshaking pass 6: 34ms, 5.54 MB / 693 MB
treeshaking pass 7: 32ms, 969 kB / 694 MB
treeshaking pass 8: 30ms, -865 kB / 693 MB
treeshaking pass 9: 29ms, -2.61 MB / 691 MB
treeshaking pass 10: 29ms, -2 MB / 689 MB
treeshaking pass 11: 27ms, 13.7 MB / 702 MB
treeshaking pass 12: 28ms, -3.05 MB / 699 MB
treeshaking pass 13: 28ms, -3.2 MB / 696 MB
treeshaking pass 14: 28ms, -3.27 MB / 693 MB
treeshaking pass 15: 28ms, -3.27 MB / 689 MB
# GENERATE: 3279ms, 41 MB / 731 MB
## initialize render: 0ms, 2.82 kB / 690 MB
## generate chunks: 5ms, 4.17 MB / 694 MB
optimize chunks: 0ms, 85.7 kB / 693 MB
## render chunks: 164ms, 25 MB / 719 MB
## transform chunks: 3110ms, 11.7 MB / 731 MB
## generate bundle: 0ms, 1.02 kB / 731 MB
# WRITE: 7ms, -12.9 MB / 718 MB

Interesting that this plugin consumes 16ms of build time.