Closed Hental closed 1 year ago
can we ignore rollup config or only use part fields when we compute hash
That's interesting... Really we don't need rollup config in the hash in watch mode, since it can't change there, but such hash would be incompatible with normal builds where we do want rollup config in case it changes...
We can either use a separate hash strategy in watch mode (with a second set of cache artifacts), or a better way would be to pre-hash rollup config and only regenerate that in watch mode if config file changes (if we can/need to detect that).
The question is, what does rollup do to handle config changes in watch mode? Does it rebuild everything? Or ignores any changes?
Workaround, use clean: true
in watch mode
Thanks for providing some profiling! I knew the Rollup config hashing was slow and buggy (see also #228) but hadn't ever seen it be that slow 😬
Resolving the root cause here is a bit complicated (as #228 alludes to), as a change in any part of the config could result in a different output. Conservatively busting the cache on any change is safer. Limiting which parts of the config are cached could make the cache busting more inaccurate, resulting in more stale results
The cache is buggy and slow enough (especially as it rolls too often) though that I wonder if we should just make it an opt-in experimental feature instead. It has enough value for larger use-cases that I'm hesitant to remove it wholesale, but it seems to be more problematic than useful on average.
Yeah, I'm leaning towards removing or reworking cache too (maybe implementing caching DocumentRegistry). I've recently seen some errors being hidden until a clean build in my own projects.
(maybe implementing caching DocumentRegistry)
Curious what you mean by this? As in a custom implementation with filesystem caching? Something like that is mentioned in the TS Compiler docs, but never thought about implementing that 👀 I wonder how much that would increase performance 🤔
In #388 I did make the optimization for DocumentRegistry
for watch mode specifically; as far as I know by its usage and docs, the built-in one is only an in-memory cache.
The dependency graph rpt2 has sounds more advanced in theory (but in practice could be less efficient), and I think something like it will still be necessary to implement shouldTransformCachedModule
Thinking back, I did at one point consider splitting out the cache implementation into its own library. That way different implementations could be plugged in and could be shared with other TS libraries. At the time though, IIRC, parts of the code were a bit tied to the rpt2 implementation (the callbacks, for instance). That and I wasn't confident enough in its correctness to externalize it 😅 (could also more easily robustly test correctness if it were its own library, that being said)
Yeah, I expect typescript to have its own cache busting logic for document registry (based on compiler options and other parameters some DocumentRegistry calls take I assume), and that might handle some of the dependency tree logic we are trying to build in cache, but rollup potentially adds other reasons to retranspile by changing plugins that affect imports, so getting away from monitoring rollup config changes would be hard anyway...
Not sure if moving cache into its own library would give us much, since I expect most bugs to be related to failing to detect reasons to break cache and that requires full integration into pipeline. Unless we can make it a separate rollup plugin somehow?
Troubleshooting
Does
tsc
have the same output? If so, please explain why this is incorrect behaviorDoes your Rollup plugin order match this plugin's compatibility? If not, please elaborate
Can you create a minimal example that reproduces this behavior? Preferably, use this environment for your reproduction
What happens and why it is incorrect
Environment
Versions
```js import path from 'path'; import { defineConfig } from 'rollup'; import { genPlugin, genTypeDefinePlugin } from './plugins'; import { resolveFile, getPackageJson } from './utils'; const pkg = getPackageJson(); const libName = pkg.name; const devPlugins = genPlugin({ isDev: true, pluginTypescriptOptions: { sourceMap: true, }, }); /** @type {import('rollup').OutputOptions[]} */ const output = [ // { // file: resolveFile(`dist/${libName}.cjs.js`), // format: 'cjs', // }, { file: resolveFile(`dist/${libName}.js`), format: 'es', }, ]; // build config for development export default defineConfig([ { watch: { include: ['src/**'], }, input: resolveFile('src/index.ts'), output, plugins: devPlugins, external: [...Object.keys(pkg.dependencies), 'protobufjs/minimal'], }, ]); ```
:rollup.config.js
```json5 { "compilerOptions": { "moduleResolution": "node", "module": "ESNext", "target": "ES2020", "lib": ["dom", "ESNext"], "sourceMap": true, "declaration": true, "declarationMap": true, "declarationDir": "./type", "outDir": "./lib", "strict": true, "noImplicitAny": true, "allowSyntheticDefaultImports": true, "strictPropertyInitialization": false, "emitDecoratorMetadata": true, "experimentalDecorators": true, "baseUrl": ".", "paths": { "@/*": ["src/*"] }, "plugins": [ { "transform": "typescript-transform-paths", "useRootDirs": true, "afterDeclarations": true } ] }, "include": ["src/**/*.ts", "src/**/*.d.ts"] } ```
:tsconfig.json
```json "rollup-plugin-typescript2": "^0.34.1", ```
:package.json
```text ```plugin output with verbosity 3
:detail
When rebuilding, rollup-plugin-typescript2 computes a hash key for the cache. One portion of the hash is the Rollup config, which is actually so big of an object that it is a significant performance degradation to compute its hash.
Referencing this part of code: https://github.com/ezolenko/rollup-plugin-typescript2/blob/f6db59613a66f58c48310aa8fa785951970b5d6d/src/tscache.ts#L93-L102
In the profiling below, we can see that computing the hash compute takes up almost all of the time, about 20s: