webpack5: getPathKey returns key not in bundlesContent; leads to sha1(undefined) in karma

[himdel]

• Operating System: linux
• Node Version: 14
• NPM Version: yarn 1.22
• webpack Version: 5.3.0 (webpack-cli 4.1.0)
• karma-webpack Version: #next, 5.0.0-alpha.3.0

Expected Behavior

using './client/app.js': ['webpack'] in karma.conf preprocessors, processing that file should work. (With an otherwise working webpack config.)

Actual Behavior

27 10 2020 21:49:28.041:ERROR [karma-server]: UnhandledRejection: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined
27 10 2020 21:49:28.042:ERROR [karma-server]: TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined
    at Hash.update (internal/crypto/hash.js:84:11)
    at Object.sha1 (/home/himdel/manageiq-ui-service/node_modules/karma/lib/utils/crypto-utils.js:9:8)
    at runProcessors (/home/himdel/manageiq-ui-service/node_modules/karma/lib/preprocessor.js:82:26)
    at async FileList.preprocess [as _preprocess] (/home/himdel/manageiq-ui-service/node_modules/karma/lib/preprocessor.js:146:5)
    at async Promise.all (index 0)
    at async /home/himdel/manageiq-ui-service/node_modules/karma/lib/file-list.js:90:11
    at async Promise.all (index 9) {
  code: 'ERR_INVALID_ARG_TYPE'

in karma/lib/preprocessor.js, the failing file is '/home/himdel/manageiq-ui-service/client/app.js' (the only file using the webpack preprocessor), initial content before the webpack preprocessor is the file content, after the webpack preprocessor, it's undefined.

In karma-webpack/lib/karma-webpack.js, after await controller.bundle() in processFile: bundleContent is undefined, getPathKey(file.path, true) returns app.1564016497.js, while the only keys matching /app/ in controller.bundlesContent are:

    'sass/_app_urls.sass',
    'sass/_app_colors.sass',
    'js/app.1564016497-3d658aca95fc6995589c.js',
    'js/app-208bf5b45fa818efa9cf.js'

So looks like webpack5 is adding a prefix and a hash suffix, which getPathKey doesn't handle.

Do I need to override some part of webpack config so that it doesn't do that for tests, or is this an actual bug?

[himdel]

(The fix in https://github.com/ryanclark/karma-webpack/pull/451 doesn't help, but that's not surprising as this is pure js, not typescript.)

[himdel]

What did help in my case was:

-    const bundleContent =
-      controller.bundlesContent[getPathKey(file.path, true)];
+    let key = `js/${getPathKey(file.path)}-`;
+    key = Object.keys(controller.bundlesContent).find((name) => name.startsWith(key));
+    const bundleContent = controller.bundlesContent[key];

[himdel]

So looks like webpack5 is adding a prefix and a hash suffix, which getPathKey doesn't handle.

Ok, it is coming from the webpack config, via output.filename.

So... I can fix this on my side, by explicitly forcing config.output.filename = '[name].js' in webpack testing config.

Feel free to close this .. but maybe you want to override that / warn about it / handle that case?

[codymikol]

It seems that this error happens in a few cases when the webpack configuration has an invalid field, I have a similar issue over here. https://github.com/ryanclark/karma-webpack/issues/458. I think we can treat this in the same way, I just want to verify that these are required settings and that we won't break anyones environment by setting these automatically and warning about invalid configuration.

[codymikol]

To limit confusion, I am combining all threads for webpack 5 / karma6 support over here ==> https://github.com/ryanclark/karma-webpack/issues/475

[codymikol]

v5 is now available

[RaphaelRoyerRivard]

I'm using webpack 5.28.0 and karma-webpack 5.0.0 and I'm still getting the TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined error when using "declaration": true in the compiler options. Is there some specific karma config I need to have to prevent that error?

[everlose]

I'm using webpack 5.38.1 and karma-webpack 5.0.0-alpha.6 and still getting

09 06 2021 14:11:35.686:ERROR [karma-server]: Error during file loading or preprocessing
TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined
    at Hash.update (internal/crypto/hash.js:84:11)
    at Object.sha1 (/Users/chenziyun/workspace/netease/im/sdk/node_modules/karma/lib/utils/crypto-utils.js:9:8)

My karma conf

const path = require('path')

module.exports = function(config) {
  config.set({

    // base path that will be used to resolve all patterns (eg. files, exclude)
    basePath: '',

    // mime: {
    //   'text/x-typescript': ['ts', 'tsx']
    // },

    frameworks: ['mocha', 'chai', 'webpack'],

    // list of files / patterns to load in the browser
    files: [
      // { pattern: '', included: false },
      // 'dist/NIM_BROWSER_SDK.js',
      // 'test/**/*.spec.js'
      'test/index.ts'
    ],

    // list of files / patterns to exclude
    exclude: [
      // 'dist/NIM_BROWSER_SDK.js'
    ],

    // preprocess matching files before serving them to the browser
    // available preprocessors: https://www.npmjs.com/search?q=keywords:karma-preprocessor
    // preprocessors: {
    //   'test/**/*.spec.js': ['webpack']
    // },
    preprocessors: {
      'test/index.ts': ['webpack']
    },

    // test results reporter to use
    // possible values: 'dots', 'progress'
    // available reporters: https://www.npmjs.com/search?q=keywords:karma-reporter
    // reporters: ['nyan', 'coverage-istanbul'],
    reporters: ['progress'],

    // web server port
    port: 9876,

    // enable / disable colors in the output (reporters and logs)
    colors: true,

    // level of logging
    // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
    logLevel: config.LOG_INFO,

    autoWatch: true,

    browsers: ['Chrome'],

    singleRun: !!process.env.CI,

    concurrency: Infinity,

    webpackMiddleware: {
      noInfo: true,
      stats: 'errors-only'
    },
    webpack: {
      mode: 'development',
      entry: './src/entry/browser.ts',
      output: {
        filename: '[name].js'
      },
      devtool: 'inline-source-map',
      module: {
        rules: [
          {
            test: /\.tsx?$/,
            use: {
              loader: 'ts-loader',
              options: {
                configFile: 'test/tsconfig.json'
              }
            },
            exclude: [path.join(__dirname, 'node_modules')]
          },
        ]
      },
      resolve: {
        extensions: ['.tsx', '.ts', '.js', '.json']
      }
    },
  })
}

My test file in test/index.ts

const tests = require.context('./', true, /\.spec\.tsx?$/)

tests.keys().forEach(tests)

const sources = require.context('../src/', true, /\.tsx?$/)

sources.keys().forEach(sources)

[everlose]

So looks like webpack5 is adding a prefix and a hash suffix, which getPathKey doesn't handle.

Ok, it is coming from the webpack config, via output.filename.

So... I can fix this on my side, by explicitly forcing config.output.filename = '[name].js' in webpack testing config.

Feel free to close this .. but maybe you want to override that / warn about it / handle that case?

I set config.output.filename = '[name].js' in webpack testing config，no use。 I print getPathKey and controller.bundlesContent in karma-webpack/lib/karma-webpack.js

getPathKey~~~~~~~~~~  index.3934682155.ts
processFile～～～～～～～～ [ 'commons.js', 'runtime.js' ]

It's weird. My karma config is in the comments above

[codymikol]

So looks like webpack5 is adding a prefix and a hash suffix, which getPathKey doesn't handle.

Ok, it is coming from the webpack config, via output.filename. So... I can fix this on my side, by explicitly forcing config.output.filename = '[name].js' in webpack testing config. Feel free to close this .. but maybe you want to override that / warn about it / handle that case?

I set config.output.filename = '[name].js' in webpack testing config，no use。 I print getPathKey and controller.bundlesContent in karma-webpack/lib/karma-webpack.js

getPathKey~~~~~~~~~~  index.3934682155.ts
processFile～～～～～～～～ [ 'commons.js', 'runtime.js' ]

It's weird. My karma config is in the comments above

Did you update to karma-webpack v5.0.0 ?

[Luthaf]

I had the same issue with karma-webpack v5.0.0, but only on macOS. No issue whatsoever on Linux & Windows machines.

Changing this line: https://github.com/ryanclark/karma-webpack/blob/ef7edb6b6756fb563871eaff88ca876892694896/lib/karma-webpack/preprocessor.js#L95

to return bundleContent; made this error message go away (I still have another issue to fix). I can do a PR with this change if you want, but I'm not sure I understand fully what is going on here. Seems related to https://github.com/karma-runner/karma/blob/2b71a3c6a01757e0b5c1bf8d241b588656b62127/lib/preprocessor.js#L47-L55

[Luthaf]

The changes bewteen master and v5.0.0 also fix this issue, could you do a v5.0.1 release with them?

[codymikol]

Which change specifically? Something in https://github.com/ryanclark/karma-webpack/commit/ef7edb6b6756fb563871eaff88ca876892694896 ?

[Luthaf]

Yes, I think these changes are the ones: https://github.com/ryanclark/karma-webpack/commit/ef7edb6b6756fb563871eaff88ca876892694896#diff-282d7a470af0c4d49f499674bcd6d416cdc3ad49918bb564a2ae5e3e8dad81ccR86-R96

[codymikol]

Yes, I think these changes are the ones: ef7edb6#diff-282d7a470af0c4d49f499674bcd6d416cdc3ad49918bb564a2ae5e3e8dad81ccR86-R96

Strange, that doesn't actually change anything. It just replaces async with promises.

[Luthaf]

I think the error is related to async/promise and the code in https://github.com/karma-runner/karma/blob/2b71a3c6a01757e0b5c1bf8d241b588656b62127/lib/preprocessor.js#L47-L55, especially the branch doing return donePromise, which ends up returning undefined, and triggering the error The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined.

From v5.0.0/e2423b47ec59059bc1808c670affa64d0f5aa8b7, changing https://github.com/ryanclark/karma-webpack/blob/e2423b47ec59059bc1808c670affa64d0f5aa8b7/lib/karma-webpack/preprocessor.js#L96 to return bundleContent; also fixes the error.

My understanding is that async function are implicitly wrapped in promise, and since the function on v5.0.0 did not return (i.e. it returned undefined), that's what karma was seeing. On master, the code is explicitly using a promise (so no implicit wrapping of the function inside another promise occurs), and so calling done actually transfer the control flow back to the outer promise.

[SevenOutman]

Update

Solved with removing some entries from preprocessors from karma config.

  preprocessors: {
    'src/**/*Spec.js': ['webpack'],
-   'test/**/*.js': ['webpack']
  },

Original comment

Similar issue here. Using:

npm ls webpack karma karma-webpack
├── karma@6.3.4 
├── karma-webpack@5.0.0 
└── webpack@5.34.0

getting:

START:
Webpack bundling...
238 assets
webpack 5.34.0 compiled successfully in 27074 ms
18 09 2021 18:13:10.990:ERROR [karma-server]: Error during file loading or preprocessing
TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or 
DataView. Received undefined