Tests Cached giving wrong results

[matthiasg]

When using karma with angular and component.io I get some very weird behaviour and would like to know whether this is actually an issue or something i can change.

The first test runs fine and shows e.g. tests running successfully. I then change code (e.g. comment out a line and save) which causes my build.js to be rebuilt which causes karma test runner to pick up this change and re-run the tests which also shows the defect in the appropriate tests. So far so good.

The issue comes when then doing another change (e.g. commenting the line back in). It still fails and keeps failing until I either restart karma (was running in ci mode) or open a new tab with debug mode. When opening a new debug tab it shows the correct console log with all successful tests.

When opening up the scripts of the first tabs in the developer section in chrome it shows that the code (e.g. build.js) is still the old code and not correctly updated. Opening a new tab as described above actually triggers a reload.

I tried to delay the batch start, I tried to comment out the caching behaviour in karma, I tried to ensure that chromes caches are disabled) all to no avail.

Anybody any hints ?

[maksimr]

@matthiasg can you show example of karma.config.js or example project where reproduce this behavior?

Thanks

[matthiasg]

the config is here .. https://gist.github.com/matthiasg/5f8c5081c71872e4b3a3

the project itself is closed source still sadly. I switched to testem this morning which works fine out of the box.

we are already rearchitecting to move almost all embedded directives out of the closed source domain into seperate component.io modules with their own tests, to make it easier to debug issues in the open like this, but we are not there yet.

I had hoped there was something we overlooked with the case of first generating a build.js (using component.io in our case) and having that as tests.

[maksimr]

@matthiasg thanks.

And as far as I understand, karma loads only third-party libraries, test files and build.js(your component). build.js was builded using component.

You have something like this:

file structure:

<project>
  - js(third-party libraries)
  - bower_components(third-party libraries)
  - src(you source files)
  - build/buil.js(builded source files)
  - main.js
  - test/client/*(tests files)

runing component builder in console. something like this:

component --watch --build src/

and runing karma server:

karma start --single-run=false

And when you change source files component build build.js, and karma run tests. In this moment you get old build.js content.

Right?

It will be cool if you create stub project where reproduce bug)

[matthiasg]

i am using nodemon to trigger a component --dev build (a dev build retains the individual file structure using string instead of symbols which is better for debugging while not being able to minimize well) but yes thats exactly right ..

i already invested a couple of hours yesterday trying to find an error on my end (code/component.io etc) so i wont have time to extract a test scenario right now .. but i will do it within the 7-10 days since this is indeed worthy of pursuit. i would assume it is an issue with how the initial website (the one with the debug button) refreshes itself. i also did not notice any changes in the parameters karma appends to the source file urls between runs which (i assume) are added to mitigate caching issues.

Once it is extracted i would assume, based on my current findings so far, that the karma webserver has some sort of caching mechanism which doesn't re-read the changed files from disk but instead serves from ram. since it sometimes works and it is only an issue in a CI scenario it might not be readily apparent to users. Of course i might be wrong but the exact same codebase works as expected with a similar testem.yml file so the code generation and tests themselves seem in order.

[matthiasg]

btw .. the FIRST change works .. the second change and more does not work ..

On Tue, Jun 3, 2014 at 6:24 PM, Maksim Ryzhikov notifications@github.com wrote:

@matthiasg https://github.com/matthiasg thanks.

And as far as I understand, karma loads only third-party libraries, test files and build.js(your component). build.js was builded using component https://github.com/component/component.

You have something like this:

file structure:

- js(third-party libraries) - bower_components(third-party libraries) - src(you source files) - build/buil.js(builded source files) - main.js - test/client/*(tests files) runing component builder in console. something like this: component --watch --build src/ and runing karma server: karma start --single-run=false And when you change source files _component_ build _build.js_, and _karma_ run tests. In this moment you get old build.js content. Right? It will be cool if you create _stub_ project where reproduce bug) — Reply to this email directly or view it on GitHub https://github.com/karma-runner/karma/issues/1087#issuecomment-44987197.

[colinhowe]

We're seeing something incredibly similar to this..

[osi-oswald]

We are observing the exact same issues in our project.

Karma is watching a test.js file which is created by the typescript compiler (tsc) from a big test.ts file. For some unknown reasons, changes from tsc to test.js are ignored by karma. It only does this after the first change to test.ts like @matthiasg described. By ignoring I mean, karmas watcher reacts to the file change in test.js, but it feeds a (cached?) old version into the test runner.

Some facts regarding this issue:

• tsc does indeed change test.js, this is not the problem (I manually checked with a text editor)
• If I do the change manually (in the text editor) that tsc would do to test.js, karma works just fine
• If I feed a copy from test.js to karma after it has been updated by tsc, karma works just fine

I tried to extract the bits and pieces to reproduce this behavior in its own project. But it looked like the problem went more and more extinct the smaller the project got. By getting "more and more extinct" I mean, the problem did not occur after the first change, but sometime after the fifth, tenth or even more (quite unsteady).

As a workaround, we are currently feeding a copy of test.js to karma as soon as it got updated by tsc. This is a strange fix for now, but at least karma behaves the way it is supposed to.

[dignifiedquire]

@osi-oswald is this still an issue with latest version of karma?

[wilsonpage]

I switched from Nodemon to Watchify and bumped the Karma autoWatchBatchDelay option to 500ms. So far so good.

[sjurba]

I had similar problems that I narrowed down to the atom-beautify pluing for atom.. https://github.com/Glavin001/atom-beautify/issues/440

I'll checkout if the autoWatchBatchDelay helps for me to.

[sjurba]

The autoWatchBatchDelay did not seem to help for me...

[sjurba]

@Dignifiedquire I'm still having this issue, described more in detail in the atom-beutify issue linked in my comment above. I have started digging into it, and it seems the issue is with karma not with the atom package. All the atom package does (to trigger this) is to re-save the file rapidly. So it is saved 2 times very rapidly.

What I see is that when things goes wrong this function is called with no content: https://github.com/karma-runner/karma/blob/master/lib/preprocessor.js#L28

I think that what happens is that there is a race condition, so karma is reading the file while the atom plugin is writing to it, effectively making karma read an empty file. When the nextPreprocessor function is called with no content it produces an actual sha, but it is the same one every time, the sha of ''. When the browser later requests the file, it has been completely written to disk. So this means that every time this race condition happens and karma reads an empty file, the browser will use the cashed version of the first time this race condition was happened. Which is exactly what I am seeing. Also I am unable to trigger this problem with small files, only files files >2-300 lines.

I'm not sure this problem is the same problem as the author of this issue is having though, so do you want me to create a new issue for it or just keep it here?

[sjurba]

@Dignifiedquire right it seems I just bumped into the issue vojta warns us about here: https://github.com/karma-runner/karma/blob/master/lib/file_list.js#L368

I think suggestion 2: "delay the promise resolution - wait for any changeFile operations to finish" would be the way to solve this, by waiting for the autoWatchBatchDelay before triggering the preprocessor..

[dignifiedquire]

@sjurba can you please check out karma@canary it contains a complete rewrite of the file-list

[sjurba]

Sure, but I'm not sure how I can install it.. I just get these:

npm ERR! peerinvalid The package karma does not satisfy its siblings' peerDependencies requirements!
npm ERR! peerinvalid Peer karma-ng-html2js-preprocessor@0.1.2 wants karma@>=0.9
npm ERR! peerinvalid Peer karma-ng-scenario@0.1.0 wants karma@>=0.9
npm ERR! peerinvalid Peer karma-safari-launcher@0.1.1 wants karma@>=0.9
npm ERR! peerinvalid Peer karma-sinon@1.0.4 wants karma@>=0.10
npm ERR! peerinvalid Peer grunt-karma@0.11.2 wants karma@^0.12 || >= 0.13.0-rc.0
npm ERR! peerinvalid Peer karma-junit-reporter@0.2.2 wants karma@>=0.9
npm ERR! peerinvalid Peer karma-spec-reporter@0.0.19 wants karma@>=0.9
npm ERR! peerinvalid Peer karma-phantomjs-launcher@0.2.0 wants karma@>=0.9

Nevermind, I figured it out..

mv node_modules/karma-* tmp/
npm install karma@canary
mv tmp/karma-* node_modules/

[dignifiedquire]

Yeah sorry about that, semver and rc versions don't get along too well :(

[sjurba]

Looks like the problem is still there in canary.. But aren't you still running preprocess right away instead of waiting for the file to be stabilised? https://github.com/karma-runner/karma/blob/v0.13.0-rc.6/lib/file-list.js#L341

When I set the autoWatchBatchDelay the problem got worse. It triggered the problem every time.. (In canary)

[dignifiedquire]

Thanks for checking, I just wanted to be sure. I think this is the same issue as described in #987 where empt files are handled incorrectly.

[Keats]

Hey @Dignifiedquire I just ran into that issue as well. Here's the repo https://github.com/Keats/react-example

For some reasons, changing tests in one of the files in tests/reducers/ work but it serves the cached version for tests/components/addForm_tests.js it never refreshes the cache. It does rerun the tests so it picks up changes though, just that it doesn't refresh

[antonioaguilar]

Hi there, I'm seeing strange caching behaviour from Karma similar to what @osi-oswald mentioned in a post last year. Basically, I have a grunt task that creates a karma.conf.js file whenever a spec file is updated which in turns runs Karma. So, in every spec file change we get a new karma.conf.js with different glob patterns for Karma to run. However, the problem I see is that Karma does not seem to pick up the newly created karma.conf.js and just re-runs the previous session? This is a very strange thing, I'm wondering what is the status of this bug and if we are going to ever be able to disable this caching behaviour in Karma?

[janpaul123]

Seeing the same problem, also when using built files (in this case using watchify).

[mannysz]

same problem here

[Cleod9]

^ I am also seeing this problem occur with karma's autoWatch enabled after webpack overwrites the build files. Doesn't happen every time which is weird, but once it starts the only way to fix is restarting karma. Would be nice to be able to force karma to start a clean session

[gcrick]

I had the same problem using autoWatch and Webpack when referencing my main webpack.config.js inside karma.conf.js (in a project based on the book https://leanpub.com/survivejs_webpack_react). I managed to solve it by removing the Webpack entry point after importing the config:

var webpackConfig = require('./webpack.config.js');
webpackConfig.entry = {};

as explained here: http://mike-ward.net/2015/09/07/tips-on-setting-up-karma-testing-with-webpack/

Update: Unfortunately, as the project has grown, the problem has returned and this solution is no longer sufficient :(

[JakeGinnivan]

Have the same issue.

My config file looks like this

const getBaseConfig = require('./karma.conf.js')
const argv = require('yargs').argv

if (!argv.file) {
  throw new Error('Expected to recieve file argument')
}

module.exports = (config) => {
  getBaseConfig(config)
  config.set({
    files: [
      './node_modules/babel-polyfill/browser.js',
      argv.file
    ],
    preprocessors: {
      [argv.file]: ['webpack', 'sourcemap']
    },
    client: {
      captureConsole: true
    },
    reporters: ['mocha'],
    mochaReporter: {
      output: 'full',
      divider: Array(40).join('=')
    }
  })
  console.log('Config: ', config, config.webpackConfig)
}

Then I have a task in VSCode which looks like this:

{
      "suppressTaskName": true,
      "taskName": "test",
      "isTestCommand": true,
      "isWatching": false,
      "args": [
        "node",
        "./node_modules/karma/bin/karma",
        "run",
        "karma.vscode.conf.js",
        "--port",
        "8001",
        "--file",
        "${file}"
      ]
    }

This allows me to simply pass the file I want to test to karma. It works if I use start but if I use run the file list seems to not be updated with the new config?

[Cleod9]

@gcrick Hmm, the thing is I don't actually use karma-webpack so Karma is unaware of this config for me. I just have webpack running its own watch in one process, and karma listening for changes to that build file in another process. Karma reacts to changes to that file properly (it's as if the build were just any other source file), though it seems to want to cache anyway.

[jchoca]

I've been running into what sounds like the same problem just using Gulp with Karma. Running karma with autoWatch: true and singleRun: false, it runs correctly the first time, but then subsequent changes are not reflected. I have my .spec files in the same directory as my app code, and I had a redundant, separate file glob for the .spec files, e.g.,

files: [
      'src/**/*.js',
      'src/**/*.spec.js'
    ]

If I remove the second file glob, it works as expected. I've replicated the problem here: https://github.com/jchoca/gulp-karma. If you run gulp tdd and comment out line 4 in hello.spec.js, it still logs 'hello'.

[levelfivehub]

Same problem here. I guess i have to subscribe in order to see if this gets resolved.

[johnkron]

I have the same problem, has any body found a solution?

[saturn72]

Facing the same issue

[ry8806]

another one here, not using webpack or anything like that, just standard karma with chrome runner and it's recognising the changed file, but serving a cached version

[evgd]

@jchoca

If I remove the second file glob, it works as expected great, seems you've found the solution that works for me. Thank you!

files: [
      'src/**/*.js'
      // 'src/**/*.spec.js'
    ]

[csvan]

I have had this issue with both Gulp and Webpack. It is crippling since it leads to so many false alarms during testing.

[nizhu]

We've been running into the same issue and have resorted to a workaround since Angular needs code to be loaded in order (not using a module system atm)

https://github.com/nizhu/karma-watch-glob-workaround

Also includes some minimal code to reproduce this issue so hopefully it can help get this fixed.

[afiedler]

I have the same same problem with karma 1.3.0. A workaround similar to the one @nizhu is using worked for me. What I did was remove all glob patterns from any files that were watched for changes. So from this:

files = [
   { pattern: 'src/**/*.js', watched: true },
   { pattern: 'vendor/**/*.js', watched: false }
]; 

To this:

files = [
   { pattern: 'src/app.js', watched: true },
   { pattern: 'src/home.js', watched: true },
   // and so on...
   { pattern: 'vendor/**/*.js', watched: false }
];

Obviously you'll want to do this programatically. The glob npm package helps. I ended up doing something similar to nizhu's npm package.

[octalmage]

I'm running into the same issue, the first test runs and works, but all auto-runs after that don't run the updated code. I tried the glob technique above but it didn't resolve the issue for me. Note that it does work if I run them locally, but in a Docker container it doesn't.

[sterzhakov]

Same problem karma 1.5.0!

[octalmage]

The webpack build time makes testing extremely painful without incremental builds/tests. Does anyone know why the watched: false solution wouldn't work for me?

[about-code]

Seems related to webpack/webpack/issues/556

[elkin0727]

I resolved my problem **adding a asterisk to file name(***) Before:

{ pattern: "/admin/moduleSearch.js", included: true, served: true }

After:

{ pattern: "/admin/*moduleSearch.js", included: true, served: true }

With a little change, the watcher don't remove my file when I change the file and the tests run again .

In my case, i need to do that because there are a lot of files in the same directory.

[theHacker]

The problem still exists with Karma 1.7.1.

I have two npm scripts: One to run karma in watch mode and another to run the webpack-dev-server in watch mode. I noticed a couple of times when karma is getting every change, the webpack-dev-server is not recompiling because of the change. As if karma were "consuming the file change notifies".

[daelmaak]

Issue still present in Karma 5.2, though it doesn't manifest itself always. However, when doing frequent changes to the source files, especially if karma is running atm, it ceases to receive updated test code and I have to restart it.

[fincha]

having the same issue here too...