ember-cli / ember-exam

Run your tests with randomization, splitting, and parallelization for beautiful tests.
MIT License
286 stars 61 forks source link

Some tests fail using --split option while using ember-i18n #100

Open jzisser9 opened 7 years ago

jzisser9 commented 7 years ago

Hi there, We've been using ember-exam in our project for a while, and it's worked great. However, recently we also began using ember-i18n and have noticed that some of our tests began to fail with the following message: Error: Assertion Failed: I18n: Cannot translate when locale is null.

We usually split our tests into 5 partitions and I can reliably reproduce these failures on partitions 1 and 3. Running the tests without the --split command causes all of the tests to pass.

We configure the default locale for i18n in our config/environment.js file. We also use testem, and our testem.json file is as follows:

{
  "framework": "qunit",
  "test_page": "tests/index.html?hidepassed",
  "phantomjs_debug_port": 5001,
  "disable_watching": true,
  "launch_in_ci": [
    "PhantomJS"
  ],
  "launch_in_dev": [
    "PhantomJS",
    "Chrome"
  ],
  "proxies": {
    "/api": {
      "port": 3000,
      "host": "localhost"
    },
    "/users": {
      "port": 3000,
      "host": "localhost"
    }
  }
}

I wanted to report this here in case it's buggy behavior, but I was also hoping to see if we can diagnose and fix the issue without needing a bugfix. How does --split work? What does the process under the hood look like compared to running tests without the --split option? It seems like maybe what's going on is that our configuration options aren't "taking" to each partition, but that's just a shot in the dark since I don't know exactly how --split works.

We are running the following software versions:

ember-cli: 2.7.0 node: 7.9.0 os: darwin x64

ekcrisp commented 6 years ago

i think we have the same issue

ekcrisp commented 6 years ago

I noticed when using ember-exam application initializers are not run before every test, this is the cause of the issue in our case, we use an initializer to load i18n content, sometimes this content changes based on the test and with exam we have to explicitly run the initializer in order for this to work, without exam it runs automatically