Flaky integration test: Switching languages, localization.test.js

Expected Behavior

A test should not fail other tests!

Actual Behavior

When adding tests to the file, it always failed with:

  ● Localization › Switching languages

    Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

      at node_modules/jest-jasmine2/build/queue_runner.js:64:21
      at Timeout.callback [as _onTimeout] (node_modules/jsdom/lib/jsdom/browser/Window.js:523:19)
      at ontimeout (timers.js:498:11)
      at tryOnTimeout (timers.js:323:5)
      at Timer.listOnTimeout (timers.js:290:5)

  ● Localization › Loading with locale shows correct translation for string length block parameter

    TimeoutError: findByXpath timed out for path: //body//*//*[contains(text(), 'Costumes')]
    Wait timed out after 20570ms

      at node_modules/selenium-webdriver/lib/promise.js:2201:17
      at ManagedPromise.invokeCallback_ (node_modules/selenium-webdriver/lib/promise.js:1376:14)
      at TaskQueue.execute_ (node_modules/selenium-webdriver/lib/promise.js:3084:14)
      at TaskQueue.executeNext_ (node_modules/selenium-webdriver/lib/promise.js:3067:27)
      at asyncRun (node_modules/selenium-webdriver/lib/promise.js:2927:27)
      at node_modules/selenium-webdriver/lib/promise.js:668:7
          at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:189:7)
      From: Task: findByXpath timed out for path: //body//*//*[contains(text(), 'Costumes')]
      at scheduleWait (node_modules/selenium-webdriver/lib/promise.js:2188:20)
      at ControlFlow.wait (node_modules/selenium-webdriver/lib/promise.js:2517:12)
      at thenableWebDriverProxy.wait (node_modules/selenium-webdriver/lib/webdriver.js:934:29)
      at SeleniumHelper.findByXpath (test/helpers/selenium-helper.js:93:26)
      at SeleniumHelper.wrapper [as findByXpath] (node_modules/lodash.bindall/index.js:550:15)
      at SeleniumHelper.findByText (test/helpers/selenium-helper.js:101:19)
      at SeleniumHelper.wrapper [as findByText] (node_modules/lodash.bindall/index.js:550:15)
      at SeleniumHelper.clickText (test/helpers/selenium-helper.js:124:19)
      at wrapper (node_modules/lodash.bindall/index.js:550:15)
      at _callee2$ (test/integration/localization.test.js:34:15)
      at tryCatch (node_modules/regenerator-runtime/runtime.js:62:40)
      at Generator.invoke [as _invoke] (node_modules/regenerator-runtime/runtime.js:296:22)
      at Generator.prototype.(anonymous function) [as next] (node_modules/regenerator-runtime/runtime.js:114:21)
      at asyncGeneratorStep (test/integration/localization.test.js:2:278)
      at _next (test/integration/localization.test.js:2:616)
          at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:189:7)

Note: the string length test is the one I added. I definitely never used findByXPath with costume related thing in that test. I skipped the Switching language test (using test.skip) and this time, my own test that failed earlier passed.

  Localization
    √ Loading with locale shows correct blocks (14775ms)
    √ Loading with locale shows correct translation for string length block parameter (4611ms)
    ○ skipped 1 test

This probably means that the skipped test, Switching languages, is causing other tests to fail.

Steps to Reproduce

Add this to the test. NOTE: Until the fix PR that I mentioned earlier is merged, replace '3' with '5'.

    test('Loading with locale shows correct translation for string length block parameter', async () => {
        await loadUri(`${uri}?locale=ja`);
        await clickText('演算'); // Operators category in Japanese
        await new Promise(resolve => setTimeout(resolve, 1000)); // wait for blocks to scroll
        await clickText('の長さ', scope.blocksTab); // Click "length <apple>" block
        await findByText('3', scope.reportedValue); // Tooltip with result
        const logs = await getLogs();
        await expect(logs).toEqual([]);
    });

Operating System and Browser

Windows 10, Node v8.16.1, npm 6.4.1

scratchfoundation / scratch-gui