[BUG] Mocha test that passes locally fails in CI

[joewiz]

Describe the bug

A mocha test passes locally for me but is failing in CI. It’s a test that originated from this one in the generator-exist package. Here’s my app's version of the test, and here’s the failure in CI. From the logs, it looks like the request to /exist/rest/db/apps/airlock is returning a 404 in CI, whereas on my machine this request is returning a 200 as I’d expect.

When I described this to @duncdrum, he wrote:

Hi joe, yes please open an issue, I had someone else report this f2f but need some time before i can reproduce, this so all info is welcome

I'm happy to provide any other info that would be useful. Thanks in advance!

Expected behavior

I'd expect a test that passes locally to also pass in CI.

To Reproduce

Here is a trimmed version of the mocha test that fails in CI but passes locally:

'use strict'

const supertest = require('supertest')
const expect = require('chai').expect

let client = supertest.agent('http://localhost:8080')

describe('rest api returns', function () {
    it('application root is available from rest endpoint', function (done) {
      client
        .get('/exist/rest/db/apps/airlock')
        .expect(200)
        .end(function (err, res) {
          expect(res.status).to.equal(200)
          if (err) return done(err)
          done()
        })
    })
  })

Running npm test locally, I get:

joe@choskimac-iii airlock % npm test       

> airlock@1.1.0-SNAPSHOT test
> mocha test/mocha/ --recursive --exit && mocha test/xqs/*.js

  file system checks
    markup files are well-formed
      ✓ *.html is xhtml
      ✓ *.xml
      ✓ *.xconf
      ✓ *.odd
    Consistent data in aux files
      ✓ should contain identical descriptions
      ✓ should contain identical versions
      ✓ should contain identical licenses
      ✓ should contain identical titles
      ✓ Readme should have latest meta-data

  rest api returns
    ✓ 404 from random page
    ✓ 200 from default rest endpoint
    ✓ application root is available from rest endpoint

  12 passing (65ms)

  0 passing (1ms)

  Xqsuite tests for http://joewiz.org/ns/app/airlock/tests
    ✓ Test: one-is-one

  1 passing (2ms)

On CI it returns:

0s
Run npm test
  npm test
  shell: /usr/bin/bash -e {0}
  env:
    JAVA_HOME_8.0.282_x64: /opt/hostedtoolcache/jdk/8.0.282/x64
    JAVA_HOME: /opt/hostedtoolcache/jdk/8.0.282/x64
    JAVA_HOME_8_0_282_X64: /opt/hostedtoolcache/jdk/8.0.282/x64

> airlock@1.1.0-SNAPSHOT test /home/runner/work/airlock/airlock
> mocha test/mocha/ --recursive --exit && mocha test/xqs/*.js

  file system checks
    markup files are well-formed
      ✓ *.html is xhtml
      ✓ *.xml
      ✓ *.xconf
      ✓ *.odd
    Consistent data in aux files
      ✓ should contain identical descriptions
      ✓ should contain identical versions
      ✓ should contain identical licenses
      ✓ should contain identical titles
      ✓ Readme should have latest meta-data

  rest api returns
    ✓ 404 from random page (77ms)
    ✓ 200 from default rest endpoint (66ms)
    1) application root is available from rest endpoint

  11 passing (226ms)
  1 failing

  1) rest api returns
       application root is available from rest endpoint:

      Uncaught AssertionError: expected 404 to equal 200
      + expected - actual

      -404
      +200

      at Test.<anonymous> (test/mocha/rest_spec.js:37:33)
      at Test.assert (node_modules/supertest/lib/test.js:209:6)
      at localAssert (node_modules/supertest/lib/test.js:159:12)
      at /home/runner/work/airlock/airlock/node_modules/supertest/lib/test.js:156:5
      at Test.Request.callback (node_modules/superagent/lib/node/index.js:905:3)
      at IncomingMessage.<anonymous> (node_modules/superagent/lib/node/index.js:1127:20)
      at endReadableNT (_stream_readable.js:1241:12)
      at processTicksAndRejections (internal/process/task_queues.js:84:21)

npm ERR! Test failed.  See above for more details.
Error: Process completed with exit code 1.

Context (please always complete the following information):

• OS: macOS 11.2.3
• eXist-db Version: eXist 5.3.0-SNAPSHOT 23f05d9f3baef31ae18b751beb398785468bb28b 20210307114828
• Java Version: openjdk version "1.8.0_282" (Liberica OpenJDK)
  • App Version: I generated my app's scaffolding with generator-exist on 11 Feb 2021.

Additional context

• How is eXist-db installed? built from source
• Any custom changes in e.g. conf.xml? none

[duncdrum]

hmm something is strange in this neighbourhood. OOTB i m getting passing tests both on CI and local when doing this.

SO my initial guess was to increase the timeout like this

describe('rest api returns', function () {
this.timeout(1500)
    it('application root is available from rest endpoint', function (done) {
      client
        .get('/exist/rest/db/apps/airlock')
        .expect(200)
        .end(function (err, res) {
          expect(res.status).to.equal(200)
          if (err) return done(err)
          done()
        })
    })
  })

which since it was green before is still green. But maybe you can give that a try to see if that solves the problem on ci.

However, digging a little deeper i noticed, that:

curl -i http://localhost:8080/exist/apps/GH646/index.html returns (among others): HTTP/1.1 200 OK

while a head request: curl -I http://localhost:8080/exist/apps/GH646/index.html returns: HTTP/1.1 400 Bad Request

i ll have to dig deeper into that. Calls to the rest endpoint are 200 as expected.

[joewiz]

@duncdrum Thanks for looking into this! I tried increasing the timeout, but it didn't change the results - so tests still pass locally but fail in CI. My experience with curl matches yours, except that in my app I get a 405 Method Not Allowed error in the -I variant. I'm still in an early phase of learning Roaster, but then again, the tests pass locally. Here's the latest: https://github.com/joewiz/airlock/actions/runs/678852673.