Cypress tests fail in air-gapped network on Firefox

ChadiEM commented 1 month ago

Current behavior

In an air gapped network, running a component or e2e tests using cypress on Firefox using the cypress/included docker image fails with the following error:

Cypress failed to make a connection to Firefox.

This usually indicates there was a problem opening the Firefox browser.

FetchError: request to https://raw.githubusercontent.com/mozilla/geckodriver/release/Cargo.toml failed, reason: read ECONNRESET
    at ClientRequest.<anonymous> (file:///root/.cache/Cypress/14.0.2/Cypress/resources/app/packages/server/node_modules/node-fetch/src/index.js:108:11)
    at ClientRequest.emit (node:events:518:28)
    at emitErrorEvent (node:_http_client:101:11)
    at TLSSocket.socketErrorListener (node:_http_client:504:5)
    at TLSSocket.emit (node:events:518:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

This is likely a regression introduced in release 13.15.1 which started using geckodriver.

Desired behavior

I expect the test to run in an air gapped network. It should not send any request to the internet.

Test code to reproduce

Any test in Firefox in an air-gapped network.

Cypress Version

14.0.2

Node version

v22.13.1

Operating System

Debian bookworm

Debug Logs

Other

I also attempted to override the environment variables supported by geckodriver to use a local path

GECKODRIVER_VERSION = '0.35.0'
GECKODRIVER_CDNURL = 'file:///path/to/geckodriver/'

But it failed node-fetch does not support file:.

MikeMcC399 commented 1 month ago

The issue is reproducible with cypress/included and cypress/browsers Docker images. Edit: ~~It does not appear to be an issue in native Ubuntu or Windows environments.~~

Steps to reproduce

git clone https://github.com/cypress-io/cypress-docker-images
cd cypress-docker-images

cd examples/basic-mini
docker run -it --rm -v .:/app -w /app --entrypoint cypress cypress/included:14.0.2 run -b firefox

OR

cd examples/basic
npm install cypress@14.0.2
docker build . -f Dockerfile.browsers -t test-browsers
docker run -it --rm --entrypoint bash test-browsers -c "npx cypress run -b firefox"

Run first with network connection
Repeat final docker run with disconnected network

Cypress:        14.0.2
Browser:        Firefox 135 (headless)
Node Version:   v22.13.1 (/usr/local/bin/node)

MikeMcC399 commented 1 month ago

https://www.npmjs.com/package/geckodriver

found in https://github.com/cypress-io/cypress/blob/d1a75e3de7947fbfff982a9d3f6ca51253fdfe04/packages/server/package.json#L76

attempts to download a driver https://github.com/webdriverio-community/node-geckodriver/blob/92a64548339a9fe2fa6902498e15e506b45213a1/src/constants.ts

export const GECKODRIVER_CARGO_YAML = 'https://raw.githubusercontent.com/mozilla/geckodriver/release/Cargo.toml'

AtofStryker commented 1 month ago

We should be able to preinstall geckodriver in those images before distributing them so at least we know.a version of the driver is there.

MikeMcC399 commented 1 month ago

@AtofStryker

We should be able to preinstall geckodriver in those images before distributing them so at least we know.a version of the driver is there.

I repeated my tests and found that the issue is also reproducible without using a Cypress Docker image, so any fix would be better placed directly in Cypress rather than in a Cypress Docker image.

Ubuntu 24.04.1 LTS, Firefox 135.0, Node.js 22.13.1 LTS

First with connected network:

git clone https://github.com/cypress-io/cypress-docker-images
cd cypress-docker-images
cd examples/basic
npm ci
npm install cypress@latest

Disconnect network, reboot, then navigate to same directory cypress-docker-images/cd examples/basic

npx cypress run -b firefox

Debug Logs

DEBUG=cypress:server:browsers:firefox npx cypress run -b firefox

DevTools listening on ws://127.0.0.1:39059/devtools/browser/693301c6-de97-411f-8194-95c0ac96e6d6

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        14.0.2                                                                         │
  │ Browser:        Firefox 135 (headless)                                                         │
  │ Node Version:   v22.13.1 (/home/mike/n/bin/node)                                               │
  │ Specs:          1 found (spec.cy.js)                                                           │
  │ Searched:       cypress/e2e/**/*.cy.{js,jsx,ts,tsx}                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  spec.cy.js                                                                      (1 of 1)
  cypress:server:browsers:firefox firefox open { browser: { name: 'firefox', family: 'firefox', channel: 'stable', displayName: 'Firefox', version: '135.0', path: 'firefox', profilePath: '/home/mike/snap/firefox/current', majorVersion: '135', isHeadless: true, isHeaded: false }, url: 'http://localhost:35789/__/#/specs/runner?file=cypress/e2e/spec.cy.js', browsers: [ { name: 'chrome', family: 'chromium', channel: 'stable', displayName: 'Chrome', version: '131.0.6778.204', path: 'google-chrome', majorVersion: '131' }, { name: 'chrome-for-testing', family: 'chromium', channel: 'stable', displayName: 'Chrome for Testing', version: '131.0.6778.204', path: 'chrome', majorVersion: '131' }, { name: 'chromium', family: 'chromium', channel: 'stable', displayName: 'Chromium', version: '132.0.6834.110', path: 'chromium-browser', profilePath: '/home/mike/snap/chromium/current', majorVersion: '132' }, { name: 'firefox', family: 'firefox', channel: 'stable', displayName: 'Firefox', version: '135.0', path: 'firefox', profilePath: '/home/mike/snap/firefox/current', majorVersion: '135' }, { name: 'edge', family: 'chromium', channel: 'stable', displayName: 'Edge', version: '129.0.2792.52', path: 'microsoft-edge', majorVersion: '129' }, { name: 'electron', channel: 'stable', family: 'chromium', displayName: 'Electron', version: '130.0.6723.137', path: '', majorVersion: 130 } ], userAgent: null, proxyUrl: 'http://localhost:35789', proxyServer: 'http://localhost:35789', socketIoRoute: '/__socket', chromeWebSecurity: true, isTextTerminal: true, downloadsFolder: '/home/mike/github/cypress-io/cypress-docker-images/examples/basic/cypress/downloads', experimentalModifyObstructiveThirdPartyCode: false, experimentalWebKitSupport: false, protocolManager: undefined, projectRoot: '/home/mike/github/cypress-io/cypress-docker-images/examples/basic', shouldLaunchNewTab: false, onError: [Function (anonymous)], videoApi: undefined, automationMiddleware: { onBeforeRequest: [Function: onBeforeRequest], onAfterResponse: [Function: onAfterResponse] }, onWarning: [Function: onWarning], onBrowserClose: [Function (anonymous)], relaunchBrowser: [AsyncFunction (anonymous)], onBrowserOpen: [Function: onBrowserOpen] } +0ms
  cypress:server:browsers:firefox available ports: { marionettePort: 43297, webDriverBiDiPort: 34367 } +3ms
  cypress:server:browsers:firefox firefox directories { path: '/home/mike/snap/firefox/current/Cypress/firefox-stable/run-7720', cacheDir: '/home/mike/snap/firefox/current/Cypress/firefox-stable/run-7720/CypressCache', extensionDest: '/home/mike/snap/firefox/current/Cypress/firefox-stable/run-7720/CypressExtension' } +27ms
  cypress:server:browsers:firefox launching geckodriver with browser envs { MOZ_REMOTE_SETTINGS_DEVTOOLS: '1', MOZ_HEADLESS_WIDTH: '1280', MOZ_HEADLESS_HEIGHT: '720' } +1ms
  cypress:server:browsers:firefox launch in firefox {
  url: 'http://localhost:35789/__/#/specs/runner?file=cypress/e2e/spec.cy.js',
  args: [
    '-new-instance',
    '-start-debugger-server',
    '-no-remote',
    '-headless'
  ]
} +0ms
Cypress failed to make a connection to Firefox.

This usually indicates there was a problem opening the Firefox browser.

FetchError: request to https://raw.githubusercontent.com/mozilla/geckodriver/release/Cargo.toml failed, reason: getaddrinfo EAI_AGAIN raw.githubusercontent.com
    at ClientRequest.<anonymous> (file:///home/mike/.cache/Cypress/14.0.2/Cypress/resources/app/packages/server/node_modules/node-fetch/src/index.js:108:11)
    at ClientRequest.emit (node:events:518:28)
    at emitErrorEvent (node:_http_client:101:11)
    at TLSSocket.socketErrorListener (node:_http_client:504:5)
    at TLSSocket.emit (node:events:518:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

AtofStryker commented 1 month ago

I repeated my tests and found that the issue is also reproducible without using a Cypress Docker image, so any fix would be better placed directly in Cypress rather than in a Cypress Docker image.

@MikeMcC399 I don't know how feasible this would be because we would need to ship every OS specific version of the driver with each OS specific version of the binary for a not very common use case. I'm suggesting we ship it in the docker container to at least mitigate the inconvenience of downloading the driver for the most likely use case of having an offline system.

The tests should still run offline, assuming geckodriver is already downloaded on the machine running the tests.

MikeMcC399 commented 1 month ago

@AtofStryker

I repeated my tests and found that the issue is also reproducible without using a Cypress Docker image, so any fix would be better placed directly in Cypress rather than in a Cypress Docker image.

@MikeMcC399 I don't know how feasible this would be because we would need to ship every OS specific version of the driver with each OS specific version of the binary for a not very common use case. I'm suggesting we ship it in the docker container to at least mitigate the inconvenience of downloading the driver for the most likely use case of having an offline system.

The tests should still run offline, assuming geckodriver is already downloaded on the machine running the tests.

I see that would be complicated to do for Cypress in general. Let me open a new issue in the Cypress Docker repo and describe the issue testing Firefox ESR offline (available for amd64 and arm64). If a fix goes into the Docker images it would only need to be for Debian.

MikeMcC399 commented 1 month ago

I can offer the following comments & suggestions:

To allow testing of Firefox with Cypress inside an air-gapped network requires the provisioning of Geckodriver binaries to the air-gapped network. Cypress uses the npm module geckodriver to download the driver for use with Firefox.

If a custom Cypress Docker image is built in a network connected to Internet with a Dockerfile which runs Cypress, such as with the addition of the following Dockerfile command:

RUN npx cypress run -b firefox

then the Docker image will include the Geckodriver saved to /tmp/geckodriver. The Internet connection can be dropped and the image can be used in the air-gapped network.

It should also be possible to use the information Setting a CDN URL for binary download which describes creating a local copy of the Geckodriver CDN and using the environment variable GECKODRIVER_CDNURL as in the following example:

GECKODRIVER_CDNURL=https://INTERNAL_CDN/geckodriver/download

As noted in the OP, the file: protocol is not supported using GECKODRIVER_CDNURL with Cypress.

Reasons for not adding a cached Geckodriver to Cypress itself were already given in https://github.com/cypress-io/cypress/issues/31036#issuecomment-2643206000. Now, after looking at this in detail, I don't feel that there should be any steps added to standard Cypress Docker images to include cached Geckodriver images either, unless there were to be a high demand from multiple users. This is a special case which is better dealt with by individual customization. I have decided against opening a new issue in the cypress-io/cypress-docker-images/issues for this reason.

MikeMcC399 commented 1 month ago

@ChadiEM

If you need more specific advice for using Cypress Docker images to test Firefox in an air-gapped network, I suggest that you open a new issue in https://github.com/cypress-io/cypress-docker-images/issues. It might be helpful to add this as a known issue with a documented workaround. For instance, using GECKODRIVER_AUTO_INSTALL=1 npm install geckodriver seems like a convenient way to get the driver installed and saved.

ChadiEM commented 1 month ago

Thanks for the help and the analysis, @MikeMcC399 & @AtofStryker.

I created https://github.com/cypress-io/cypress-docker-images/issues/1298.

MikeMcC399 commented 1 month ago

@ChadiEM

Many thanks for opening the new issue https://github.com/cypress-io/cypress-docker-images/issues/1298 where I have responded with a proposal for you to build a custom Docker image that contains a cached geckodriver. This is a solution which is immediately available with no additional development effort needed from the Cypress side.

MikeMcC399 commented 1 month ago

As a spin-off to this issue I would suggest that the need for Internet connectivity when testing Firefox be added to the Cypress documentation, for instance https://docs.cypress.io/app/references/launching-browsers#Firefox-Browsers
PR https://github.com/cypress-io/cypress/pull/30250 was merged into cypress@13.15.1 in Oct 2024, which is 3 months ago, and this issue is the first to report a problem with air-gapped testing. This probably means that this issue is not affecting most users, otherwise it would have been reported earlier.

MikeMcC399 commented 1 week ago

Similar issue without Docker reported in https://github.com/cypress-io/cypress/issues/31230

cypress-io / cypress