water-abstraction-service npm ci fail in dev but not test

[Cruikshanks]

Specifically, this issue happened when we deployed this commit to our dev and tst environments.

It's not that change which is the problem, it just happens to be the version we were pushing. Most likely it is this change; Bump puppeteer from 19.8.0 to 19.8.2

The water-api and service-background apps in pm2 had errored. When we checked the logs we saw this

Error: Cannot find module 'dotenv'
Require stack:
- /srv/app/node/water-abstraction-service/releases/20230331_084029/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
    at Module.Hook._require.Module.require (/usr/lib/node_modules/pm2/node_modules/require-in-the-middle/index.js:61:29)
    at require (internal/modules/cjs/helpers.js:101:18)
    at Object.<anonymous> (/srv/app/node/water-abstraction-service/releases/20230331_084029/index.js:3:1)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Object.<anonymous> (/usr/lib/node_modules/pm2/lib/ProcessContainerFork.js:33:23)
    at Module._compile (internal/modules/cjs/loader.js:1085:14) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/srv/app/node/water-abstraction-service/releases/20230331_084029/index.js'
  ]
}

The fault isn't with dotenv. It's because npm ci is failing. When you run it you see

npm ERR! code 1
npm ERR! path /srv/app/node/water-abstraction-service/releases/20230331_084029/node_modules/puppeteer
npm ERR! command failed
npm ERR! command sh -c node install.js
npm ERR! ERROR: Failed to set up Chromium r1108766! Set "PUPPETEER_SKIP_DOWNLOAD" env variable to skip download.
npm ERR! Error: Download failed: server returned code 502. URL: https://storage.googleapis.com/chromium-browser-snapshots/Linux_x64/1108766/chrome-linux.zip
npm ERR!     at /srv/app/node/water-abstraction-service/releases/20230331_084029/node_modules/@puppeteer/browsers/lib/cjs/httpUtil.js:121:31
npm ERR!     at ClientRequest.requestCallback (/srv/app/node/water-abstraction-service/releases/20230331_084029/node_modules/@puppeteer/browsers/lib/cjs/httpUtil.js:98:13)
npm ERR!     at Object.onceWrapper (events.js:520:26)
npm ERR!     at ClientRequest.emit (events.js:400:28)
npm ERR!     at HTTPParser.parserOnIncomingClient (_http_client.js:647:27)
npm ERR!     at HTTPParser.parserOnHeadersComplete (_http_common.js:127:17)
npm ERR!     at Socket.socketOnData (_http_client.js:515:22)
npm ERR!     at Socket.emit (events.js:400:28)
npm ERR!     at addChunk (internal/streams/readable.js:293:12)
npm ERR!     at readableAddChunk (internal/streams/readable.js:267:9)

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/ubuntu/.npm/_logs/2023-03-31T08_49_38_713Z-debug-0.log

To get the apps running in dev we did what it suggested and ran PUPPETEER_SKIP_DOWNLOAD=true npm ci in /srv/app/node/water-abstraction-service/current on both BES001 and BES002.

It could just be we got unlucky with a network issue. But it could also be a difference in the environments we are not aware of blocking the download.

We will need to keep an eye on it. And if there is any concern either revert to the previous version and hope that fixes the issue or dig into the root cause. The concern is the same thing happens when we next try to release to production.