Build failure: Error: Build aborted, Error: Could not find expected browser (chrome) locally.

[samreid]

While deploying 1.6 via grunt production --brands=phet,phet-io --branch=1.6, I received this email:

Build failure: Error: Build aborted, Error: Could not find expected browser (chrome) locally. Run npm install to download the correct Chromium revision (970485).. Sim = gravity-and-orbits Version = 1.6.0 Brands = phet,phet-io Locales = * Shas = {"comment":"# gravity-and-orbits 1.6.0 Thu Nov 17 2022 17:21:51 GMT-0700 (Mountain Standard Time)","assert":{"sha":"c13a8c7741d95fbd8e37619c66ebcca8686dffe7","branch":"HEAD"},"axon":{"sha":"b04fe3c1aa6176c44f6b033d827166c388b78afa","branch":"HEAD"},"brand":{"sha":"1e38bb54fec31bc737e3cb5f7118021abdeef076","branch":"HEAD"},"chipper":{"sha":"6ce9bf076772c2c7094bfed1a1efc3a2e5fa0325","branch":"HEAD"},"dot":{"sha":"d800d2c36cf7c621ab38fda5e8e59b06344732c7","branch":"HEAD"},"gravity-and-orbits":{"sha":"afb0896d2c3fa75143c8652fbe500cd889aed554","branch":"1.6"},"joist":{"sha":"948113fb3cb5268e8593cbaecf9c3d70344e3f19","branch":"HEAD"},"kite":{"sha":"82cc108b7790cd9a1b72e5950ce5f72b9f663d3c","branch":"HEAD"},"perennial-alias":{"sha":"24decc6b648229b6816e1f57a4323ae66d853f6a","branch":"HEAD"},"phet-core":{"sha":"955837b7b74bc7816d0c755b83eef4d8c6c2688a","branch":"HEAD"},"phet-io":{"sha":"3dd8813ee39213080f8fd014b6076fb4ca579deb","branch":"HEAD"},"phet-io-sim-specific":{"sha":"5cf91cad26dcf1828e1882ba00e7257e5d5591d3","branch":"HEAD"},"phet-io-wrappers":{"sha":"4561b5ea8c137661654d6ebafb0c72218141769e","branch":"HEAD"},"phetcommon":{"sha":"2e3125f719cec7fa52fbdea894c8cd75c257346c","branch":"HEAD"},"phetmarks":{"sha":"143e268ad1707316c7007ea09dc7c1600840a9e1","branch":"HEAD"},"query-string-machine":{"sha":"03e27127517adb19fba52be93acd7db989d9d6e8","branch":"HEAD"},"scenery":{"sha":"5df51f70e8aaeef36074a38cc872ac6a0c09a5f0","branch":"HEAD"},"scenery-phet":{"sha":"66487128e39357a47a28c4ed53436b0e8238c5bf","branch":"HEAD"},"sherpa":{"sha":"45afe7dd94ac065e1f54e82fdfce01ba2c8c3a2c","branch":"HEAD"},"studio":{"sha":"da310ee16acc13e7baa958e9ee7c4896fb7c0323","branch":"HEAD"},"sun":{"sha":"79e01827560f5480fe9231901d6ab4936f5d51c9","branch":"HEAD"},"tambo":{"sha":"a9826c1dbac6de85e621bb3c9bd176af052ab985","branch":"HEAD"},"tandem":{"sha":"112e3b29a8d3e0d5c759981870466fa0252923d1","branch":"HEAD"},"twixt":{"sha":"6f791754ff35cd6f8d349ab396ffa37374053778","branch":"HEAD"},"utterance-queue":{"sha":"388e1950fbba71718e903b9abf6e6b5aab706721","branch":"HEAD"}}

It appears the 1.6 phet brand version was built correctly, I can see it at: https://phet.colorado.edu/sims/html/gravity-and-orbits/1.6.0/gravity-and-orbits_all.html

Maybe something went wrong with the chromium puppeteer step on the build server? Could it be related to changes in https://github.com/phetsims/perennial/issues/268 or an associated maintenance release? Assigning @jonathanolson and @zepumph since they were working on puppeteer today, and assigning @mattpen in case it's more related to the build server.

This is the last step before publication of the sim.

[zepumph]

Hmmm I am confused by this. If it has something to do with my changes (commits right below https://github.com/phetsims/perennial/issues/268#issuecomment-1319114110) then (I'm especially confused). This is because I would have thought that the build server would be using the older version of puppeteer, with no trouble (chipper sha:https://github.com/phetsims/gravity-and-orbits/blob/c544ccefac43c2f4b3b01f739ffbb951c493eb46/dependencies.json#L16 puppeteer version: https://github.com/phetsims/chipper/blob/6ce9bf076772c2c7094bfed1a1efc3a2e5fa0325/package.json#L44).

So my best guess is that it has something to do with npm prune/update commands (swapping between different versions), or something unrelated that @jonathanolson has done today. I'm out tomorrow but will check in on this Monday. Good luck!

[samreid]

Perhaps this failed build left the PhET brand in an inconsistent state, the images are not loading:

[samreid]

The relevant part from the build logs is:

Nov 17 17:29:15 phet-server2.int.colorado.edu build-server[2767]: debug: Copy finished
Nov 17 17:29:15 phet-server2.int.colorado.edu build-server[2767]: info: Writing file to path: /data/web/static/phetsims/sims/html/gravity-and-orbits/.htaccess
Nov 17 17:29:16 phet-server2.int.colorado.edu build-server[2767]: debug: Write success.
Nov 17 17:29:16 phet-server2.int.colorado.edu build-server[2767]: error: BUILD ABORTED! Error: Could not find expected browser (chrome) locally. Run `npm install` to download the correct Chromium revision (970485).
Nov 17 17:29:16 phet-server2.int.colorado.edu build-server[2767]: info: build aborted: checking out master for every repo in case build shas are still checked out

The relevant code is in taskWorker.js in this section:

          winston.debug( 'Copy finished' );

          // Post-copy steps
          if ( brand === constants.PHET_BRAND ) {
            await writePhetHtaccess( simName, version );
            await createTranslationsXML( simName, version );
            await notifyServer( {
              simName: simName,
              email: email,
              brand: brand,
              locales: locales,
              translatorId: isTranslationRequest ? userId : undefined
            } );
          }

So it seems the writePhetHtaccess succeeded because it said Write success. on the .htaccess. But it seems createTranslationsXML failed because it has winston.log( 'info', 'wrote XML file' ); at the end that never prints.

[samreid]

parseScreenNames is called from createTranslationsXML and has this code:

const puppeteer = require( 'puppeteer' );

And it uses puppeteer to load the screen names. So it is probably using the new puppeteer 19.2 introduced in https://github.com/phetsims/perennial/issues/268 in https://github.com/phetsims/binder/commit/ff58030c89dc12eaac796362858cc15bdf984a4c

[samreid]

Using this patch,

```diff Subject: [PATCH] Avoid accessing undefined card, see https://github.com/phetsims/center-and-variability/issues/97 --- Index: js/build-server/parseScreenNames.js IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/build-server/parseScreenNames.js b/js/build-server/parseScreenNames.js --- a/js/build-server/parseScreenNames.js (revision b13d8ed5ce2dd21bc1d5e52c10b31bfbb4c1ecea) +++ b/js/build-server/parseScreenNames.js (date 1668741353564) @@ -66,4 +66,9 @@ module.exports = { parseScreenNames: parseScreenNames, parseScreenNamesAllSimulations: parseScreenNamesAllSimulations -}; \ No newline at end of file +}; + +( async () => { + const m = await parseScreenNames( 'gravity-and-orbits', 'en' ); + console.log( m ); +} )(); ```

I ran (locally)

~/apache-document-root/main/perennial-alias$ npm info puppeteer

puppeteer@19.2.2 | Apache-2.0 | deps: 6 | versions: 780
A high-level API to control headless Chrome over the DevTools Protocol
https://github.com/puppeteer/puppeteer/tree/main#readme

....

    at node:internal/main/run_main_module:17:47
~/apache-document-root/main/perennial-alias$ node js/build-server/parseScreenNames.js 
{ e: [ '‪Model‬', '‪To Scale‬' ], n: [ '‪Model‬', '‪To Scale‬' ]

So it does not seem to be an inherent problem or incompatibility with 19.2

[samreid]

In the change logs, puppeteer says 19.0.0 introduced a breaking change:

https://github.com/puppeteer/puppeteer/releases/tag/v19.0.0

Others have reported problems like ours: https://github.com/puppeteer/puppeteer/issues/9218

Maybe I should run an experiment by rolling back this change to see how it goes. But we should try to investigate and understand this caching better, since @jonathanolson commented today he was looking into ways to cache the puppeteer chromiums.

[zepumph]

I am so very sorry that you ran into this. I didn't know that the build server used puppeteer. I do not believe that this has anything to do with the above breaking changes, but rather, I didn't realize that the build server most likely needs to be restarted to accommodate the new puppeteer version that was installed. I'll work with @mattpen on this next week.

[mattpen]

Looks like this is being addressed in https://github.com/phetsims/perennial/issues/268

@zepumph - is this ok to close?

[zepumph]

Yes thanks. This worked great over in https://github.com/phetsims/perennial/issues/268. We just needed to restart the build server to get the new puppeteer npm version into memory. Closing