Investigate recent build failures

[jbphet]

I had seven "BUILD ERROR" messages in my inbox this morn. They started at 3:36 AM, and the most recent one occurred at 7:22 AM. Based on a quick look, these all seem related to translation build requests, and the locales are ha, de, and it. The core error is the same in al seven emails, and is:

Error: Could not find expected browser (chrome) locally. Run `npm install` to download the correct Chromium revision (970485).

Below is a full copy of one of the emails (the first one):

Build failure: Error: Build aborted, Error: Could not find expected browser (chrome) locally. Run `npm install` to download the correct Chromium revision (970485).. Sim = states-of-matter-basics Version = 1.2.4 Brands = phet Locales = ha Shas = {"comment":"# states-of-matter-basics 1.2.4 Mon Jan 31 2022 14:44:01 GMT-0700 (Mountain Standard Time)","assert":{"sha":"1a41cef653bbed1025af5768bea958d7e3e0f8ae","branch":"HEAD"},"axon":{"sha":"6ae790812d04ab65fd72c79c7da1e27442c0a70d","branch":"HEAD"},"brand":{"sha":"83c51961edf5b255206c82de7d6f4df6b4ea41bc","branch":"HEAD"},"chipper":{"sha":"843b2dd2cad4030906dd26c6a1b0271342915f89","branch":"HEAD"},"dot":{"sha":"3aa05218545989fcd42fcf53c6c4e6982605a6f2","branch":"HEAD"},"joist":{"sha":"4bcb0bd42a3133bc48d8522fc64e17e884ebbac9","branch":"HEAD"},"kite":{"sha":"ed0c49f276c62558ca1d695362807ee4dcaed80c","branch":"HEAD"},"nitroglycerin":{"sha":"419889d5d52004b54d09261c9f2aca42654d0c98","branch":"HEAD"},"phet-core":{"sha":"6144cb330e52b20cadda567aa3dcb592f668ff17","branch":"HEAD"},"phet-io":{"sha":"d519ff505fe4233887188c30ef3a8ef3b730fb06","branch":"HEAD"},"phet-io-client-guides":{"sha":"abe6a71ed5456d7e98a2c757122e9b4aadf2cb2d","branch":"HEAD"},"phet-io-wrappers":{"sha":"35caf0824fa7f1619a54a09d413659648c6d45fc","branch":"HEAD"},"phetcommon":{"sha":"7efe73b2044421640a7a2c71a9a41d3e37ecd0bb","branch":"HEAD"},"phetmarks":{"sha":"29eff6760574ef63197aba73b578bdb683cfac64","branch":"HEAD"},"query-string-machine":{"sha":"98c27e38eb6ce672dd8908c5210a521f112ff906","branch":"HEAD"},"scenery":{"sha":"5d44177d69f760ecd3dad33f98a7b47fc19d9077","branch":"HEAD"},"scenery-phet":{"sha":"561fdfe616c8ddee6a34414858218ab4e696b0d3","branch":"HEAD"},"sherpa":{"sha":"c351f9babb7af524b2eb713b82b11fd4379b466c","branch":"HEAD"},"states-of-matter":{"sha":"900e7eeff4aca6fa5e489d4ecd09c867606907e0","branch":"HEAD"},"states-of-matter-basics":{"sha":"317730ac371db54cd1a88d2eac4b8ed91b01412a","branch":"HEAD"},"studio":{"sha":"2eff97e07ec82d3ee0e80d2fd586c1f496fecd07","branch":"HEAD"},"sun":{"sha":"6f5cab57e8f897575e48e4c255a7e19d7b6bdd79","branch":"HEAD"},"tambo":{"sha":"d66cceac1ef50830a081a8cb695706dbf2b8d3fe","branch":"HEAD"},"tandem":{"sha":"0d7299a5895ca9fac44a7df550c715a26d01a017","branch":"HEAD"},"utterance-queue":{"sha":"c2b44d1ca355707b2c2bae2a173c6ad26b5e6613","branch":"HEAD"}}

[jbphet]

I just looked at the build server logs, and it looks like all build requests since the first error report have failed. The last success was at around 2 pm yesterday, here's the log message.

Apr 19 14:42:34 phet-server2.int.colorado.edu build-server[3761273]: info: build for geometric-optics finished successfully

After this, everything has failed. There are additional failures shown in the log for which I didn't receive emails. I'm not sure why not, perhaps they were blocked by my spam filter.

[mattpen]

@jbphet - we think we have a fix in #271 - can you resubmit the latest translation that failed so we can see if that fixes the failure on the build-server?

[zepumph]

@mattpen @samreid and I fixed these in https://github.com/phetsims/perennial/issues/271, sorry for the duplicate.

There were 10 failed builds before we solved this. These are all from translations, @jbphet can you kick off these again to make sure they succeeded. Also, @mattpen, @samreid and I don't know how to kick off these again, so perhaps mentioning some documentation for next time so we don't have to bug you about it.

[jbphet]

I've analyzed the build logs using some grep patterns, and the raw output from that process is included at the bottom of this comment. The bottom line is that, due to some redundancies, there are actually 8 rebuilds that need to happen. They are:

• [x] sim: states-of-matter-basics, locale: ha
• [x] sim: states-of-matter, locale: ha
• [x] sim: projectile-motion, locale: ha
• [x] sim: pendulum-lab, locale: ha
• [x] sim: geometric-optics, locale: de
• [x] sim: geometric-optics, locale: it
• [x] sim: friction, locale: ha
• [x] sim: blackbody-spectrum, locale: ha

Extracted failed build log entries (the raw output):

``` Apr 20 03:32:57 phet-server2.int.colorado.edu build-server[3761273]: simName:"states-of-matter-basics" Apr 20 03:32:57 phet-server2.int.colorado.edu build-server[3761273]: locales:["ha"] Apr 20 03:36:48 phet-server2.int.colorado.edu build-server[3761273]: simName:"states-of-matter" Apr 20 03:36:48 phet-server2.int.colorado.edu build-server[3761273]: locales:["ha"] Apr 20 03:43:36 phet-server2.int.colorado.edu build-server[3761273]: simName:"states-of-matter" Apr 20 03:43:36 phet-server2.int.colorado.edu build-server[3761273]: locales:["ha"] Apr 20 03:45:00 phet-server2.int.colorado.edu build-server[3761273]: simName:"projectile-motion" Apr 20 03:45:00 phet-server2.int.colorado.edu build-server[3761273]: locales:["ha"] Apr 20 04:01:23 phet-server2.int.colorado.edu build-server[3761273]: simName:"pendulum-lab" Apr 20 04:01:23 phet-server2.int.colorado.edu build-server[3761273]: locales:["ha"] Apr 20 04:14:30 phet-server2.int.colorado.edu build-server[3761273]: simName:"geometric-optics" Apr 20 04:14:30 phet-server2.int.colorado.edu build-server[3761273]: locales:["de"] Apr 20 06:50:26 phet-server2.int.colorado.edu build-server[3761273]: simName:"geometric-optics" Apr 20 06:50:26 phet-server2.int.colorado.edu build-server[3761273]: locales:["de"] Apr 20 07:15:29 phet-server2.int.colorado.edu build-server[3761273]: simName:"geometric-optics" Apr 20 07:15:29 phet-server2.int.colorado.edu build-server[3761273]: locales:["it"] Apr 20 09:52:43 phet-server2.int.colorado.edu build-server[3761273]: simName:"friction" Apr 20 09:52:43 phet-server2.int.colorado.edu build-server[3761273]: locales:["ha"] Apr 20 10:00:27 phet-server2.int.colorado.edu build-server[3761273]: simName:"blackbody-spectrum" Apr 20 10:00:27 phet-server2.int.colorado.edu build-server[3761273]: locales:["ha"] ```

[jbphet]

I have triggered a build for each of the sim-locale pairs above and monitored the build sever log after each. All of them finished successfully. I can see them at the fixed URL where they reside, such as https://phet.colorado.edu/sims/html/states-of-matter-basics/latest/states-of-matter-basics_ha.html, but they are not yet on the web site. I'll check back tomorrow and make sure they appear.

[zepumph]

I have triggered a build for each of the sim-locale pairs above and monitored the build sever log after each.

How did you do this?

[mattpen]

@zepumph - I think the docs are here: https://github.com/phetsims/rosetta/blob/master/doc/admin-guide.md#triggering-a-rebuild-without-being-credited. I haven't tried this myself

[zepumph]

Thanks!

[jbphet]

I spot checked most of the translations to verify that they are now up on the web site, and the are, so I think we're good. Closing.