Failure during graphing quadratics 1.2 deploy

[samreid]

@pixelzoom reported:

My production build of Graphing Quadratics has failed. According to the email received from build-server:

Build failure: Error: Build aborted, Error: grunt --brands=phet,phet-io --repo=graphing-quadratics build-images in ../chipper failed with exit code 3
stdout:
Loading "Gruntfile.js" tasks...ERROR
Error: Cannot find module '../../../perennial-alias/js/common/isRepoTypeScript'
Require stack:
- /data/share/phet/phet-repos/chipper/js/grunt/webpackBuild.js
- /data/share/phet/phet-repos/chipper/js/grunt/buildStandalone.js
- /data/share/phet/phet-repos/chipper/js/grunt/copySupplementalPhetioFiles.js
- /data/share/phet/phet-repos/chipper/js/grunt/buildRunnable.js
- /data/share/phet/phet-repos/chipper/js/grunt/Gruntfile.js
- /data/share/phet/phet-repos/chipper/Gruntfile.js
- /data/share/phet/phet-repos/chipper/node_modules/grunt/lib/grunt/task.js
- /data/share/phet/phet-repos/chipper/node_modules/grunt/lib/grunt.js
- /usr/lib/node_modules/grunt-cli/bin/grunt
Warning: Task "build-images" not found. Use --force to continue.

Aborted due to warnings.
Sim = graphing-quadratics Version = 1.2.0 Brands = phet,phet-io Locales = * Shas = {"comment":"# graphing-quadratics 1.2.0 Wed Sep 08 2021 18:30:22 GMT-0400 (Eastern Daylight Time)","assert":{"sha":"f360fa1bd09a47b3c9e49cd90c8dddd136a13e0c","branch":"HEAD"},"axon":{"sha":"d09f8f50b1a8b97145612bece9e7d91a20e05b28","branch":"HEAD"},"brand":{"sha":"0c723e76a7bad4344cb588c486ae41d0f7fc4995","branch":"HEAD"},"chipper":{"sha":"7aee98a05c3a13c5000e04ed2ea1ddfe84dad715","branch":"HEAD"},"dot":{"sha":"aacdd9db44ef332395aa95b0b134b6032d5b8a59","branch":"HEAD"},"graphing-lines":{"sha":"7bc6a8269b9571a7788bee67edb62642f9f28770","branch":"HEAD"},"graphing-quadratics":{"sha":"826f094b7d9fe5557b2fb3a8677968ae89dfe394","branch":"1.2"},"joist":{"sha":"843feeb1a1c9e32194961faec92e1d15523bbcf6","branch":"HEAD"},"kite":{"sha":"a5a45cba30bc83d0400d83f03c7e4ffe06d75501","branch":"HEAD"},"perennial-alias":{"sha":"695911d0604c2bd18e9031ea4f5b26d6586bd025","branch":"HEAD"},"phet-core":{"sha":"354cc8ce376a142df9f579e2e44e438a07b56605","branch":"HEAD"},"phet-io":{"sha":"5ff1615454c98ae6bf43ccb9149ec17282d66c70","branch":"HEAD"},"phet-io-client-guides":{"sha":"1a588f974609e9665b4bdeb6c839762b7a60a4fd","branch":"HEAD"},"phet-io-wrappers":{"sha":"dd9755228f1eff6bd01448d2c6825f2e1471fcbc","branch":"HEAD"},"phetcommon":{"sha":"702911858bafc8e772a2b9d8747ef599cc2bce5c","branch":"HEAD"},"phetmarks":{"sha":"589cdbe7c499adfc1cc478b11340fc0ac61ff93d","branch":"HEAD"},"query-string-machine":{"sha":"12cfdab188ee5be9a8c1932411ad41da8b175c88","branch":"HEAD"},"scenery":{"sha":"dfc38683f72c688970cd03c26843685db92a3894","branch":"HEAD"},"scenery-phet":{"sha":"7a6b0480e7b81dc39ec48dc369bd951f59df9523","branch":"HEAD"},"sherpa":{"sha":"ec49f43aee29c836c13884be095e2772bc9faaa0","branch":"HEAD"},"studio":{"sha":"6000377b355bd90674414eae4fa4f695dc7fb40b","branch":"HEAD"},"sun":{"sha":"ca4c2fb861e678adc960a84f67b90875eaab3e1c","branch":"HEAD"},"tambo":{"sha":"e4fda961e4983c1f10aceffff14d0c91e0f50c31","branch":"HEAD"},"tandem":{"sha":"dcea61ce3483056a7!
d4750d3c51aa3f6d20365cd","branch":"HEAD"},"twixt":{"sha":"35e83120131ca1265e72d84c822a21bb557e08cb","branch":"HEAD"},"utterance-queue":{"sha":"c5024988addef3818199c1bad305d4b9c6e1e385","branch":"HEAD"}}

We confirmed that the perennial-alias sha is being recorded and it seems reasonable (from a few days ago). We think there is a problem in the image generation phase, since 1.2 seems like it deployed OK, except the image was broken.

@jonathanolson and I looked through the thumbnail generating code on the build server, but it was unclear where the problem lies.

Instead we added a "band-aid" of coping the images from the previous version of graphing quadratics (after checking that MD5 shas were matching up). Now the image is showing OK and launching OK, but we still see errors in the console, some of them trying to load from a hardcoded 127.0.0.1:

We also noticed problems on other sim pages, such as energy-skate-park:

@jonathanolson can you please add anything I missed?

@mattpen can you please take a look?

[jonathanolson]

It seems like somehow the build-images step ran with an out-of-date perennial-alias (but I'm not sure how that would have happened).

Sim pages seem to make broken localhost image requests, which also results in breaking JS (if you go to a sim page with devtools open, and go back-and-forth between "About" and "Teaching Resources", it will error with this:

[jonathanolson]

I copied images from graphing-quadratics 1.1.6 => 1.2.0. It looks like the images are showing up on the sim page now.

[jonathanolson]

I also shut down the build-server and recloned perennial while working on this (shut down 17:05, started back 17:38).

A third issue we'll need to handle is incorrect user permissions (non-phet-admin) in build-server's phet-repos directory. We'll probably want to reclone all repos if possible @mattpen. Thoughts, or should we fix this by adjusting permissions and users?

[pixelzoom]

Graphing Quadratics 1.2.0 was for --brands=phet,phet-io, so this may also impact the PhET-iO website. The deployed links should be:

https://phet.colorado.edu/sims/html/graphing-quadratics/latest/graphing-quadratics_en.html https://phet-io.colorado.edu/sims/graphing-quadratics/1.2

[mattpen]

we still see errors in the console, some of them trying to load from a hardcoded 127.0.0.1

These should be transient, but I opened https://github.com/phetsims/website-meteor/issues/337 to fix them.

---

I'm really confused why deploy-images would be using perennial-alias. The build server should check out chipper master branch and run the code from there. I'll look through the logs and see what happened.

[mattpen]

It looks like the code that prepares to run build-images is still doing an npm prune && npm install in annual, but chipper master now depends on perennial-alias, so this needs to be changed.

[mattpen]

It looks like the rename of annual to perennial-alias in the build-server code was implemented but never deployed. The fixed in 32276f5b5e298e1ebcdc54f522665a63930f51da, but this wasn't ever deployed. As part of the GQ failure, this change was finally deployed (the build-server crashed and automatically restarted itself at master). I think this should have corrected the problem. I'll try a production deploy of chains, but since it doesn't include any images I don't know if this will be an effective test.

[mattpen]

Somehow chains had a dependency on annual in branch 1.34 still, I just changed it to perennial-alias.

[mattpen]

I'm not able to test chains at the moment due to some problems with my npm configuration. However, it looks like the only thing that failed is the images themselves, and @jonathanolson corrected this manually. I don't recommend a redeploy of GQ 1.2 as there is nothing to fix. I believe this problem self-corrected and won't happen again going forward, but I currently can't test to confirm.

[mattpen]

A production of deploy for chains worked successfully. We still have not confirmed that the image deploys are working correctly, but that will need to be tested on a public facing sim.

[pixelzoom]

@mattpen said:

I don't recommend a redeploy of GQ 1.2 as there is nothing to fix.

@mattpen Is that true for both the phet and phet-io brands? I'm trying to determine whether my work on Graphing Quadratics is indeed done, and whether https://github.com/phetsims/graphing-quadratics/issues/137 can be closed.

[mattpen]

@pixelzoom I'm not familiar with what a correct phet-io deploy looks like these days, it would be good to have one of the experts confirm. However, the only thing that should happen after the step that failed is the final cleanup (checking out master in all repos and deleting temporary files), so I'm fairly confident it was successful.

Maybe @samreid or @zepumph could take a quick look?

[samreid]

I clicked through https://phet-io.colorado.edu/sims/graphing-quadratics/1.2/ and it seemed ok.

[amanda-phet]

@jonathanolson said

I copied images from graphing-quadratics 1.1.6 => 1.2.0. It looks like the images are showing up on the sim page now.

I just went to https://phet.colorado.edu/en/simulations/graphing-quadratics and I don't see the alt images. I don't understand the full scope of the problems in this issue, but wanted to clarify that 1.2 is still not looking good to me.

[mattpen]

This is likely because the image deployment failed and was corrected manually. I added the alt images to the database, but it will be a while before they appear due to server side caching (24hours).

[zepumph]

I think the cache has finally lapsed:

Anything else here?

[amanda-phet]

Thanks @mattpen and @zepumph ! I see the alt screenshots now too.

[mattpen]

This problem looks like it is resolved. We can reopen the issue if the problem recurs.