Build failures: git pull fails with 'You are not currently on a branch'

[jbphet]

We had three build failures that occurred over the weekend, and all three seemed to be trigged by translations. I've pasted the info from each of the emails into a separate details section below. We should figure out why these happened and whether there is anything we need to do about it.

Build Error Sun 10/17/2021 8:04 AM

``` Build failure: Error: Build aborted, Error: git pull in ../wave-on-a-string failed with exit code 1 stderr: You are not currently on a branch. Please specify which branch you want to merge with. See git-pull(1) for details. git pull Sim = wave-on-a-string Version = 1.1.24 Brands = phet Locales = da Shas = {"comment":"# wave-on-a-string 1.1.24 Thu Sep 23 2021 07:18:38 GMT-0600 (Mountain Daylight Time)","assert":{"sha":"7d27130a4d27da15aa7f8a21bbfb9245288c1b44","branch":"HEAD"},"axon":{"sha":"9457f36aec22935c7b5f41a9d0777255dd5d9884","branch":"HEAD"},"brand":{"sha":"78d99a4f0fefd28d3ccd1f221974a2735539f3fc","branch":"HEAD"},"chipper":{"sha":"b212e1ef5add46a0a16aee25fbc58da4c6a2a432","branch":"HEAD"},"dot":{"sha":"da78031b6f080d20bc8346beaa1d37148188371e","branch":"HEAD"},"joist":{"sha":"c2d7dcbc6844d079d008a4b28d839653d856f2f4","branch":"HEAD"},"kite":{"sha":"10ca8bf2b2712307ff528352de8451c0c8afe093","branch":"HEAD"},"phet-core":{"sha":"de29611a6ac322da53a4302ca65f40ff9713e497","branch":"HEAD"},"phetcommon":{"sha":"bf99b8eed30505a759a3a06928b3ee29c03f92e9","branch":"HEAD"},"phetmarks":{"sha":"3fdfece0fe0262594bb06856833b8b16be25deb8","branch":"HEAD"},"scenery":{"sha":"3eed565828d9b0d4a46b04fb45840dcb2a1d9e59","branch":"HEAD"},"scenery-phet":{"sha":"68b8cb3994b5d8570d75c0027dc60d23c3d74a9c","branch":"HEAD"},"sherpa":{"sha":"b999987ecbc9d23c011e8114f41ea486397c9f1c","branch":"HEAD"},"sun":{"sha":"2b83e994039b2d33845d5e51d7735d144c188d97","branch":"HEAD"},"tandem":{"sha":"2441eec6b422cb20e08682da02ab2bf7bf73250c","branch":"HEAD"},"wave-on-a-string":{"sha":"16fbd15db8faee1d039da3892be672b406a293a0","branch":"HEAD"}} ```

Build Error Sun 10/17/2021 8:12 AM

``` Build failure: Error: Build aborted, Error: git pull in ../wave-interference failed with exit code 1 stderr: You are not currently on a branch. Please specify which branch you want to merge with. See git-pull(1) for details. git pull Sim = wave-interference Version = 2.0.5 Brands = phet Locales = da Shas = {"comment":"# wave-interference 2.0.5 Fri Oct 15 2021 10:57:00 GMT-0600 (Mountain Daylight Time)","assert":{"sha":"201c50a840e082354b013f4ab9e9d45006f98cfe","branch":"HEAD"},"axon":{"sha":"d3bd7132637430f96000140c09c27c04775af0d2","branch":"HEAD"},"brand":{"sha":"40dd405ec950d10e15e957cbda3d97362e925bef","branch":"HEAD"},"chipper":{"sha":"ea9b9a711c6c817d4b5813fff7112dd6e25ec6ff","branch":"HEAD"},"dot":{"sha":"29e6b8aa840d467420c601c918556117dbaf9df2","branch":"HEAD"},"griddle":{"sha":"0ed03b9efcd50cd2b0b015084ae13aaf1c40455f","branch":"HEAD"},"joist":{"sha":"a37fee3315e4ef36ac7c287410608e5c1ad3763e","branch":"HEAD"},"kite":{"sha":"34e7bdb4379b8faab55c728fdd462d4477f77fe7","branch":"HEAD"},"phet-core":{"sha":"c45ad1a746e393037ca1e75dc5b236749d2ca683","branch":"HEAD"},"phet-io":{"sha":"3e2819bcb98e1fe3ef10cc272ea432bb3e091aa9","branch":"HEAD"},"phet-io-wrappers":{"sha":"6d48cabead79a1e232f757c0203cc31ae5a229b3","branch":"HEAD"},"phetcommon":{"sha":"b82ae2142fe0874561941895a4b36ce6f91428b9","branch":"HEAD"},"phetmarks":{"sha":"1dcdf67f09932ff86941f4f765b12dd20d7f18c0","branch":"HEAD"},"query-string-machine":{"sha":"c4c6d494b474aeb14d700a8142da81776f62a2ba","branch":"HEAD"},"scenery":{"sha":"e477a2baeafa9017ce6b5c03c4e45e7fad2836a6","branch":"HEAD"},"scenery-phet":{"sha":"441f7609a5eb69d9def891b3ff3184df1fc5a91f","branch":"HEAD"},"sherpa":{"sha":"4877464e8013ff1459091c854a8c2234c3d0e7d6","branch":"HEAD"},"sun":{"sha":"2b9c6f202e653f23c8590e7a24d5dbc828ed6955","branch":"HEAD"},"tambo":{"sha":"ce84e44a34a43905bd646eaf36ea86e65a3ccfba","branch":"HEAD"},"tandem":{"sha":"efc5178813482458cb63a75b2cedd46d6d8c8ec1","branch":"HEAD"},"twixt":{"sha":"f8dcf79d9eea76c707574aeb02c758930895b35e","branch":"HEAD"},"wave-interference":{"sha":"e9101d479f61c8dc899caeaaac5c0afe4e7f75ae","branch":"HEAD"}} ```

Build Error Sun 10/17/2021 8:17 AM

``` Build failure: Error: Build aborted, Error: git pull in ../unit-rates failed with exit code 1 stderr: You are not currently on a branch. Please specify which branch you want to merge with. See git-pull(1) for details. git pull Sim = unit-rates Version = 1.0.19 Brands = phet Locales = da Shas = {"comment":"# unit-rates 1.0.19 Thu Sep 23 2021 06:53:05 GMT-0600 (Mountain Daylight Time)","assert":{"sha":"a707328cc11048787a807714496af23e0bd25be1","branch":"HEAD"},"axon":{"sha":"f7720d0e57be3884aedaa78bd799d1fac8168711","branch":"HEAD"},"brand":{"sha":"3b689e0de730f0946b356be26a03c8e6b2588014","branch":"HEAD"},"chipper":{"sha":"f8690c71de9d11d8af631e0afcb3e876f574d9b9","branch":"HEAD"},"dot":{"sha":"569939e1f45973234a3819255a88cd2b80006f76","branch":"HEAD"},"joist":{"sha":"647cce0cbb4c51dc7dab84e7a766028bfd145cfd","branch":"HEAD"},"kite":{"sha":"81166ce9735f0bcb11b9e68b74eeed46b6f24da6","branch":"HEAD"},"phet-core":{"sha":"b9b67611be6138642d25b2c67cbc70ff432fd83d","branch":"HEAD"},"phetcommon":{"sha":"6eb40bae1cf1ad457ed2c79456f41dbaef5accf8","branch":"HEAD"},"phetmarks":{"sha":"3a5691551b8be942d4b0534c82e86701882529f9","branch":"HEAD"},"query-string-machine":{"sha":"d8a4ff18a12aff003e4896213c1dfed5a79c1a30","branch":"HEAD"},"scenery":{"sha":"44194294bb5dc036339e97d6be0ad636eefaf156","branch":"HEAD"},"scenery-phet":{"sha":"d757c70356febf448bf73c3cab0730318856a275","branch":"HEAD"},"sherpa":{"sha":"3255de0f1edda165ff360076051a56cd31cd3ea5","branch":"HEAD"},"sun":{"sha":"bffd225eb8be0412385b8c542bcf8edc282fe31d","branch":"HEAD"},"tandem":{"sha":"a668abd5e2f1d6b80de925c9e7567c4468a34f3a","branch":"HEAD"},"twixt":{"sha":"a6bd3909751e280b7fdbe2f0ecd9ec066d152ff5","branch":"HEAD"},"unit-rates":{"sha":"b404be0442570da4c2302bfe47bfc4de08812390","branch":"HEAD"}} ```

One thing that's weird about these is how close together in time they are. I'm not at all sure how that could happen with rosetta, unless three requests just happened to all get queued at the same time, which seems unlikely.

[jbphet]

A few notes that might help:

• There have been successful translations submitted both before and after these errors, so it seems like a transient problem.
• I looked through Rosetta's log, and it looks like the translator, who was submitting sims for the "da" locale (Danish), really did submit these translations within a few minutes of one another. I looked at the commits, and the first one is significant, and the next two are fairly small, so it seems like they submitted one, then just did some touchups on a couple others, so it conceivable that they really did them this close together. I didn't see signs of activity at the same time by other translators.
• I looked through the build server log, and I can see that the correspond build requests were received, and some were queued before the previous one finished. I don't know if that's related to the problem, but it may be a hint.
• The errors in the log seem to indicate a problem with executing git pull in the main simulation repo after the build completed, and the log indicates that shortly before this occurs the grunt checkout-master-all command was run, so perhaps there is some issue with that task getting the main repo back onto the master branch.

[mattpen]

Looking through the code, it looks like this probably failed in deployImages.js line 18/19. We run a git pull on the repo being built directly after running grunt checkout-master-all. This raises 2 questions, (a) why are we running image deploys for translation builds and (b) is grunt checkout-master-all not actually completing before it returns and its process completes? It looks like even though we are using await that it hasn't completed when we evaluate the next line.

(a) should be easy to fix, by putting a guard that skips the deployImages step if the build is for a single locale that is not equal to en. (b) will take some more investigation. That behavior is very unexpected.

[jbphet]

Looks like another failure of the same type occurred yesterday evening. Here is the content of the email:

``` Build failure: Error: Build aborted, Error: git pull in ../wave-interference failed with exit code 1 stderr: You are not currently on a branch. Please specify which branch you want to merge with. See git-pull(1) for details. git pull Sim = wave-interference Version = 2.0.5 Brands = phet Locales = ms Shas = {"comment":"# wave-interference 2.0.5 Thu Oct 21 2021 00:53:46 GMT-0600 (Mountain Daylight Time)","assert":{"sha":"201c50a840e082354b013f4ab9e9d45006f98cfe","branch":"HEAD"},"axon":{"sha":"d3bd7132637430f96000140c09c27c04775af0d2","branch":"HEAD"},"brand":{"sha":"40dd405ec950d10e15e957cbda3d97362e925bef","branch":"HEAD"},"chipper":{"sha":"ea9b9a711c6c817d4b5813fff7112dd6e25ec6ff","branch":"HEAD"},"dot":{"sha":"29e6b8aa840d467420c601c918556117dbaf9df2","branch":"HEAD"},"griddle":{"sha":"0ed03b9efcd50cd2b0b015084ae13aaf1c40455f","branch":"HEAD"},"joist":{"sha":"a37fee3315e4ef36ac7c287410608e5c1ad3763e","branch":"HEAD"},"kite":{"sha":"34e7bdb4379b8faab55c728fdd462d4477f77fe7","branch":"HEAD"},"phet-core":{"sha":"c45ad1a746e393037ca1e75dc5b236749d2ca683","branch":"HEAD"},"phet-io":{"sha":"3e2819bcb98e1fe3ef10cc272ea432bb3e091aa9","branch":"HEAD"},"phet-io-wrappers":{"sha":"6d48cabead79a1e232f757c0203cc31ae5a229b3","branch":"HEAD"},"phetcommon":{"sha":"b82ae2142fe0874561941895a4b36ce6f91428b9","branch":"HEAD"},"phetmarks":{"sha":"1dcdf67f09932ff86941f4f765b12dd20d7f18c0","branch":"HEAD"},"query-string-machine":{"sha":"c4c6d494b474aeb14d700a8142da81776f62a2ba","branch":"HEAD"},"scenery":{"sha":"e477a2baeafa9017ce6b5c03c4e45e7fad2836a6","branch":"HEAD"},"scenery-phet":{"sha":"441f7609a5eb69d9def891b3ff3184df1fc5a91f","branch":"HEAD"},"sherpa":{"sha":"4877464e8013ff1459091c854a8c2234c3d0e7d6","branch":"HEAD"},"sun":{"sha":"2b9c6f202e653f23c8590e7a24d5dbc828ed6955","branch":"HEAD"},"tambo":{"sha":"ce84e44a34a43905bd646eaf36ea86e65a3ccfba","branch":"HEAD"},"tandem":{"sha":"efc5178813482458cb63a75b2cedd46d6d8c8ec1","branch":"HEAD"},"twixt":{"sha":"f8dcf79d9eea76c707574aeb02c758930895b35e","branch":"HEAD"},"wave-interference":{"sha":"e9101d479f61c8dc899caeaaac5c0afe4e7f75ae","branch":"HEAD"}} ```

[mattpen]

EDIT: typed a comment in the wrong issue.

[mattpen]

Reviewed this again today, it definitely looks like there is an error in grunt checkout-master-all or it is completing asyncronously.

The code in question: https://github.com/phetsims/perennial/blob/a17af9f4488f91027374d2e4e8b4e8cabd4415a9/js/build-server/deployImages.js#L19-L20

Output:

Nov 30 09:27:25 phet-server.int.colorado.edu build-server[23489]: debug: stdout: Running "checkout-master-all" task
Nov 30 09:27:25 phet-server.int.colorado.edu build-server[23489]: Done.
Nov 30 09:27:25 phet-server.int.colorado.edu build-server[23489]: info: git pull on wave-interference
Nov 30 09:27:25 phet-server.int.colorado.edu build-server[23489]: debug: Running git pull from ../wave-interference
Nov 30 09:27:25 phet-server.int.colorado.edu build-server[23489]: debug: Command git finished. Output is below.
Nov 30 09:27:25 phet-server.int.colorado.edu build-server[23489]: debug: stderr: You are not currently on a branch. Please specify which
Nov 30 09:27:25 phet-server.int.colorado.edu build-server[23489]: branch you want to merge with. See git-pull(1) for details.
Nov 30 09:27:25 phet-server.int.colorado.edu build-server[23489]: git pull <remote> <branch>
Nov 30 09:27:25 phet-server.int.colorado.edu build-server[23489]: debug: stdout is empty.
Nov 30 09:27:25 phet-server.int.colorado.edu build-server[23489]: error: BUILD ABORTED! Error: git pull in ../wave-interference failed with exit code 1

Immediately after grunt checkout-master-all reports "Done", wave-interference was still not on master and git pull failed. I'm going to change this code to execute some native bash instead to get better guarantees on completion.

[mattpen]

On the positive side, since the only thing that failed is the image deployment these were completely successful and don't need corrections for the sim file or translation credits.

I'll also add a guard to prevent image deployment for translation builds.

[mattpen]

I submitted a translation for chains in yo with the above changes to the build-server, so I'm going to close this issue with tentative optimism.