Build failure: `Error: rsync exited with code 11`

[zepumph]

Build failure: Error: Build aborted, Error: rsync exited with code 11. 
Sim = quadrilateral 
Version = 1.0.1 
Brands = phet 
Locales = da 
Shas = {"comment":"# quadrilateral 1.0.1 Sat Jun 03 2023 03:07:46 GMT-0600 (Mountain Daylight Time)","assert":
{"sha":"c13a8c7741d95fbd8e37619c66ebcca8686dffe7","branch":"HEAD"},"axon":{"sha":"e0968f7c0cbb8cc6a538afb4a5e71f988912599b","branch":"HEAD"},"brand":{"sha":"d2b54ef4d29edc31e64adb2f9488803811cad03a","branch":"HEAD"},"chipper":{"sha":"83b6dcc1aa7b210f2fe59b85917d75ffa22f6649","branch":"HEAD"},"dot":{"sha":"e0acfaf31c8847d2e95dbe116d270c22fadccad2","branch":"HEAD"},"joist":{"sha":"63178766e18032460eded6bfc522a59ee99030b6","branch":"HEAD"},"kite":{"sha":"1aeb2c03719b4445ad031ba112a419195fd072c9","branch":"HEAD"},"perennial-alias":{"sha":"faff56fb552f23c61d68fc440e97b781aafd1362","branch":"HEAD"},"phet-core":{"sha":"3846be1447221ccc8b627c8c9e4126593b93e076","branch":"HEAD"},"phet-io":{"sha":"26f080f275b3272715160f7add4048385ee33316","branch":"HEAD"},"phet-io-sim-specific":{"sha":"4d80483f15480d3b35564159ac09dff2f99de8d5","branch":"HEAD"},"phet-io-wrappers":{"sha":"91c730aa306d6c441b8f20e06ce921f39357f874","branch":"HEAD"},"phetcommon":{"sha":"1017d4f9b97256302f796146da155742e4eb5c3c","branch":"HEAD"},"phetmarks":{"sha":"c6c4e824e4bf13d0e467069c49381461787db16f","branch":"HEAD"},"quadrilateral":{"sha":"ad52addca69d2e4ae9e856471df2364f87903074","branch":"HEAD"},"query-string-machine":{"sha":"ff7b4489cd1ab78b08f1e3a38e89c6d89abcd56a","branch":"HEAD"},"scenery":{"sha":"4b6e0a2cf22d2a08ce22493a7bce68d1e3452f3b","branch":"HEAD"},"scenery-phet":{"sha":"8bb2c7dbc03c484b8fb3ea190fb3126117b4abfc","branch":"HEAD"},"sherpa":{"sha":"efff295ef6ba7c442cd868fb71b6d51d560104b1","branch":"HEAD"},"studio":{"sha":"2c31c400423560749216231fabd87f311b5263e9","branch":"HEAD"},"sun":{"sha":"dc743285f7fd10e85d475b26d030142ec565dcb4","branch":"HEAD"},"tambo":{"sha":"4e4aab289995d7f6e3ddb0f77116618a6aa33b43","branch":"HEAD"},"tandem":{"sha":"c55de9187e936d5cdfeb4aa26be7879417019ad3","branch":"HEAD"},"tangible":{"sha":"a5a119ddd62d853ba412463a420b9b17c25e95ce","branch":"HEAD"},"twixt":{"sha":"408daa991138eeaacd7dcd0c144ebace9b6b36ad","branch":"HEAD"},"utterance-queue":{"sha":"5bc222ddf20559f18eb38f51a9e16b9af1a29106","branch":"HEAD"}}

[zepumph]

@jbphet, @chrisklus and I were able to kick off another build for this, and it was successful.

Likely this was some sort of fluke, but we were looking at the cron jobs and see that the peta backups start at 3am. @mattpen do you think this could be related?

################################################
# PETA BACKUPS
################################################

0 1 * * * /data/share/phet/smithers/peta-backup/phet-server/peta-daily-phet-server-backup.sh >> /data/share/phet/peta-logs/daily-stdout 2>> /data/share/phet/peta-logs/daily-stderr
0 3 * * 0 /data/share/phet/smithers/peta-backup/phet-server/peta-weekly-phet-server-backup.sh >> /data/share/phet/peta-logs/weekly-stdout 2>> /data/share/phet/peta-logs/weekly-stderr

[mattpen]

We had another problem like this today, at 11:37am. So I think this means it is unlikely that there is a conflict with the peta backups or other nightly cron jobs.

Here is the full error for the failed build today:

Jun 16 11:37:30 phet-server2.int.colorado.edu build-server[1098237]: debug: rsync: close failed on "/data/web/static/phetsims/sims/html/energy-forms-and-changes/1.4.13/.energy-forms-and-changes_all_debug.
html.Ipiw67": Permission denied (13)
Jun 16 11:37:30 phet-server2.int.colorado.edu build-server[1098237]: rsync error: error in file IO (code 11) at receiver.c(856) [receiver=3.1.3]
Jun 16 11:37:30 phet-server2.int.colorado.edu build-server[1098237]: debug: 11
Jun 16 11:37:30 phet-server2.int.colorado.edu build-server[1098237]: debug: rsync -razpO --no-perms --exclude=.rsync-filter ../release-branches/energy-forms-and-changes-1.4/energy-forms-and-changes/build/
phet/ /data/web/static/phetsims/sims/html/energy-forms-and-changes/1.4.13/

The file name /.energy-forms-and-changes_all_debug.html.Ipiw67 looks like a temp file, so there are probably some race conditions problems going on here. Not really sure what this could be yet.

[mattpen]

I rebuilt this with rosetta's trigger build and there was no error. Scratching my head on this one for now.

[mattpen]

This occurred again today for a greenhouse-effect, sw translation. A trigger build in rosetta succeeded with no other intervention needed. I wonder if there is a missing await in the code somewhere.