Closed TheMetalCode closed 10 months ago
In case this helps anybody else: given the changes in #21536 to force an error when altool
exits prematurely, I'm able to implement an automatic retry in my CI configuration, which so far has always succeeded.
export DELIVER_ALTOOL_ADDITIONAL_UPLOAD_PARAMETERS="--verbose"
EXIT_CODE=0
bundle exec fastlane publish_testflight | tee build_logs/testflight_upload_output.log || EXIT_CODE=$?
ALTOOL_RETRY=$(grep "The call to the altool completed with a non-zero exit status: -1. This indicates a failure." build_logs/testflight_upload_output.log)
if [[ -n "$ALTOOL_RETRY" ]]; then
EXIT_CODE=0
bundle exec fastlane publish_testflight | tee build_logs/testflight_upload_output-retry.log || EXIT_CODE=$?
fi
exit $EXIT_CODE
Addendum: here's what the failure looks like when --verbose
flag is added to altool
command via DELIVER_ALTOOL_ADDITIONAL_UPLOAD_PARAMETERS
:
[14:39:12]: [32m-------------------[0m [14:39:12]: [32m--- Step: pilot ---[0m [14:39:12]: [32m-------------------[0m [14:39:12]: Creating authorization token for App Store Connect API [14:39:12]: [32mReady to upload new build to TestFlight (App: REDACTED)...[0m [14:39:15]: Going to upload updated app to App Store Connect [14:39:15]: [32mThis might take a few minutes. Please don't interrupt the script.[0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.521 DEBUG: [ContentDelivery.Uploader] COMPLETED - PART 4 - 064d7cd6-0f87-41d3-b5ab-1c9be3c7f14c.ipa - eTag: "1CB469EC474939C3011FC7457809463F" [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.522 DEBUG: [ContentDelivery.Uploader] Removed temporary part file '/private/var/folders/6j/z_jm_fbd1rb44rx_9513tbk40000gn/T/com.apple.cds.vbtx/com.apple.cds_795CE584-24D6-4C25-974B-5FEB5E4E5AB5/Part-4.tmp'. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.523 DEBUG: [ContentDelivery.Uploader] Saving uploader state (CDUploaderStateCreateAssetChunksForUpload) for identifier 'com.apple.cds_795CE584-24D6-4C25-974B-5FEB5E4E5AB5'. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.527 DEBUG: [ContentDelivery.Uploader] There are 30 parts remaining to upload. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.528 DEBUG: [ContentDelivery.Uploader] Uploading part 5 COMPLETE. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.529 DEBUG: [ContentDelivery.Uploader] COMPLETED - PART 5 - 064d7cd6-0f87-41d3-b5ab-1c9be3c7f14c.ipa - eTag: "C04838C4F7F8938002CC1F499C652E69" [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.530 DEBUG: [ContentDelivery.Uploader] Removed temporary part file '/private/var/folders/6j/z_jm_fbd1rb44rx_9513tbk40000gn/T/com.apple.cds.vbtx/com.apple.cds_795CE584-24D6-4C25-974B-5FEB5E4E5AB5/Part-5.tmp'. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.531 DEBUG: [ContentDelivery.Uploader] Adding upload task 23 for part 18. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.532 DEBUG: [ContentDelivery.Uploader] Saving uploader state (CDUploaderStateCreateAssetChunksForUpload) for identifier 'com.apple.cds_795CE584-24D6-4C25-974B-5FEB5E4E5AB5'. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.533 DEBUG: [ContentDelivery.Uploader] Saving uploader state (CDUploaderStateCreateAssetChunksForUpload) for identifier 'com.apple.cds_795CE584-24D6-4C25-974B-5FEB5E4E5AB5'. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.544 DEBUG: [ContentDelivery.Uploader] There are 29 parts remaining to upload. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.548 DEBUG: [ContentDelivery.Uploader] PROGRESS - PART 10 (5242880) - '064d7cd6-0f87-41d3-b5ab-1c9be3c7f14c.ipa' 100.00% (5242880/5242880) [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.551 DEBUG: [ContentDelivery.Uploader] PROGRESS - PART 11 (5242880) - '064d7cd6-0f87-41d3-b5ab-1c9be3c7f14c.ipa' 100.00% (5242880/5242880) [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.552 DEBUG: [ContentDelivery.Uploader] Wrote part 19 to a temp file '/private/var/folders/6j/z_jm_fbd1rb44rx_9513tbk40000gn/T/com.apple.cds.vbtx/com.apple.cds_795CE584-24D6-4C25-974B-5FEB5E4E5AB5/Part-19.tmp'. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.563 DEBUG: [ContentDelivery.Uploader] Adding upload task 24 for part 19. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.564 DEBUG: [ContentDelivery.Uploader] Uploading part 6 COMPLETE. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.565 DEBUG: [ContentDelivery.Uploader] COMPLETED - PART 6 - 064d7cd6-0f87-41d3-b5ab-1c9be3c7f14c.ipa - eTag: "DCA3EDF7341A8534E582AAB981C76952" [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.566 DEBUG: [ContentDelivery.Uploader] Saving uploader state (CDUploaderStateCreateAssetChunksForUpload) for identifier 'com.apple.cds_795CE584-24D6-4C25-974B-5FEB5E4E5AB5'. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.567 DEBUG: [ContentDelivery.Uploader] Removed temporary part file '/private/var/folders/6j/z_jm_fbd1rb44rx_9513tbk40000gn/T/com.apple.cds.vbtx/com.apple.cds_795CE584-24D6-4C25-974B-5FEB5E4E5AB5/Part-6.tmp'. [0m [14:40:38]: [33m[altool] 2023-09-27 14:40:38.568 DEBUG: [ContentDelivery.Uploader] Saving uploader state (CDUploaderStateCreateAssetChunksForUpload) for identifier 'com.apple.cds_795CE584-24D6-4C25-974B-5FEB5E4E5AB5'. [0m [14:40:38]: Application Loader output above ^ [14:40:38]: [31mThe call to the altool completed with a non-zero exit status: -1. This indicates a failure.[0m [14:40:38]: [31mCould not download/upload from App Store Connect![0m
It appears to...just quit, with no obvious error messaging, in the middle of the upload process. π
This makes me think about some kind of timeout threshold being exceeded on the command, but it's not obvious to me how to configure any sort of timeout via Fastlane.Pty
so I'm not quite sure how to test this theory.
I notice there is a hardcoded value of 100000
being passed as an upload speed throttle to the altool
upload command: https://github.com/fastlane/fastlane/blob/master/fastlane_core/lib/fastlane_core/itunes_transporter.rb#L288
I'm not sure what removing or altering that arg would do, but 100000 Kbps / 100 Mbps as a throttle doesn't seem like it should seriously impede the upload progress of a (in my case) 178 MB .ipa.
In case this helps anybody else: given the changes in #21536 to force an error when
altool
exits prematurely, I'm able to implement an automatic retry in my CI configuration, which so far has always succeeded.export DELIVER_ALTOOL_ADDITIONAL_UPLOAD_PARAMETERS="--verbose" EXIT_CODE=0 bundle exec fastlane publish_testflight | tee build_logs/testflight_upload_output.log || EXIT_CODE=$? ALTOOL_RETRY=$(grep "The call to the altool completed with a non-zero exit status: -1. This indicates a failure." build_logs/testflight_upload_output.log) if [[ -n "$ALTOOL_RETRY" ]]; then EXIT_CODE=0 bundle exec fastlane publish_testflight | tee build_logs/testflight_upload_output-retry.log || EXIT_CODE=$? fi exit $EXIT_CODE
@TheMetalCode Unfortunately, it doesn't help me. After the retry I still get the same error " The call to the altool completed with a non-zero exit status: -1. This indicates a failure" .
Hello, I have the same problem on 2.216.0.
In case this helps anybody else: given the changes in #21536 to force an error when
altool
exits prematurely, I'm able to implement an automatic retry in my CI configuration, which so far has always succeeded.export DELIVER_ALTOOL_ADDITIONAL_UPLOAD_PARAMETERS="--verbose" EXIT_CODE=0 bundle exec fastlane publish_testflight | tee build_logs/testflight_upload_output.log || EXIT_CODE=$? ALTOOL_RETRY=$(grep "The call to the altool completed with a non-zero exit status: -1. This indicates a failure." build_logs/testflight_upload_output.log) if [[ -n "$ALTOOL_RETRY" ]]; then EXIT_CODE=0 bundle exec fastlane publish_testflight | tee build_logs/testflight_upload_output-retry.log || EXIT_CODE=$? fi exit $EXIT_CODE
@TheMetalCode Unfortunately, it doesn't help me. After the retry I still get the same error " The call to the altool completed with a non-zero exit status: -1. This indicates a failure" .
This occurred on one of my builds this morning where even with an automatic retry it still failed with two -1
exit codes. I was able to rerun the CI job itself and it succeeded then. So whatever is going on with this altool failure, it does seem to eventually succeed if you retry enough times.
In case this helps anybody else: given the changes in #21536 to force an error when
altool
exits prematurely, I'm able to implement an automatic retry in my CI configuration, which so far has always succeeded.export DELIVER_ALTOOL_ADDITIONAL_UPLOAD_PARAMETERS="--verbose" EXIT_CODE=0 bundle exec fastlane publish_testflight | tee build_logs/testflight_upload_output.log || EXIT_CODE=$? ALTOOL_RETRY=$(grep "The call to the altool completed with a non-zero exit status: -1. This indicates a failure." build_logs/testflight_upload_output.log) if [[ -n "$ALTOOL_RETRY" ]]; then EXIT_CODE=0 bundle exec fastlane publish_testflight | tee build_logs/testflight_upload_output-retry.log || EXIT_CODE=$? fi exit $EXIT_CODE
@TheMetalCode Unfortunately, it doesn't help me. After the retry I still get the same error " The call to the altool completed with a non-zero exit status: -1. This indicates a failure" .
This occurred on one of my builds this morning where even with an automatic retry it still failed with two
-1
exit codes. I was able to rerun the CI job itself and it succeeded then. So whatever is going on with this altool failure, it does seem to eventually succeed if you retry enough times.
Seems that it somehow depends on ipa size. The error occurs from time to time for ipa ~3gb or so but any amount of retries doesn't help when ipa is 10GB+
I've started seeing this on IPA just ~180MB. Seems to have started either last Friday (13th π) and/or after moving to Xcode 15. Anyone have any ideas on what is causing this? Because it sounds like the fix here would mean it still errors, but just doesn't crash. π€
Update: while I don't see anything in the release notes that would implicate this issue, I've stopped seeing altool uploads crash since upgrading to Xcode 15.0.1.
https://github.com/fastlane/fastlane/pull/21618 includes enhancements to the underlying FastlanePty
code and should ensure that a nil
exit status is never returned.
https://github.com/fastlane/fastlane/pull/21536 has been adjusted as a result, but still implements a specific error message when an altool
crash is suspected.
We've seen it a few times in Xcode 15.0.1 still but it seems to lend to the argument it was a bad response from the server which was fixed around the same time.
Really nice to see these fixes though, thanks for sharing the links. π
New Issue Checklist
Issue Description
I've only noticed this occurring since upgrading to Xcode 15 - it would appear that when
pilot
runsaltool
to upload an app to Testflight, it's possible foraltool
to exit prematurely, without an exit code or stdout to even begin to diagnose what might have gone wrong.Command executed
bundle exec fastlane publish_testflight
- see fastlane env output, this is the action in which I wrap the call topilot
.Complete output when running fastlane, including the stack trace and command used
Environment