codemagic-ci-cd / cli-tools

Various utilities to managing Android and iOS app builds, code signing, and deployment.
https://codemagic.io/start/
GNU General Public License v3.0
235 stars 39 forks source link

Interrupt `altool`'s CFNetworking retry loop #394

Closed priitlatt closed 4 months ago

priitlatt commented 4 months ago

Sometimes app-store-connect publish action gets stuck while binary is being uploaded using altool. From verbose altool logs it can be seen that uploading fails because specified hostname could not be found, and then internally altool just keeps on retrying the operation indefinitely.

Example of altool verbose log output ```log 2024-03-13 14:26:15.227 DEBUG: [ContentDelivery.Uploader] Wrote part 1 to temp file '/var/folders/w2/rrf5p87d1bbfyphxc7jdnyvh0000gn/T/com.apple.cds.vbtx/com.apple.cds_6A4F9402-15C4-49DA-8C2C-57950EFCD52F/Part-1.tmp'. 2024-03-13 14:26:15.227 DEBUG: [ContentDelivery.Uploader] Adding upload task for part 1. 2024-03-13 14:26:15.228 DEBUG: [ContentDelivery.Uploader] Upload tasks created. Waiting for them to finish uploading. 2024-03-13 14:26:15.236 DEBUG: [ContentDelivery.Uploader] LOST 0 bytes for part 1. 2024-03-13 14:26:15.236 DEBUG: [ContentDelivery.Uploader] Adding upload task 3 for part 1. 2024-03-13 14:26:15.236 *** Error: ERROR: [ContentDelivery.Uploader] WILL RETRY PART 1. Failed with error: Error Domain=NSURLErrorDomain Code=-1003 "A server with the specified hostname could not be found." UserInfo={_kCFStreamErrorCodeKey=8, NSUnderlyingError=0x6000039494a0 {Error Domain=kCFErrorDomainCFNetwork Code=-1003 "(null)" UserInfo={_kCFStreamErrorDomainKey=12, _kCFStreamErrorCodeKey=8, _NSURLErrorNWResolutionReportKey=Resolved 0 endpoints in 6ms using unknown from cache, _NSURLErrorNWPathKey=satisfied (Path is satisfied), interface: en0, ipv4, dns}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalUploadTask <89821E5C-11AB-4F69-B1CB-69AAC3D6FBB8>.<2>, _NSURLErrorRelatedURLSessionTaskErrorKey=( "LocalUploadTask <89821E5C-11AB-4F69-B1CB-69AAC3D6FBB8>.<2>" ), NSLocalizedDescription=A server with the specified hostname could not be found., NSErrorFailingURLStringKey=https://northamerica-1.object-storage.apple.com/itmspod12-assets-massilia-200001/PurpleSource221%2Fv4%2F9c%2F70%2F3c%2F9c703cfe-477c-c38d-a274-508c2ee1a2b2%2F7FnMpSoadCNA8WVLq8VxeS_wSsKqcEDWTeNWS3RMilg_U003d-1710339974733?uploadId=a5eba160-e145-11ee-a95f-5a73240ad8a5&Signature=X6I5Qk04%2Ba4RXRIQJW3%2B6MMUuXk%3D&AWSAccessKeyId=MKIAP7F9QNTEY48OTE7F&partNumber=1&Expires=1710944775, NSErrorFailingURLKey=https://northamerica-1.object-storage.apple.com/itmspod12-assets-massilia-200001/PurpleSource221%2Fv4%2F9c%2F70%2F3c%2F9c703cfe-477c-c38d-a274-508c2ee1a2b2%2F7FnMpSoadCNA8WVLq8VxeS_wSsKqcEDWTeNWS3RMilg_U003d-1710339974733?uploadId=a5eba160-e145-11ee-a95f-5a73240ad8a5&Signature=X6I5Qk04%2Ba4RXRIQJW3%2B6MMUuXk%3D&AWSAccessKeyId=MKIAP7F9QNTEY48OTE7F&partNumber=1&Expires=1710944775, _kCFStreamErrorDomainKey=12} 2024-03-13 14:26:15.237 DEBUG: [ContentDelivery.Uploader] Created new upload task (0x13264e8f0) for part 1. 2024-03-13 14:26:15.237 DEBUG: [ContentDelivery.Uploader] Saving uploader state (CDUploaderStateUploadAssetDescription) for identifier 'com.apple.cds_6A4F9402-15C4-49DA-8C2C-57950EFCD52F'. 2024-03-13 14:26:15.239 DEBUG: [ContentDelivery.Uploader] There is one part remaining to upload. 2024-03-13 14:26:15.240 DEBUG: [ContentDelivery.Uploader] LOST 0 bytes for part 1. 2024-03-13 14:26:15.240 DEBUG: [ContentDelivery.Uploader] Adding upload task 4 for part 1. 2024-03-13 14:26:15.240 *** Error: ERROR: [ContentDelivery.Uploader] WILL RETRY PART 1. Failed with error: Error Domain=NSURLErrorDomain Code=-1003 "A server with the specified hostname could not be found." UserInfo={_kCFStreamErrorCodeKey=8, NSUnderlyingError=0x600003949dd0 {Error Domain=kCFErrorDomainCFNetwork Code=-1003 "(null)" UserInfo={_kCFStreamErrorDomainKey=12, _kCFStreamErrorCodeKey=8, _NSURLErrorNWResolutionReportKey=Resolved 0 endpoints in 0ms using unknown from cache, _NSURLErrorNWPathKey=satisfied (Path is satisfied), interface: en0, ipv4, dns}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalUploadTask <88A22F39-3A2A-40A0-8EA2-80DB60C997E3>.<3>, _NSURLErrorRelatedURLSessionTaskErrorKey=( "LocalUploadTask <88A22F39-3A2A-40A0-8EA2-80DB60C997E3>.<3>" ... ```

As verbose log output is not guaranteed we cannot rely on this output to detect whether the upload has gotten stuck or not. Instead add a background system log checker which looks for similar entries for altool process starting from the moment when the subprocess was launched.

In case the logs indicate that altool uploading might be stuck, then the same retry logic is applied as was already introduced for failed uploads in #163.

Updated actions: