search

git-for-windows / git-for-windows-automation

A few GitHub workflows and support code to help with Git for Windows' day-to-day tasks
10 stars 9 forks source link

Self-hosted Azure runner failed to create #59

Open dennisameling opened 7 months ago

dennisameling commented 7 months ago

I just kicked off a run to build the ARM64 version of 2.43.0, but it looks like the deployment of the self-hosted Azure runner failed:

Error: ERROR: ***"status":"Failed","error":***"code":"DeploymentFailed","target":"/subscriptions/***/resourceGroups/***/providers/Microsoft.Resources/deployments/deploy-actions-runner-20231121103205908804162","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[***"code":"ResourceDeploymentFailure","target":"/subscriptions/***/resourceGroups/***/providers/Microsoft.Compute/virtualMachines/actions-runner-20231121103205908804162","message":"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.","details":[***"code":"OSProvisioningTimedOut","message":"OS Provisioning for VM 'actions-runner-20231121103205908804162' did not finish in the allotted time. The VM may still finish provisioning successfully. Please check provisioning state later. For details on how to check current provisioning state of Windows VMs, refer to https://aka.ms/WindowsVMLifecycle and Linux VMs, refer to https://aka.ms/LinuxVMLifecycle."***]***]***

I don't have access to these resources, so can't see what exactly failed here. @dscho would you mind checking your VM overview in Azure? šŸ™šŸ¼

dscho commented 7 months ago

Hmm. From a cursory look, it seems that all the resources were provisioned all right. The VM was running. I deleted those resources and kicked off a re-run of the workflow spinning up a new VM. Let's šŸ¤ž that it will work this time.

dscho commented 7 months ago

The pkg is building!

dennisameling commented 7 months ago

Thank you and sorry for bothering! I forgot that we have the "delete azure-self-hosted-runner" workflow that I could've used to manually remove the VM instead of you having to go into the Azure Portal. Great to see it's working again šŸ˜Š

dennisameling commented 7 months ago

Hmm, things seem to be very unstable.

I'll try to replicate this in my own Azure tenant, but have an open ticket with Azure support to enable arm-based runners in my tenant. That'll take a while.

dscho commented 7 months ago

No worries about bothering me, all good. One thing I noticed too late, recently, is that the VMs are created without auto-shutdown. I noticed only after three weeks of the VM running, running through half of my monthly credit...

Do you think you could add an auto shut-down after, say, 6h?

dscho commented 7 months ago

Another idea: we should be able to teach GitForWindowsHelper to detect failed create-self-hosted-runner runs and automatically kick off the delete run, no? And maybe also try re-running the failed run, up to five times?