search

philips-labs / terraform-aws-github-runner

Terraform module for scalable GitHub action runners on AWS
https://philips-labs.github.io/terraform-aws-github-runner/
MIT License
2.62k stars 627 forks source link

Slower npm execution on self hosted windows images. #3938

Open AlexShemeshWix opened 5 months ago

AlexShemeshWix commented 5 months ago

Hi. We use self hosted github runner in AWS cloud to run tests for typescript react apps. We have a problem running npm install command on windows github runners. It runs extremely slow. For example if i execute npm install for my project on github runners provided by github it will complete in 1 min. If i run same workflow on self hosted github runner it will take 5 minutes. Most of the time it will spend extracting packages from tgz files in npm cache. We use Windows Core 2022 image as base for our AMI.Powershell is 7.1 Latest version of github runner. LTS version of node and npm.Ive tried to create ReFS partition and run everything from it. Ive tried different types of volumes mounted from AWS to it. Nothing helped. If i access ec2 instance of one of the runners over Remote Desktop and clone my repository manually and execute npm install in it, it will take under a minute to complete. So its not hardware or software problem. Its something with github runner integration into system. Any advice how to debug it?

AlexShemeshWix commented 5 months ago

Any ideas?

stuartp44 commented 4 months ago

Windows is a pain for this type of issue, couple of things to try. Proxying is a killer, If you are using a private source like Artifactory, make sure you have proxy bypass set as even though it may work, you are typically going out of the network boarders to come back into the network boarders. This is a waste when and can cause performance problems, even though your github runner is sat next to the artifactory server. Its also maybe worth doing some simple stuff and working your way up the networking stack to help resolve your issue. Maybe do some simple pinging first and a traceroute to really spot the differences and rule them out to get to your actual problem. I would look at your env to see if there is anything being set that you dont want and also on the other side what you "need" but isnt being set.

AlexShemeshWix commented 1 month ago

After much debugging we established the cause of the problem. In the packer script, github runner is configured to run as scheduled task. See script here. If we run it as windows service under NETWORK_SERVICE credentials - it installs npm modules much faster.