testflows / TestFlows-GitHub-Hetzner-Runners

Autoscaling Self-Hosted GitHub Actions Runners on Hetzner Cloud.
https://testflows.com
Other
37 stars 3 forks source link

The self-hosted runner lost communication with the server #26

Closed Crease29 closed 1 week ago

Crease29 commented 2 months ago

Heya,

I'm using this project for about a week now and it has been working very good for me. Thank you for your work and making this public! Since today I'm getting more and more failed jobs for the following reason:

The self-hosted runner: github-hetzner-runner---cpx21-hel1 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

I haven't touched any of the default values.

Any idea what the issue could be here? It sounds like the runner informed GitHub that it will take over the job but then it shut down?

Logs that I have found for one of these jobs:

00:45:20 create_server  INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
00:45:20 create_server  INFO     šŸ€ Creating server github-hetzner-runner-10802907644-29965869241 with labels {'self-
00:45:22 create_server  INFO     šŸ€ Waiting for server github-hetzner-runner-10802907644-29965869241 to be ready
00:45:22 create_server  INFO        github-hetzner-runner-10802907644-29965869241 initializing
00:45:23 create_server  INFO        github-hetzner-runner-10802907644-29965869241 initializing
00:45:24 create_server  INFO        github-hetzner-runner-10802907644-29965869241 initializing
00:45:25 create_server  INFO        github-hetzner-runner-10802907644-29965869241 initializing
00:45:26 create_server  INFO        github-hetzner-runner-10802907644-29965869241 starting
00:45:28 create_server  INFO        github-hetzner-runner-10802907644-29965869241 starting
00:45:29 create_server  INFO        github-hetzner-runner-10802907644-29965869241 starting
00:45:30 create_server  INFO        github-hetzner-runner-10802907644-29965869241 starting
00:45:31 create_server  INFO        github-hetzner-runner-10802907644-29965869241 running
00:45:32 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...0
00:45:35 scale_down     INFO     šŸ€ Found new potential zombie server github-hetzner-runner-10802907644-29965869241
00:45:40 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...1
00:45:45 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...2
00:45:45 ssh            INFO        > github-hetzner-runner-10802907644-29965869241
                                 runner-10802907644-29965869241-cpx21-hel1 --runnergroup Default --labels self-hosted,type-
00:46:05 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:46:21 scale_down     INFO     šŸ€ Found new unused runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 runner-10802907644-29965869241-cpx21-hel1
00:49:28 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h4m
00:49:29 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:49:30 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:49:30 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:49:31 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:49:38 recycle_server INFO     šŸ€ Waiting for server github-hetzner-runner-10802907644-29965869241 to be ready
00:49:38 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 off
00:49:39 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:49:40 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:49:42 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:49:43 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 running
00:49:43 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...0
                                 runner-10802907644-29965869241-cpx21-hel1
00:49:46 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h49m
00:49:47 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:49:51 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...1
00:49:57 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...2
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:49:58 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:49:58 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:49:59 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
                                 runner-10802907644-29965869241-cpx21-hel1
00:50:04 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h49m
00:50:04 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:50:06 recycle_server INFO     šŸ€ Waiting for server github-hetzner-runner-10802907644-29965869241 to be ready
00:50:06 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 off
00:50:07 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:50:09 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:50:10 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 running
00:50:11 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...0
00:50:19 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...1
00:50:19 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...3
                                 runner-10802907644-29965869241-cpx21-hel1
00:50:20 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h49m
00:50:21 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:50:24 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...4
00:50:24 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...2
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:50:26 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:50:26 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:50:26 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:50:37 scale_down     INFO     šŸ€ Found new powered off server github-hetzner-runner-10802907644-29965869241
                                 runner-10802907644-29965869241-cpx21-hel1
00:50:37 scale_down     INFO     šŸ€ Try deleting unused runner server github-hetzner-runner-10802907644-29965869241 used
00:50:37 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:50:43 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:50:43 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:50:43 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:50:49 recycle_server INFO     šŸ€ Waiting for server github-hetzner-runner-10802907644-29965869241 to be ready
00:50:49 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 off
00:50:50 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:50:52 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
                                 runner-10802907644-29965869241-cpx21-hel1
00:50:53 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:50:53 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h34m
00:50:53 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:50:54 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:50:55 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 running
00:50:55 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...0
00:51:04 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...1
00:51:09 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...2
                                 runner-10802907644-29965869241-cpx21-hel1
00:51:09 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h34m
00:51:09 ssh            INFO        > github-hetzner-runner-10802907644-29965869241
00:51:10 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:51:11 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:51:11 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:51:11 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:51:25 scale_down     INFO     šŸ€ Found new powered off server github-hetzner-runner-10802907644-29965869241
                                 runner-10802907644-29965869241-cpx21-hel1
00:51:25 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h35m
00:51:26 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:51:27 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:51:27 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:51:28 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:51:33 recycle_server INFO     šŸ€ Waiting for server github-hetzner-runner-10802907644-29965869241 to be ready
00:51:33 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 off
00:51:34 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:51:36 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:51:37 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:51:38 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:51:39 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 running
00:51:40 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...0
                                 runner-10802907644-29965869241-cpx21-hel1
00:51:42 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h35m
00:51:43 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:51:46 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...1
00:51:53 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...2
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:51:56 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:51:56 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:51:56 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
                                 runner-10802907644-29965869241-cpx21-hel1
00:51:59 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h35m
00:51:59 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:52:02 recycle_server INFO     šŸ€ Waiting for server github-hetzner-runner-10802907644-29965869241 to be ready
00:52:02 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 off
00:52:04 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:52:05 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:52:06 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:52:07 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:52:08 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 running
00:52:09 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...0
                                 runner-10802907644-29965869241-cpx21-hel1
00:52:15 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h35m
00:52:16 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:52:16 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...3
00:52:16 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...1
00:52:21 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...4
00:52:21 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...2
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:52:24 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:52:24 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:52:25 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:52:30 recycle_server INFO     šŸ€ Waiting for server github-hetzner-runner-10802907644-29965869241 to be ready
00:52:30 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 off
                                 runner-10802907644-29965869241-cpx21-hel1
00:52:31 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h36m
00:52:31 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:52:32 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:52:33 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:52:34 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:52:35 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:52:36 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 running
00:52:36 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...0
00:52:38 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...3
00:52:38 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...5
00:52:44 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...5
00:52:44 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...3
00:52:44 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...1
                                 runner-10802907644-29965869241-cpx21-hel1
00:52:48 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h36m
00:52:48 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:52:49 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...2
00:52:49 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...4
00:52:49 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...6
00:52:50 ssh            INFO        > github-hetzner-runner-10802907644-29965869241
00:52:50 ssh            INFO        > github-hetzner-runner-10802907644-29965869241
00:52:50 ssh            INFO        > github-hetzner-runner-10802907644-29965869241
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:52:52 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:52:52 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 runner-10802907644-29965869241
00:52:53 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:53:04 scale_down     INFO     šŸ€ Found new powered off server github-hetzner-runner-10802907644-29965869241
                                 runner-10802907644-29965869241-cpx21-hel1
00:53:04 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h36m
00:53:05 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:53:10 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:53:10 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:53:10 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:53:17 recycle_server INFO     šŸ€ Waiting for server github-hetzner-runner-10802907644-29965869241 to be ready
00:53:17 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 off
00:53:19 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:53:20 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
                                 runner-10802907644-29965869241-cpx21-hel1
00:53:21 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h37m
00:53:21 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:53:21 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:53:22 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:53:24 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 running
00:53:24 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...0
00:53:30 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...1
00:53:36 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...2
                                 runner-10802907644-29965869241-cpx21-hel1
00:53:37 ssh            INFO        > github-hetzner-runner-10802907644-29965869241
00:53:37 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h37m
00:53:37 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:53:39 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:53:39 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:53:40 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:53:53 scale_down     INFO     šŸ€ Found new powered off server github-hetzner-runner-10802907644-29965869241
                                 runner-10802907644-29965869241-cpx21-hel1
00:53:53 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h37m
00:53:55 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:53:55 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:53:55 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:53:56 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:54:03 recycle_server INFO     šŸ€ Waiting for server github-hetzner-runner-10802907644-29965869241 to be ready
00:54:03 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 off
00:54:04 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:54:06 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:54:07 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 running
00:54:07 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...0
                                 runner-10802907644-29965869241-cpx21-hel1
00:54:10 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h37m
00:54:11 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:54:15 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...1
00:54:21 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...2
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:54:23 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:54:23 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:54:24 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
                                 runner-10802907644-29965869241-cpx21-hel1
00:54:27 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h38m
00:54:27 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:54:29 recycle_server INFO     šŸ€ Waiting for server github-hetzner-runner-10802907644-29965869241 to be ready
00:54:29 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 off
00:54:30 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:54:32 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:54:33 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:54:34 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:54:35 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 running
00:54:35 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...0
00:54:42 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...1
00:54:42 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...3
                                 runner-10802907644-29965869241-cpx21-hel1
00:54:43 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h38m
00:54:44 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:54:47 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...2
00:54:47 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...4
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:54:50 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:54:50 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:54:51 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:55:00 scale_down     INFO     šŸ€ Found new powered off server github-hetzner-runner-10802907644-29965869241
                                 runner-10802907644-29965869241-cpx21-hel1
00:55:00 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h38m
00:55:01 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:55:07 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:55:07 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:55:07 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:55:14 recycle_server INFO     šŸ€ Waiting for server github-hetzner-runner-10802907644-29965869241 to be ready
00:55:14 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 off
00:55:15 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
                                 runner-10802907644-29965869241-cpx21-hel1
00:55:17 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h38m
00:55:17 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:55:17 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:55:18 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:55:19 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 running
00:55:19 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...0
00:55:27 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...5
00:55:27 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...3
00:55:27 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...1
00:55:32 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...4
00:55:32 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...2
00:55:32 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...6
                                 runner-10802907644-29965869241-cpx21-hel1
00:55:33 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h39m
00:55:33 ssh            INFO        > github-hetzner-runner-10802907644-29965869241
00:55:33 ssh            INFO        > github-hetzner-runner-10802907644-29965869241
00:55:33 ssh            INFO        > github-hetzner-runner-10802907644-29965869241
00:55:33 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:55:35 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:55:35 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 hetzner-runner-10802907644-29965869241
00:55:36 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:55:49 scale_down     INFO     šŸ€ Found new powered off server github-hetzner-runner-10802907644-29965869241
                                 runner-10802907644-29965869241-cpx21-hel1
00:55:49 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h39m
00:55:50 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 scraper/actions/jobs/29965869241", id=29965869241)
                                 data-scraper/actions/jobs/29965869241", id=29965869241)
00:55:51 scale_up       INFO     šŸ€ Waiting to finish creating server github-hetzner-runner-10802907644-29965869241
00:55:51 recycle_server INFO     šŸ€ Validating server github-hetzner-runner-10802907644-29965869241 labels
                                 runner-10802907644-29965869241
00:55:52 recycle_server INFO     šŸ€ Rebuilding recycled server github-hetzner-runner-10802907644-29965869241 image
00:55:58 recycle_server INFO     šŸ€ Waiting for server github-hetzner-runner-10802907644-29965869241 to be ready
00:55:58 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 off
00:55:59 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:56:00 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:56:01 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:56:02 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 starting
00:56:04 recycle_server INFO        github-hetzner-runner-10802907644-29965869241 running
00:56:04 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...0
00:56:06 scale_down     INFO     šŸ€ Found new potential zombie server github-hetzner-runner-10802907644-29965869241
00:56:06 scale_down     INFO     šŸ€ Forgetting about unused runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
00:56:12 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...1
00:56:17 server_setup   INFO     šŸ€ Trying to connect to github-hetzner-runner-10802907644-29965869241@XX.XX.XX.XX...2
00:56:18 ssh            INFO        > github-hetzner-runner-10802907644-29965869241
                                 runner-10802907644-29965869241-cpx21-hel1 --runnergroup Default --labels self-hosted,type-
00:57:08 scale_down     INFO     šŸ€ Found new unused runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 runner-10802907644-29965869241-cpx21-hel1
01:00:14 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h43m
01:00:15 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 runner-10802907644-29965869241-cpx21-hel1
01:00:30 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 runner-10802907644-29965869241-cpx21-hel1
01:00:46 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
                                 runner-10802907644-29965869241-cpx21-hel1
01:01:02 scale_down     INFO     šŸ€ Removing self-hosted runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
01:01:18 scale_down     INFO     šŸ€ Forgetting about unused runner github-hetzner-runner-10802907644-29965869241-cpx21-hel1
vzakaznikov commented 2 months ago

Most likely your runner server type is too small and does not have enought resources? Can you try using a bigger server type? For example, try using cpx31 instead of cpx21.

vzakaznikov commented 2 months ago

The runner will power itself off after GitHub Actions runner process (run.sh) exits https://github.com/testflows/TestFlows-GitHub-Hetzner-Runners/blob/main/testflows/github/hetzner/runners/scripts/startup-x64.sh#L13.

vzakaznikov commented 2 months ago

However, some messages look strange. We need to check why the used time jumped from 0d0h4m to 0d0h49m while it does not match the timestamps of the log.

00:49:28 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h4m
...
00:49:46 scale_down     INFO     šŸ€ Marking unused runner server github-hetzner-runner-10802907644-29965869241 used 0d0h49m
Crease29 commented 2 months ago

Most likely your runner server type is too small and does not have enought resources? Can you try using a bigger server type?

Hmm, I kinda doubt it. All they're doing is installing some yarn packages and doing a few HTTP requests, and writing something to a JSON file. Nothing very resource intensive. I took a look at the graphs during build time and they were all just fine.

vzakaznikov commented 2 months ago

Ok, thanks for checking. Then we might have some race condition. I will check how used time is being calculated.

Crease29 commented 2 months ago

Ok, thanks for checking. Then we might have some race condition. I will check how used time is being calculated.

Thank you, if I can provide you with any more information that could help to identify the root cause, please let me know.

vzakaznikov commented 2 months ago

@Crease29, thanks. It would be helpful if this occurs again, to see the full log including specific job id that misbehaved. Of course, the log should be checked to make sure no private data is revealed.

vzakaznikov commented 2 months ago

It would be nice to see that run.sh actually started correctly.

Crease29 commented 2 months ago

@vzakaznikov I will try my best to catch it. Just to be sure, you mean the log output from github-hetzner-runners cloud log, right?

vzakaznikov commented 2 months ago

@vzakaznikov I will try my best to catch it. Just to be sure, you mean the log output from github-hetzner-runners cloud log, right?

Yes, if possible. That would be very useful.

Crease29 commented 2 months ago

It has been pretty stable today, so I wasn't able to gather any further logs so far.

vzakaznikov commented 2 months ago

Hi @Crease29, any more occurances when you've run into the same issue?

Crease29 commented 2 months ago

Every now and then but not as often anymore as when I reported the issue. Iā€™m currently on holiday so Iā€™m not much at my pc but when Iā€™m back and it happens again I will comment here.

vzakaznikov commented 2 months ago

Thanks @Crease29 for the update. One suggestion I can make is to try to increase the following timeouts:

Looking again at your log messsages, looks like the runner gets created but then it is being marked as "unused" and powered off to be recycled. This can happen either if for some reason GitHub API is not updating runner status https://github.com/testflows/TestFlows-GitHub-Hetzner-Runners/blob/main/testflows/github/hetzner/runners/scale_down.py#L347 or maybe each scale down loop iteration is slow and takes more than max-unused-runner-time.

I would try to set these timeouts to something like 600 and check if this helps.

vzakaznikov commented 2 months ago

@Crease29, could you try https://pypi.org/project/testflows.github.hetzner.runners/1.7.240926.1135125/?

vzakaznikov commented 1 week ago

I will close this issue for now. @Crease29, we can re-open if you have more information.