shapehq / tartelet

⚙️💻 A macOS app that makes it a breeze to manage multiple GitHub Actions runners in ephemeral virtual machines on a single host machine. The benefits are that runners can run in parallel, and each job runs in an isolated environment.
MIT License
520 stars 19 forks source link

Trouble connecting to VM on start #94

Open ryanhanks-bestow opened 1 month ago

ryanhanks-bestow commented 1 month ago

Your Question

I read through the recommended steps for creating a new VM from scratch, but didn't feel I needed that level of customization in my image over what tart clone ghcr.io/cirruslabs/macos-sonoma-xcode:latest runner would offer.

Is there a reason using this as a starting point would cause me issues? I supply tartelet with the proper username / password for the vm but get stuck in a loop of failed attempts to connect to the machine:

15:17:01.227 INFO: Start virtual machine named runner-1
15:17:14.189 INFO: The GitHub app has not been installed. Please install it from the developer settings.
15:17:14.190 INFO: Could not connect to the virtual machine over SSH, so the virtual machine will be shut down.
15:17:14.193 INFO: Virtual machine named runner-1 stopped with message: Failed establishing SSH connection
15:17:14.525 INFO: Start virtual machine named runner-1
15:17:24.629 INFO: The GitHub app has not been installed. Please install it from the developer settings.
15:17:24.633 INFO: Could not connect to the virtual machine over SSH, so the virtual machine will be shut down.
15:17:24.638 INFO: Virtual machine named runner-1 stopped with message: Failed establishing SSH connection

I confirm the machine when booted with tart run will accept my ssh logins using the same username/password I'm giving tartelet (admin/admin), and I believe my github app and tartelet github settings are correctly configured and supplied with the proper key...

Anyone know what I'm running into here? Are both of the errors I see in these logs potential culprits? Or is one related to the other?

Did I miss something in the VM setup instructions that I need to do to this VM? Or is there a reason it's not a good fit for tartelet?

Thanks.

https://github.com/user-attachments/assets/ba3505f4-f8a7-4b7b-a1f7-a6836bbfb49d

(this screen grab was taken after playing around in the VM a bit, but I originally produced this same behavior on a pristine clone)

Any additional context?

No response

krzysztof-pawlicki-nemo commented 1 month ago

it behaves similarly after yesterday's upgrade to macOS sequoia on host. I was debugging the app a bit and seems like:

nmichlo commented 1 month ago

Same issue here after upgrading to sequoia

krzysztof-pawlicki-nemo commented 1 month ago

I've spent a bit more time on trying to figure out what's behind and so far seems like there is a new Local Network Privacy feature (https://forums.developer.apple.com/forums/thread/760964) which probably causes the issue.

ryanhanks-bestow commented 1 month ago

@krzysztof-pawlicki-nemo in regards to the test you mentioned, I assume you're probably aware, but in case you missed it, I believe the list of VMs by default pulls from ~/.tart and is configurable via TART_HOME

Screenshot 2024-09-25 at 4 17 29 AM
ryanhanks-bestow commented 1 month ago

FWIW, I reproduced this again on a fresh macOS install (14.7). Steps to reproduce:

  1. Clean macOS install
  2. install brew
  3. install tart
  4. clone ghcr.io/cirruslabs/macos-sonoma-xcode:latest
  5. Install tartelet
  6. configure tartelet

Also, I experienced this with the prior version of tartelet.

If anyone knows the max supported macOS (or can confirm a working version), please let me know.

ryanhanks-bestow commented 1 month ago

Noticed some stuff in the diff of https://github.com/shapehq/tartelet/releases/tag/0.8.1 that looked possibly related to what I saw in the logs and what @krzysztof-pawlicki-nemo was digging into so I tried https://github.com/shapehq/tartelet/releases/tag/0.8.0 and looks like this version is working (at least doesn't cause the restart-loop I was seeing in in 0.10.0 and 0.9.0). I'll play around with it a bit later to see if I can get some workflows running and report back.

cc @nmichlo

ryanhanks-bestow commented 1 month ago

I got this running workflow with the 0.8.0 release. With this release I was able to see that the failure I was experiencing was indeed inherent to the specifics of ghcr.io/cirruslabs/macos-sonoma-xcode.

The clone of ghcr.io/cirruslabs/macos-sonoma-xcode comes with lots of nice things – one of those happens to be a preinstallation of actions-runner in ~/. This conflicts with the boot.sh script and causes it to fail.

Editing the virtual machine and removing ~/actions-runner allowed the machine to fully boot and register, and I've been successfully running workflows.

This problem seems to be what was also ailing my tests on 0.9.0 and 0.10.0, which are also now working for me. The 0.8.0 release just happened to give me a bit more visibility into what was happening.

In summary, to use ghcr.io/cirruslabs/macos-sonoma-xcode:

  1. tart clone ghcr.io/cirruslabs/macos-sonoma-xcode:latest runner
  2. Configure tartelet
  3. Edit the Virtual Machine and:
    ssh github.com # this adds github as a known host
    rm -rf ~/actions-runner # removes conflicting actions-runner installation
  4. Shutdown the vm
  5. Start tartelet

Again, I'm running this on Sonoma 14.7 metal, so the issues @krzysztof-pawlicki-nemo and @nmichlo chimed in with may or may not be related. I never confirmed – are y'all also using some variant of ghcr.io/cirruslabs/macos-sonoma-xcode?

krzysztof-pawlicki-nemo commented 1 month ago

Hey @ryanhanks-bestow , thanks for your input. I was busy with other tasks and put it on a side for some time but now can digg again. I use locally build (via packer) tart image. This is a completely fresh install from macOS Sonoma image without GitHub Actions preinstall. So far the very same image was used as a custom GitLab runner but I'd like to migrate to self hosted GitHub. I wanted to do small audit of the code and compiled/notarised Tartelet on my own. The same Tartelet binary works on different host (Sonoma) with the very same tart image. The issue appeared after I upgraded my dev machine (working as additional runner - host) to Sequoia.

Btw. this project looks very promising, thank you @simonbs for all the nice work you've put into this! I hope my feasibility phase will end with a success on Sequoia host and I'll be able to contribute bit more in here.