Network-accessible TILT servers should be available

[beckettsean]

As SL We're electing to create TILT servers in the cloud for the dev team to use Bc TILT doesn't run correctly on ARM hardware (and we don't want to tackle solving the ARM problem)

### Accessible TILT Servers
- [x] create a base TILT server image
- [x] https://github.com/strangelove-ventures/wormhole/issues/68
- [x] make changes and extensions to the base image
- [x] publish
- [x] rm public keys etc
- [x] use it and kick the tires and validate it's solid
- [ ] frequently recreate dev-specific instances to stay on latest config
- [ ] https://github.com/strangelove-ventures/wormhole/issues/72

[beckettsean]

Continuation of learnings from #52

[pharr117]

Met with the team and suggested using Packer (for creating the image) and Terraform for deploying the image.

A separate Terraform project can be used to deploy the image to a server for developer access.

[jonathanpberger]

@jtieri sez "client asked for this to be visible and / or open-sourced". Can we do this? Any contractual or other constraints? @beckettsean would love yr thoughts.

[beckettsean]

Get someone with a reasonable title (VP Eng, CTO, or higher) from the client to state in writing that they want us to open-source this code. (e.g. email or a public post in GitHub, something permanent and discoverable in a couple years if there's legal action)

Ask them to determine the license, with us defaulting to Apache 2 if they have no opinion. Copy the code into an empty, history-free, public repo under strangelove-ventures, add the license, give the client the link.

We would not make the actual VPS instances public, nor should anything from our GCP projects allow public or unauthenticated access.

I would say that in this case, the client can locally override any contractual IP considerations that might be in the SOW or MSA. cc @chipcope for verification of that assumption.

[pharr117]

I have checked off the "remove pubkeys" after cleaning up the repo, generalizing more of the Packer and Terraform code and rewriting history on the repo to remove any items that we would not want public-facing. If you have pulled this repo recently, you will need to reset to the origin as the history has been rewritten

[chipcope]

@beckettsean - We'd need to pair briefly so that I can make sure I'm clear on the details, but I generally agree that this assumption is correct. Depending on what trying to accomplish, there are a few different ways we could set this up. For example, if we want a license election in a GH repo to govern, we can provide a URL and expressly incorporate the language at that URL into the Agreement by reference. I think we'd likely have language in the Agreement stating that Apache 2.0 is the default, subject to a contrary election (I'm assuming the Customer will have unilateral discretion to alter the license) at the GH URL. Another option would be to enshrine a license change procedure in the Agreement setting forth a simplified process whereby the Customer and SL can formally amend the Agreement itself in order to affect any changes to the license.

I would say that in this case, the client can locally override any contractual IP considerations that might be in the SOW or MSA. cc @chipcope for verification of that assumption.

[pharr117]

Tire Kicking

Current results of my tire kicking:

1. minikube works with the docker driver (default). The Womrhole docs say to use the kvm driver but its probably not required. If it is required, then GCP Compute Engine supports nested virtualization with kvm anyway so we can make it work
2. Tilt probably requires a good amount of resources. With a smallish CE instance (e2-medium), Tilt is starting to get context deadline exceeded during k8s cluster liveness checks, possibly to to resource constraints
   • CPU seems find but memory may be the issue, see the htop image
   • k8s locks up pretty hard when running tilt up
3. Tilt has a great web interface that it makes available on a random port at startup. This is easier to work with than tilt up --stream=true which streams the logs.
   • I will need to come up with a safe way of making the server available from the local browser
4. I have built in a wormchain-dev user that has a pubkey added to SSH auth. This could be useful for/if
   • Wanting to SSH from your CLI instead of through the GCP web interface
   • It has the wormhole repo git clone already, so less work but not a huge deal
5. GCP allows SSH through the web interface, see the image blow on where to find it
6. Addendum to (4), it looks like GCP deletes the authorized_keys file from users because they want to manage SSH entirely using a daemon. We may just need to work with GCE here and not setup users with extra stuff at build time
   • See here: https://stackoverflow.com/questions/76927224/why-are-my-authorized-ssh-keys-being-deleted-in-my-google-vm-and-how-can-i-keep

HTOP image:

Browser SSH Connection Location

[pharr117]

This PR added helper scripts as suggested in the previous meetings that allow quickly starting the required infra/commands for running Tilt: https://github.com/strangelove-ventures/sl-vps/pull/1

It adds 2 scripts to the images that are globally accessible for all users on the machine:

1. wormchain-start-minikube - starts minikube with sane defaults
2. wormchain-start-tilt - starts minikube (if not already started), then runs tilt up with sane defaults

Running these commands with --help will provide usage info.

The final issue to tackle is for secure access to the images, which is an infra-specific (GCP) issue. This is documented in the internal repo so we can track security-specific tasks on the deployment code.