Consider alternatives to Jenkins

[caufieldjh]

Could use as alternatives to Jenkins:

• Google Cloud Terraform
• Ansible, with GCP plugin

[caufieldjh]

Re: GCP and Terraform, see also:

• https://cloud.google.com/community/tutorials/getting-started-on-gcp-with-terraform
• https://cloud.google.com/community/tutorials/managing-gcp-projects-with-terraform
• https://registry.terraform.io/providers/hashicorp/google/latest/docs

Terraform also has a "Terraform Cloud" variant:

Terraform Cloud (TFC) is a free to use, self-service SaaS platform that extends the capabilities of the open source Terraform CLI and adds collaboration and automation features.

Terraform Cloud enables connecting to common VCS platforms (GitHub, GitLab, Bitbucket) and triggering Terraform runs (plan and apply) from changes to configuration within the VCS. TFC manages state for the user, including keeping a history of changes. Terraform Cloud exposes an HTTP API that anyone can integrate with to build more automation around infrastructure change.

This might be overkill but it may also be more closely aligned with our intent for NEAT and KG-Hub.

[caufieldjh]

A nice thing about the Terraform Cloud: it includes cost estimations for each run.

[justaddcoffee]

I'm warming up to using Terraform.

One way we could do this:

• run terraform in our Jenkinsfile in this scheduler repo, from within a Docker image to make the Jenkinsfile more portable
• have the terraform code spin up either
  • a GCE instance that we prepopulate with all the dependencies (the code in this scheduler repo itself, along with NEAT, CUDA, tensorflow, Embiggen, Ensmallen, etc) OR
  • a from-scratch GCE instance that we build from a template (this is likely to cause some CUDA/tensorflow headaches)
• from the GCE instance, run a script (that lives in the current scheduling repo) that queries each project in the KG-Hub S3 bucket for 1) a template neat.yaml and 2) any build directories that don't have a graph_ml directory (indicating that we need to run NEAT

[justaddcoffee]

Adding: I realize plan this doesn't exactly comport with the title of this ticket, "Consider alternatives to Jenkins", since I'm suggesting we use both Jenkins and Terraform.

However, I think it does make some sense to either keep using Jenkins for ETL and also NEAT/GCE stuff (with Terraform for GCE stuff), or abandon Jenkins altogether and go all in with Terraform (which we could also do)

[caufieldjh]

I think Jenkins still works very well for the ETL functions, KG publishing, and even things like starting DB instances/journals, but I think part of the value added by Terraform may be that it reduces the number of points of failure between users and GCE. Terraform could certainly be set to run on a cron schedule much like Jenkins, spin up a pre-made listener instance to do the bucket query, and then spin up GCE instances as needed based on an image containing the necessary dependencies.

But in the end, how do we abstract away most of the Terraform parts?

[justaddcoffee]

This is a helpful little walkthrough/getting-started on Terraform, and also this Medium post shows one way of doing something similar to what I'm proposing