search

hashicorp / waypoint

A tool to build, deploy, and release any application on any platform.
https://waypointproject.io
Other
4.76k stars 327 forks source link

Allow using Nomad Job Specification for configuring Waypoint Cluster, Runner and On-demand Runners #3267

Open kingindanord opened 2 years ago

kingindanord commented 2 years ago

Is your feature request related to a problem? Please describe.

there are many pieces missing when working with Waypoint on Nomad Platform

for example:

Describe the solution you'd like

Be able to provide a complete Nomad Job Specification when creating Waypoint cluster and runners. In this case, we will be able to provide all the needed secrets and artifiacts at the run time and have the possibility to render them from Vault, in addition to having more control over other aspects of the job like network stanza.

Describe alternatives you've considered

I couldn't make the official guidelines work for my setup yet. My last try will be to hard code all of missing configuration on a custom ODR image and use it, which is far away of being a good practice.

Explain any additional use-cases

It will be very helpful if we could specify in the runner profile which Nomad Job Specification template On-demand runners should use.

something like this:

waypoint runner profile set ... -nomad-odr-job-template=odr-template.hcl

odr-template.hcl

job "waypoint-odr" {
  datacenters = ["dc1"]
  type = "service"
  group "service" {
    count = 1
    network {
      mode = "host"
      dns {
        servers = ["172.17.0.1"]
      }
    }

    task "odr-task-xxx" {
      driver = "docker"

      env {
        NOMAD_ADDR                      = "https://nomad.service.consul:4646"
        NOMAD_SKIP_VERIFY               = "false"
        WAYPOINT_SERVER_ADDR            = "waypoint-server.service.consul:9701"
        WAYPOINT_SERVER_TLS             = "true"
        WAYPOINT_SERVER_TLS_SKIP_VERIFY = "true"
        NOMAD_CACERT                    = "${NOMAD_SECRETS_DIR}/cachain_nomad.pem"
        NOMAD_CLIENT_KEY                = "${NOMAD_SECRETS_DIR}/nomad.key"
        NOMAD_CLIENT_CERT               = "${NOMAD_SECRETS_DIR}/nomad.pem"
      }

      template {
        data        = <<EOH
{{ with secret "pki/v1/ica2/v1/issue/blinchik_user_cert_ica2" "common_name=nomad.service.consul" "ttl=30536000" }}
{{- .Data.issuing_ca -}}
{{ end }}
EOH
        destination = "${NOMAD_SECRETS_DIR}/cachain_nomad.pem"
        change_mode = "restart"
      }

      template {
        data        = <<EOH
{{ with secret "pki/v1/ica2/v1/issue/cert_ica2" "common_name=nomad.service.consul" "ttl=30536000" }}
{{- .Data.certificate -}}
{{ end }}
EOH
        destination = "${NOMAD_SECRETS_DIR}/nomad.pem"
        change_mode = "restart"
      }

      template {
        data        = <<EOH
{{ with secret "pki/v1/ica2/v1/issue/cert_ica2" "common_name=nomad.service.consul" "ttl=30536000" }}
{{- .Data.private_key -}}
{{ end }}
EOH
        destination = "${NOMAD_SECRETS_DIR}/nomad.key"
        change_mode = "restart"
      }

      template {
        data = <<EOH
                WAYPOINT_SERVER_TOKEN="{{ with secret "kv/data/nomad" }}{{.Data.data.runner_token}}{{ end }}"
                NOMAD_TOKEN="{{ with secret "kv/data/nomad" }}{{.Data.data.nomad_admin}}{{ end }}"

            EOH

        destination = "secrets/file.env"
        env         = true
      }

      config {
        image = "hashicorp/waypoint-odr"
        auth_soft_fail = "true"
        args = [
          "runner",
          "agent",
          "-vv",
          "-id",
          "xxx",
          "-odr",
          "-odr-profile-id",
          "zzz"
        ]
      }

      resources {
        cpu    = 256  # MHz
        memory = 300 # MB
      }
    }
  }
}
paladin-devops commented 2 years ago

Hi @kingindanord! Can you help us clarify what you're looking for - recommended jobspec files/templates for installing the Waypoint server & runner, or do you mean supplying a jobspec (for the server, static runner, and ODR) to the waypoint install command?

kingindanord commented 2 years ago

Hi @paladin-devops! a tutorial or more documentation on deploying Waypoint in a production environment with the assumption that all other related compononts have TLS and ACLs enabled would be great for sure.

Regarding the installation, I am not using waypoint install anymore. I switched to creating the waypoint server with a Nomad Job, bootstraping the server then deploying a remote Waypoint runner also with a Nomad Job. This part works fine.

However, my current pain point is to customize ODR and make it work at all. So what I am doing now is that I am creating a new Docker/OCI image extended from hashicorp/waypoint-odr and add my workarounds there. For example, changing the DNS resolver configuration or else the ODR containers can not communicate with the server. For this in Dockerfile I add ENTRYPOINT ["/etc/run.sh"] and the /etc/run.sh looks like this

#!/bin/ash

cat <<EOF > /etc/resolv.conf
search eu-west-1.compute.internal
nameserver 172.17.0.1
options edns0
EOF

/kaniko/waypoint "$@"

I thought this one could be changed within the -plugin-config when creating a runner profile, somewhere within the nomad_plugin.json file bellow, but I did not have luck with that.

cat <<EOF > nomad_plugin.json
{
        "datacenter": "brain",
        "namespace": "default",
        "nomad_host": "https://nomad.service.brain.consul:4646",
        "region": "global",
        "resources_cpu": "1200",
        "resources_memory": "1200"
}
EOF

waypoint runner profile set -app=app -project=test -name=nomad -oci-url=local-docker-registry.service.brain.consul:5000/waypoint-odr-vault:v0.1.1 \
-plugin-type=nomad -plugin-config=nomad_plugin.json -default \
-env-var='WAYPOINT_SERVER_ADDR=waypoint-server.service.consul:9701' \
-env-var='WAYPOINT_SERVER_TLS=true' \
-env-var='WAYPOINT_SERVER_TLS_SKIP_VERIFY=true'

So my idea is, why not to allow to supply a complete nomad jobspec in -plugin-config or a new option when creating a runner profile.

I hope this helps and give more clarity.

jhillyerd commented 2 years ago

I would also like to see a more production like tutorial for Nomad. I'm trying Waypoint out in my home-lab, and have spent hours trying to get it working with my TLS+ACL'd Nomad cluster.

Some non-obvious things that should be included in a Nomad tutorial:

# Runner can't spawn jobs on ACL nomad without:
waypoint config set -runner -scope=global NOMAD_TOKEN=22d...

# Kaniko doesn't think it is in a container without:
waypoint config set -runner -scope=global container=docker
dotdiego commented 2 years ago

It would really be helpful to set the DNS for waypoint in some way. Currently it's defaulting to 8.8.8.8 and it can't resolve local dns addresses provided by consul.

Allan-Nava commented 2 years ago

How to set programmatically in waypoint.hcl the nomad url?


app "app-frontend-waypoint" {
    #
    labels = {
    }
    #
    build {
        #use "pack" {}
        use "docker" {}
        registry {
            # so now we can create the correct image for production
            use "docker" {
            }
        }
    }
    #
    deploy {
        use "nomad-jobspec" {
       }
}
paladin-devops commented 2 years ago

Just to summarize the work items from this ticket:

I think that the 1st item in this list could be accomplished with a Nomad Pack for Waypoint, where variables could be templated into the pack for custom values, not unlike how there is a Helm chart for installing Waypoint to Kubernetes.

paladin-devops commented 2 years ago

And @Allan-Nava, to set the Nomad URL, there are a couple of options. The plugin configuration for the Nomad task launcher has the nomad_host configuration. This is the address which will be used to launch on-demand runners in Nomad. You can also set the NOMAD_ADDR environment variable for runner configuration via this command: waypoint config set -runner -scope=global NOMAD_ADDR=http://localhost:4646.

Allan-Nava commented 2 years ago

And @Allan-Nava, to set the Nomad URL, there are a couple of options. The plugin configuration for the Nomad task launcher has the nomad_host configuration. This is the address which will be used to launch on-demand runners in Nomad. You can also set the NOMAD_ADDR environment variable for runner configuration via this command: waypoint config set -runner -scope=global NOMAD_ADDR=http://localhost:46461.

I wanna use waypoint in production to CI for my applications

Allan-Nava commented 2 years ago

But in "nomad-jobspec" is it possible to force the nomad_host ?

And is there any CI like Circle CI to implement the testing code?

paladin-devops commented 2 years ago

But in "nomad-jobspec" is it possible to force the nomad_host ?

If you're using a local runner (meaning if when you run Waypoint operations such as waypoint build, you use the flag -local=true), simply exporting the environment variable NOMAD_ADDR= should be sufficient.

If you're using remote runners, then I recommend using the other method I mentioned, waypoint config set -runner -scope=global NOMAD_ADDR=http://localhost:4646. The runner system will use this configuration to inform the Nomad plugin of what Nomad endpoint to communicate with for deploying your job. 😃

And is there any CI like Circle CI to implement the testing code?

There is no "test" phase of the Waypoint lifecycle. However, with the recent release of custom pipelines, you can absolutely run tests in a pipeline!

Allan-Nava commented 2 years ago

But in "nomad-jobspec" is it possible to force the nomad_host ?

If you're using a local runner (meaning if when you run Waypoint operations such as waypoint build, you use the flag -local=true), simply exporting the environment variable NOMAD_ADDR= should be sufficient.

If you're using remote runners, then I recommend using the other method I mentioned, waypoint config set -runner -scope=global NOMAD_ADDR=http://localhost:4646. The runner system will use this configuration to inform the Nomad plugin of what Nomad endpoint to communicate with for deploying your job. 😃

And is there any CI like Circle CI to implement the testing code?

There is no "test" phase of the Waypoint lifecycle. However, with the recent release of custom pipelines, you can absolutely run tests in a pipeline!

Ok perfect! thanks for the explanation

How can I implement the custom pipelines?

paladin-devops commented 2 years ago

@Allan-Nava our latest and greatest documentation on pipelines is here on our docs site! Let us know if you have any feedback. 😄

Allan-Nava commented 2 years ago

@Allan-Nava our latest and greatest documentation on pipelines is here on our docs site! Let us know if you have any feedback. 😄

I will try