Incoherence between documentation and reality (build stanza is required)

[dotdiego]

The documentation states :

Each stage in Waypoint is optional. Waypoint doesn't need to own or run each stage. For example, your builds may happen in CI using other tools. Waypoint can use the result of that build to perform only the "deploy" and "release" steps. Any combination can be mixed.

These stages can also be broken down to the individual commands waypoint build, waypoint deploy, and waypoint release. This allows customization of the application lifecycle by inserting your own phases into the Waypoint workflow.

However creating an waypoint.hcl without a build stanza results in :

The following validation issues were detected:
! /waypoint/waypoint.hcl:14,1-10: 'build' stanza required;

What can I do to only deploy an app that has already been packaged on a registry by another tool (or someone else) ?

For notes: I'm trying to deploy all my homelab applications from a repository with only nomad-jobspec.
Following the GitOps way even tough we don't have a controller with a reconciliation loop yet in Nomad.

[briancain]

@dotdiego - Yes, that specific piece of documentation is mostly meant to say that you don't have to use Waypoint to build your application. But Waypoint still expects you to be deploying some built artifact. In that case, your build stanza would be a docker-pull, so from that sense yes building is optional, but the build stanza isn't.

[dotdiego]

@briancain ok nice one !

So the docker-pull only pulls the image.

Then when I try to add a data_source to enable git source.

runner {
  enable = true

  data_source "git" {
    url  = "https://github.com/dotdiego/homelab-nomad-jobs.git"
  }
}

I get an error

 | ==> Missing arguments:
  | 
  |     - docker.AccessInfo

The waypoint workflow i'm trying to achieve is the following :

• Get git repository hosting all my nomad jobspec files
• Deploy every nomad job within that repository with waypoint.

All jobs are using images already built from public registries.

Currently I could get this workflow running by cloning manually my repo and using templating to create the app stanza and it's children. But i'd prefer to automate it and tell waypoint to pick up a repository and execute waypoint up.

[briancain]

Yes @dotdiego - That error is telling you that if you are going to pull a container from a registry, you need to provide access credentials. So something like:

 build {
    use "docker-pull" {
      image = var.image
      tag   = var.tag
    }

    registry {
      use "docker" {
        image    = var.image
        tag      = var.tag
        username = var.registry_username
        password = var.registry_password
        local    = var.registry_local
      }
    }
  }

[dotdiego]

@briancain retrying again with waypoint.

I'm pulling the hashicorp/demo-webapp-lb-guide so no authentication should be needed as it's a public image from docker hub.

It's unfortunate that there are no documentation to do a basic : docker pull image -> deploy using nomad (plus nomad is already fetching the image from docker if needed, so this build step should not be mandatory)

[dotdiego]

After a bit of work I got something working.

So for the users that will get in the same situation as me :

• Waypoint (v0.10.0) can't do this kind of workflow empty/absent build -> deploy.
  That's unfortunate because sometimes you just need a tool that deploys image, not build them or pull them from a registry (because k8s/nomad/etc can do it).
• Waypoint using the UI or the "GitOps" functionality or enabling a data_source in your waypoint.hcl can't do a build step without a registry set up to upload the downloaded artifact (because of remote runners). Another blocking point to the workaround suggested here to use a docker-pull during build to create an "empty build".

That kind of workflow must be run in a waypoint console to run locally so no automation there.

I'll go back using other tools for CI/CD, but I hope those kind of ways to use waypoint will be added in future releases.

[dotdiego]

After many trials and errors, I got something working that seems better (issues mentioned in my previous comment are still present but I have some workarounds).

On the ideas an integration with Vault would be amazing. Because using the UI to add for each project secrets is not really fun, and the input variables in the UI have no possibility to be hidden.

I'm facing the same kaniko issue as #2984 so I used the same workaround.

waypoint runner profile set -name nomad-bootstrap-profile -plugin-type nomad -env-var "container=docker" -default

Then I used this waypoint.hcl to finally deploy something from the UI.

project = "init"

variable "registry_username" {
  type = string
}

variable "registry_password" {
  type      = string
  sensitive = true
}

runner {
  enabled = true
  // profile set to avoid kaniko --force issue
  profile = "nomad-bootstrap-profile"
  data_source "git" {
    url = "https://github.com/dotdiego/waypoint-test.git"
  }
}

app "demo" {
  // get a small image from docker because we can't bypass the build step
  build {
    use "docker-pull" {
      image              = "hello-world"
      tag                = "latest"
      disable_entrypoint = true
    }

    // push it to docker registry because remote-runner needs a registry block
    registry {
      use "docker" {
        image    = "${var.registry_username}/hello-world"
        tag      = "latest"
        username = var.registry_username
        password = var.registry_password
      }
    }
  }

  // finally deploy to nomad
  deploy {
    use "nomad-jobspec" {
      jobspec = templatefile("${path.project}/webapp.nomad")
    }
  }
}

[briancain]

On the ideas an integration with Vault would be amazing. Because using the UI to add for each project secrets is not really fun, and the input variables in the UI have no possibility to be hidden.

You can do this today with the Vault config sourcer plugin!

https://waypointproject.io/plugins/vault

You can see a real example in one of my example Waypoint projects: https://github.com/briancain/waypoint-tetris/blob/main/waypoint.hcl#L252-L270

variable "registry_username" {
  default = dynamic("vault", {
    path = "secret/data/registry"
    key  = "/data/registry_username"
  })
  type        = string
  sensitive   = true
  description = "username for container registry"
}

variable "registry_password" {
  default = dynamic("vault", {
    path = "secret/data/registry"
    key  = "/data/registry_password"
  })
  type        = string
  sensitive   = true
  description = "password for registry" // don't hack me plz
}

[briancain]

Hey there @dotdiego - can you link me to where you found this documentation you reported? Thanks.

[dotdiego]

@briancain sure : https://developer.hashicorp.com/waypoint/docs/lifecycle

The github file behind this documentation I think is this one : https://github.com/hashicorp/waypoint/blob/4f7174861053bb0bf51a17a26d56120867c5b7b1/website/content/docs/lifecycle/index.mdx

[briancain]

Thanks @dotdiego ! That explains why I couldn't find that sentence 😅 We had made some docs fixes but they hadn't yet made it to our stable website branch. This should be resolved now!