Waypoint Remote Runner Returning Error Code (1)

[zenchild]

NOTE: This is also logged in the discussions

Describe the bug We just started receiving an error today after our Server updated to 0.8.2. We have Github Actions in place that build our code on waypoint runners and they are now coming back with a return code of "1" instead of "0," causing the Github Action to fail.

Steps to Reproduce If I run waypoint locally with a waypoint up and do an echo $? afterward, it is "0."

If I run it remotely like, waypoint up -local=false -remote-source="ref=develop" and do an echo $? afterward, it is "1."

Please include any waypoint.hcl files if applicable, as well as a GitHub Gist of any relevant logs or steps to reproduce the bug. Running waypoint commands with -v up to -vvv will include any additional debugging info in the log.

Expected behavior Running waypoint remotely should return a "0" instead of a "1"

Waypoint Platform Versions Additional version and platform information to help triage the issue if applicable:

• Waypoint CLI Version: 0.8.2
• Waypoint Server Platform and Version: nomad - 0.8.2
• Waypoint Plugin: Remote runner

Additional context

• The develop branch is sync'd correctly for the app so the local and remote builds are working with the same code.
• The server is 0.8.2
• The runner is 0.8.2
• My local client is 0.8.2
• We also saw the issue when the runner was still at 0.8.0.

[paladin-devops]

Hey @zenchild, thanks for opening this issue! Could you provide a few more things for troubleshooting?

• [ ] server logs
• [ ] job logs from running waypoint up -local=false -vvv

[zenchild]

Sorry all. I've been slammed this week. Will try and get these soon. We have a really sad work-around of just doing a || echo "Done" after the remote command 🤣

[zenchild]

Sorry it's taken so long to get these rounded up. I've included:

• The waypoint server logs
  • waypoint-server.logs.txt
• The messages logs from the runner
  • waypoint-runner.logs.txt
• The console output from waypoint up -local=false -remote-source="ref=develop"
  • waypoint-cli.logs.txt

[evanphx]

Hi @zenchild,

During discussion we were trying to figure out a way to reproduce the issue, since we'll need that to get the bottom of it. Do you have a reproduction you can share with us to, at minimum the waypoint.hcl file.

Thanks!

[zenchild]

@evanphx, Here is an example waypoint.hcl file. We are seeing this in all of our waypoint deployments though. The common items:

• waypoint server ( running in nomad )
• waypoint runner ( EC2 instance )
• dynamic config pulling from Vault
• images pushing to ECR
• jobs running in Nomad (using nomad-jobspec)