search

hashicorp / waypoint-helm

Helm chart to install Waypoint and other associated components.
Mozilla Public License 2.0
29 stars 14 forks source link

Waypoint never creates token to log in #22

Open alexmherrmann opened 2 years ago

alexmherrmann commented 2 years ago

Describe the bug A clear and concise description of what the bug is.

Whatever step of the process that is supposed to populate the token secret never does, meaning the runner never gets brought up. Waypoint cannot login either:

Waypoint server URL: 
! The Waypoint token in the Kubernetes secret is empty. This is usually
  for one of two reasons. First, the Waypoint server may not be bootstrapped.
  After installing Waypoint on Kubernetes, it takes a few minutes for Waypoint
  to bootstrap itself.

  If Waypoint is already bootstrapped, it's possible the server administrator
  already deleted the secret. Future users should not use this authentication
  method and should instead ask another Waypoint user to generate an invite token
  for them.

I have let it sit for 40 minutes and still nothing.

Steps to Reproduce Use the following terraform helm_release resource to create the waypoint server (will need kubernetes and a "sandbox" namespace)


resource helm_release waypoint {
  timeout = "600"
  repository = "https://helm.releases.hashicorp.com"
  chart      = "waypoint"
  name       = "waypoint"
  namespace  = "sandbox"
}

Expected behavior The helm install should bootstrap correctly

Waypoint Platform Versions Terraform v1.0.11 Digitalocean Kubernetes platform waypoint cli 0.7

Additional context Add any other context about the problem here.

briancain commented 2 years ago

Hey there @alexmherrmann - Thanks for reporting the issue! Could you provide us with the output from Terraform running helm install?

Also, the "chart" name is hashicorp/waypoint, not just waypoint. I wonder if that's the issue here 😄

alexmherrmann commented 2 years ago

It's just terraform timing out but here's what I have, thanks for the reply!

module.waypoint.helm_release.waypoint: Still creating... [9m30s elapsed]
module.waypoint.helm_release.waypoint: Still creating... [9m40s elapsed]
module.waypoint.helm_release.waypoint: Still creating... [9m50s elapsed]
module.waypoint.helm_release.waypoint: Still creating... [10m0s elapsed]
╷
│ Warning: Helm release "waypoint" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│ 
│   with module.waypoint.helm_release.waypoint,
│   on waypoint_server/waypoint.tf line 4, in resource "helm_release" "waypoint":
│    4: resource helm_release waypoint {
│ 
╵
╷
│ Error: timed out waiting for the condition
│ 
│   with module.waypoint.helm_release.waypoint,
│   on waypoint_server/waypoint.tf line 4, in resource "helm_release" "waypoint":
│    4: resource helm_release waypoint {
│
briancain commented 2 years ago

@alexmherrmann - Can you try the chart name I suggested? I imagine it's timing out because it's using a chart named waypoint instead of hashicorp/waypoint. Thanks!

alexmherrmann commented 2 years ago

Ahh it's not that unfortunately, I can see waypoint resources being created and because terraform doesn't actually "register" (not sure what it's called) the helm repository that actually fails with the below message.

╷
│ Error: chart "hashicorp/waypoint" not found in https://helm.releases.hashicorp.com repository
│ 
│   with module.waypoint.helm_release.waypoint,
│   on waypoint_server/waypoint.tf line 4, in resource "helm_release" "waypoint":
│    4: resource helm_release waypoint {
│ 
╵

For reference here is what I updated the terraform script to.

resource helm_release waypoint {
  timeout = "100"
  repository = "https://helm.releases.hashicorp.com"
  chart      = "hashicorp/waypoint"
  name       = "waypoint"
  namespace  = var.namespace
}
alexmherrmann commented 2 years ago

Here is what is happening after letting the waypoint server sit for 20 minutes or so.

Screen Shot 2022-01-21 at 8 43 08 AM

Notice how the server token is still 0 bytes. Are there any other logs you'd need?

briancain commented 2 years ago

Not at the moment, thank you for the extra information @alexmherrmann !

Just as a quick smoke test, if you install the chart via helm directly, does it successfully install on your cluster? So instead of terraform, just try helm install waypoint hashicorp/waypoint. Does that behave in the same way? Thank you again for providing all the extra information to help us debug this!

alexmherrmann commented 2 years ago

Appreciate you helping out!

And unfortunately it's the same thing it looks like 😞.

Screen Shot 2022-01-21 at 12 19 31 PM

mitchellh commented 2 years ago

Hm, I've been trying to reproduce this but can't. I've installed Waypoint successfully in GCP, DO, and Linode K8S using the helm chart with no custom values (just a helm install CLI). That bootstrap token is configured with the bootstrap job. Can you check the logs for that pod to see if bootstrapping ever succeeds?

rhabbachi commented 2 years ago

I've been consistently able to reproduce the issue on an eks cluster bootstaped by https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest

Logs from the bootstrap pod

2022-02-08T21:15:51.748Z [INFO]  waypoint: waypoint version: full_string="v0.7.1 (062857d16+CHANGES)" version=v0.7.1 prerelease="" metadata="" revision=062857d16+CHANGES
2022-02-08T21:15:51.748Z [TRACE] waypoint: starting interrupt listener for context cancellation
2022-02-08T21:15:51.749Z [DEBUG] waypoint: home configuration directory: path=/home/waypoint/.config/waypoint
Checking for service readiness every 5 seconds...
2022-02-08T21:15:51.750Z [TRACE] waypoint: interrupt listener goroutine started
2022-02-08T21:15:51.778Z [INFO]  waypoint: service ready: advertise_addr=6399845c2cd74c79a0cc-4f51b24a457a.eu-central-1.elb.amazonaws.com:9701
2022-02-08T21:15:51.778Z [INFO]  waypoint: initializing server connection
2022-02-08T21:15:51.779Z [TRACE] waypoint: no API client provided, initializing connection if possible
2022-02-08T21:15:51.779Z [INFO]  waypoint.server: attempting to source credentials and connect
2022-02-08T21:15:51.779Z [DEBUG] waypoint.serverclient: connection information: address=waypoint-server:9701 tls=true tls_skip_verify=true send_auth=false has_token=false
2022-02-08T21:17:51.779Z [TRACE] waypoint: stopping signal listeners and cancelling the context
! Error reconnecting with token: context deadline exceeded
rhabbachi commented 2 years ago

The issue is due to the terraform eks module default security groups being more restrictive then say the default eksctl configuration. The newly added Security Groups section should provide a workaround to the allow pod to pod communication.

alexmherrmann commented 2 years ago

Sorry for the late reply. I can try it on my microk8s cluster too. I'm not super well versed in kubernetes security yet and I can absolutely go back through @mitchellh and see if my problem is the same as @rhabbachi has, but just on DOKS. I haven't played with anything and just have a little namespace I put all my pet-project stuff in with whatever defaults are there.

I apologize but I probably won't be able to get to it until next week, very much appreciate the time looking from the project maintainers and the contributions from other users.

alexmherrmann commented 2 years ago

Hmm, I don't see a job being created 🤔

I'm using microk8s with rbac enabled in a separate namespace now.

I can see that the pods are created but I don't see the job.

Screen Shot 2022-02-25 at 3 39 42 PM Screen Shot 2022-02-25 at 3 39 47 PM

Sorry for taking so long to get back to you.

mhmnemati commented 2 years ago

Hey guys @briancain @mitchellh, I also have this problem,

I'm trying to install waypoint using this terraform resource stanza:

resource "helm_release" "waypoint" {
  name       = "waypoint"
  namespace  = "kube-system"
  repository = "https://helm.releases.hashicorp.com"
  chart      = "waypoint"
  version    = "0.1.6"
}

The CRD's will be created after applying terraform, but the waypoint-runner-0 pod will stucks on Init:0/1 state, I've checked the describe of pod and I got this message:

  Normal  Scheduled  16m   default-scheduler  Successfully assigned kube-system/waypoint-runner-0 to master-0
  Normal  Pulling    16m   kubelet            Pulling image "busybox"
  Normal  Pulled     16m   kubelet            Successfully pulled image "busybox" in 12.685709286s
  Normal  Created    15m   kubelet            Created container wait-for-token
  Normal  Started    15m   kubelet            Started container wait-for-token

Also, I've checked the created secrets and I got these:

sh.helm.release.v1.waypoint.v1                       helm.sh/release.v1                    1      17m
waypoint-bootstrap-token-m7mlc                       kubernetes.io/service-account-token   3      17m
waypoint-runner-odr-token-975h2                      kubernetes.io/service-account-token   3      17m
waypoint-runner-token                                Opaque                                1      17m
waypoint-runner-token-rbf2q                          kubernetes.io/service-account-token   3      17m
waypoint-server-token                                Opaque                                1      17m
waypoint-token-td2q6                                 kubernetes.io/service-account-token   3      17m
alexmherrmann commented 2 years ago

Looks like the job is now being created (on microk8s) but it just sits and sits and sits.

This is what I see in the logs of the job 10 minutes after it created Screen Shot 2022-03-23 at 3 04 49 PM .

jason-riddle commented 2 years ago

I ran into this.

I noticed that waypoint-bootstrap is waiting for a service to be created.

Checking for service readiness every 5 seconds...

Looking at the services, I noticed that the waypoint ui service is not being created since it's defined as type LoadBalancer. For me, I'm running minikube so this will never complete as LoadBalancer isn't available as a service option.

My workaround was to define the ui service to use a Cluster IP via helm values.

ui:
  service:
    enabled: true
    type: ClusterIP
jason-riddle commented 2 years ago

I misspoke, minikube does support LoadBalancer but it requires setting up a tunnel (https://minikube.sigs.k8s.io/docs/handbook/accessing/#loadbalancer-access).

Also, using ClusterIP is not a workaround because yes, the service now starts, but the waypoint client from outside the cluster can't connect to that IP address.

ryanwholey commented 2 years ago

I am able to reproduce this behavior when re-installing with old persistent volume claims.

With old PVCs

$ kubectl get pod
NAME                                     READY   STATUS      RESTARTS   AGE
waypoint-runner-0                        0/1     Init:0/1    0          2m14s
waypoint-server-0                        1/1     Running     0          2m14s
waypoint-bootstrap--1-gx4bd              0/1     Completed   0          2m14s
server logs 
2022-05-02T22:04:08.899Z [INFO]  waypoint: waypoint version: full_string="v0.7.2 (6e06e1a2d+CHANGES)" version=v0.7.2 prerelease="" metadata="" revision=6e06e1a2d+CHANGES
2022-05-02T22:04:08.901Z [DEBUG] waypoint: home configuration directory: path=/home/waypoint/.config/waypoint
2022-05-02T22:04:08.901Z [INFO]  waypoint.server: opening DB: path=/data/data.db
2022-05-02T22:04:08.902Z [DEBUG] waypoint.server.singleprocess: checking if DB restore is requested
2022-05-02T22:04:08.902Z [DEBUG] waypoint.server.singleprocess: no restore file found, no DB restore requested
2022-05-02T22:04:08.915Z [DEBUG] waypoint.server.singleprocess.url_service: API token not set in config, initializing guest account
2022-05-02T22:04:08.915Z [DEBUG] waypoint.server.singleprocess.url_service: using saved URL guest token
2022-05-02T22:04:09.109Z [DEBUG] waypoint.server.singleprocess.url_service: connection is ready
2022-05-02T22:04:09.109Z [INFO]  waypoint.server.singleprocess.url_service: URL service client successfully initialized
2022-05-02T22:04:09.109Z [DEBUG] waypoint.server.grpc: starting listener: addr=0.0.0.0:9701
2022-05-02T22:04:09.109Z [INFO]  waypoint.server.singleprocess.poll_queuer.project: starting
2022-05-02T22:04:09.109Z [INFO]  waypoint.server.singleprocess.poll_queuer.application_statusreport: starting
2022-05-02T22:04:09.109Z [INFO]  waypoint.server.singleprocess.prune: starting
2022-05-02T22:04:09.110Z [INFO]  waypoint.server.grpc: TLS cert wasn't specified, a self-signed certificate will be created
2022-05-02T22:04:09.205Z [INFO]  waypoint.server.grpc: listener is wrapped with TLS
2022-05-02T22:04:09.205Z [DEBUG] waypoint.server.http: starting listener: addr=0.0.0.0:9702
2022-05-02T22:04:09.205Z [INFO]  waypoint.server.http: TLS cert wasn't specified, a self-signed certificate will be created
2022-05-02T22:04:09.287Z [INFO]  waypoint.server.http: listener is wrapped with TLS
2022-05-02T22:04:09.287Z [DEBUG] waypoint.server.http_insecure: starting listener: addr=0.0.0.0:9703
2022-05-02T22:04:09.287Z [WARN]  waypoint.server.http_insecure: TLS is disabled for this listener
2022-05-02T22:04:09.287Z [INFO]  waypoint.server: starting built-in server: addr=[::]:9701

» Server configuration:

                  DB Path: /data/data.db
             gRPC Address: [::]:9701
             HTTP Address: [::]:9702
  HTTP Address (Insecure): [::]:9703
            Auth Required: yes
       Browser UI Enabled: yes
              URL Service: api.waypoint.run:443 (account: guest)

» Server logs:

2022-05-02T22:04:09.288Z [INFO]  waypoint.server.grpc: starting gRPC server: addr=[::]:9701
2022-05-02T22:04:09.288Z [INFO]  waypoint.server.http: starting HTTP server: ln=[::]:9703 addr=[::]:9703
2022-05-02T22:04:09.288Z [INFO]  waypoint.server.http: starting HTTP server: ln=[::]:9702 addr=[::]:9702
2022-05-02T22:04:10.126Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-05-02T22:04:10.126Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error= duration=65.961µs
bootstrap logs 
2022-05-02T22:04:10.024Z [INFO]  waypoint: waypoint version: full_string="v0.7.2 (6e06e1a2d+CHANGES)" version=v0.7.2 prerelease="" metadata="" revision=6e06e1a2d+CHANGES
2022-05-02T22:04:10.025Z [TRACE] waypoint: starting interrupt listener for context cancellation
2022-05-02T22:04:10.025Z [TRACE] waypoint: interrupt listener goroutine started
2022-05-02T22:04:10.026Z [DEBUG] waypoint: home configuration directory: path=/home/waypoint/.config/waypoint
Checking for service readiness every 5 seconds...
2022-05-02T22:04:10.055Z [INFO]  waypoint: service ready: advertise_addr=192.168.1.242:9701
2022-05-02T22:04:10.056Z [INFO]  waypoint: initializing server connection
2022-05-02T22:04:10.056Z [TRACE] waypoint: no API client provided, initializing connection if possible
2022-05-02T22:04:10.056Z [INFO]  waypoint.server: attempting to source credentials and connect
2022-05-02T22:04:10.056Z [DEBUG] waypoint.serverclient: connection information: address=waypoint-server:9701 tls=true tls_skip_verify=true send_auth=false has_token=false
2022-05-02T22:04:10.124Z [DEBUG] waypoint.server: connection established with sourced credentials
2022-05-02T22:04:10.124Z [TRACE] waypoint: requesting version info from server
2022-05-02T22:04:10.144Z [INFO]  waypoint: server version info: version=v0.7.2 api_min=1 api_current=1 entrypoint_min=1 entrypoint_current=1
2022-05-02T22:04:10.144Z [INFO]  waypoint: negotiated api version: version=1
2022-05-02T22:04:10.144Z [INFO]  waypoint: bootstrapping the server
Waypoint already bootstrapped. Doing nothing.
2022-05-02T22:04:10.146Z [TRACE] waypoint: stopping signal listeners and cancelling the context
runner wait-for-token logs 
Waiting for runner token secret '/secret/token' to be populated from secret...
Waiting for runner token secret '/secret/token' to be populated from secret...
Waiting for runner token secret '/secret/token' to be populated from secret...
....

tokens:

$ kubectl get secret waypoint-runner-token -o json | jq -cr '.data | map_values(@base64d)'
{"token":""}
$ kubectl get secret waypoint-server-token -o json | jq -cr '.data | map_values(@base64d)'
{"token":""}

After removing old PVCs

$ kubectl delete pvc data-default-waypoint-server-0
$ kubectl delete pvc data-default-waypoint-runner-0
server logs 
2022-05-02T22:40:44.690Z [INFO]  waypoint: waypoint version: full_string="v0.7.2 (6e06e1a2d+CHANGES)" version=v0.7.2 prerelease="" metadata="" revision=6e06e1a2d+CHANGES
2022-05-02T22:40:44.696Z [DEBUG] waypoint: home configuration directory: path=/home/waypoint/.config/waypoint
2022-05-02T22:40:44.696Z [INFO]  waypoint.server: opening DB: path=/data/data.db
2022-05-02T22:40:44.710Z [DEBUG] waypoint.server.singleprocess: checking if DB restore is requested
2022-05-02T22:40:44.710Z [DEBUG] waypoint.server.singleprocess: no restore file found, no DB restore requested
2022-05-02T22:40:44.731Z [DEBUG] waypoint.server.singleprocess.url_service: API token not set in config, initializing guest account
2022-05-02T22:40:44.731Z [DEBUG] waypoint.server.singleprocess.url_service: connecting to URL service to retrieve guest token: addr=api.waypoint.run:443 tls=true
2022-05-02T22:40:44.731Z [DEBUG] waypoint.server.singleprocess.url_service: waiting on server connection state to become ready
2022-05-02T22:40:46.216Z [DEBUG] waypoint.server.singleprocess.url_service: connection is ready
2022-05-02T22:40:46.389Z [DEBUG] waypoint.server.singleprocess.url_service: connection is ready
2022-05-02T22:40:46.389Z [INFO]  waypoint.server.singleprocess.url_service: URL service client successfully initialized
2022-05-02T22:40:46.389Z [DEBUG] waypoint.server.grpc: starting listener: addr=0.0.0.0:9701
2022-05-02T22:40:46.390Z [INFO]  waypoint.server.singleprocess.poll_queuer.application_statusreport: starting
2022-05-02T22:40:46.390Z [INFO]  waypoint.server.singleprocess.prune: starting
2022-05-02T22:40:46.389Z [INFO]  waypoint.server.singleprocess.poll_queuer.project: starting
2022-05-02T22:40:46.390Z [INFO]  waypoint.server.grpc: TLS cert wasn't specified, a self-signed certificate will be created
2022-05-02T22:40:46.480Z [INFO]  waypoint.server.grpc: listener is wrapped with TLS
2022-05-02T22:40:46.481Z [DEBUG] waypoint.server.http: starting listener: addr=0.0.0.0:9702
2022-05-02T22:40:46.481Z [INFO]  waypoint.server.http: TLS cert wasn't specified, a self-signed certificate will be created

» Server configuration:

                  DB Path: /data/data.db
             gRPC Address: [::]:9701
             HTTP Address: [::]:9702
  HTTP Address (Insecure): [::]:9703
            Auth Required: yes
       Browser UI Enabled: yes
              URL Service: api.waypoint.run:443 (account: guest)

» Server requires bootstrapping!

  New servers must be bootstrapped to retrieve the initial auth token for
  connections. To bootstrap this server, run the following command in your
  terminal once the server is up and running.

    waypoint server bootstrap -server-addr=[::]:9701 -server-tls-skip-verify

  This command will bootstrap the server and setup a CLI context.

» Server logs:

2022-05-02T22:40:46.570Z [INFO]  waypoint.server.http: listener is wrapped with TLS
2022-05-02T22:40:46.570Z [DEBUG] waypoint.server.http_insecure: starting listener: addr=0.0.0.0:9703
2022-05-02T22:40:46.570Z [WARN]  waypoint.server.http_insecure: TLS is disabled for this listener
2022-05-02T22:40:46.570Z [INFO]  waypoint.server: starting built-in server: addr=[::]:9701
2022-05-02T22:40:46.571Z [INFO]  waypoint.server.grpc: starting gRPC server: addr=[::]:9701
2022-05-02T22:40:46.571Z [INFO]  waypoint.server.http: starting HTTP server: ln=[::]:9703 addr=[::]:9703
2022-05-02T22:40:46.571Z [INFO]  waypoint.server.http: starting HTTP server: ln=[::]:9702 addr=[::]:9702
2022-05-02T22:40:48.924Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-05-02T22:40:48.924Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error= duration=134.387µs
2022-05-02T22:40:48.954Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/BootstrapToken request
2022-05-02T22:40:48.963Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/BootstrapToken response: error= duration=8.874821ms
2022-05-02T22:40:49.025Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2022-05-02T22:40:49.026Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error= duration=526.937µs
2022-05-02T22:40:49.029Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/SetServerConfig request
2022-05-02T22:40:49.036Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/SetServerConfig response: error= duration=7.554321ms
2022-05-02T22:40:49.040Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GenerateLoginToken request
2022-05-02T22:40:49.042Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GenerateLoginToken response: error= duration=2.370764ms
2022-05-02T22:40:49.100Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/UpsertOnDemandRunnerConfig request
2022-05-02T22:40:49.105Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/UpsertOnDemandRunnerConfig response: error= duration=4.87734ms
bootstrap logs 
2022-05-02T22:40:39.405Z [INFO]  waypoint: waypoint version: full_string="v0.7.2 (6e06e1a2d+CHANGES)" version=v0.7.2 prerelease="" metadata="" revision=6e06e1a2d+CHANGES
2022-05-02T22:40:39.405Z [TRACE] waypoint: starting interrupt listener for context cancellation
2022-05-02T22:40:39.406Z [TRACE] waypoint: interrupt listener goroutine started
2022-05-02T22:40:39.951Z [DEBUG] waypoint: home configuration directory: path=/home/waypoint/.config/waypoint
Checking for service readiness every 5 seconds...
2022-05-02T22:40:39.990Z [INFO]  waypoint: service ready: advertise_addr=192.168.1.242:9701
2022-05-02T22:40:39.991Z [INFO]  waypoint: initializing server connection
2022-05-02T22:40:39.991Z [TRACE] waypoint: no API client provided, initializing connection if possible
2022-05-02T22:40:39.991Z [INFO]  waypoint.server: attempting to source credentials and connect
2022-05-02T22:40:39.991Z [DEBUG] waypoint.serverclient: connection information: address=waypoint-server:9701 tls=true tls_skip_verify=true send_auth=false has_token=false
2022-05-02T22:40:48.922Z [DEBUG] waypoint.server: connection established with sourced credentials
2022-05-02T22:40:48.922Z [TRACE] waypoint: requesting version info from server
2022-05-02T22:40:48.949Z [INFO]  waypoint: server version info: version=v0.7.2 api_min=1 api_current=1 entrypoint_min=1 entrypoint_current=1
2022-05-02T22:40:48.949Z [INFO]  waypoint: negotiated api version: version=1
2022-05-02T22:40:48.949Z [INFO]  waypoint: bootstrapping the server
2022-05-02T22:40:48.964Z [INFO]  waypoint: bootstrapping complete
2022-05-02T22:40:48.964Z [INFO]  waypoint: reconnecting to the server with the bootstrap token
2022-05-02T22:40:48.964Z [TRACE] waypoint: no API client provided, initializing connection if possible
2022-05-02T22:40:48.964Z [INFO]  waypoint.server: attempting to source credentials and connect
2022-05-02T22:40:48.964Z [DEBUG] waypoint.serverclient: connection information: address=waypoint-server:9701 tls=true tls_skip_verify=true send_auth=true has_token=true
2022-05-02T22:40:49.024Z [DEBUG] waypoint.server: connection established with sourced credentials
2022-05-02T22:40:49.024Z [TRACE] waypoint: requesting version info from server
2022-05-02T22:40:49.027Z [INFO]  waypoint: server version info: version=v0.7.2 api_min=1 api_current=1 entrypoint_min=1 entrypoint_current=1
2022-05-02T22:40:49.027Z [INFO]  waypoint: negotiated api version: version=1
2022-05-02T22:40:49.027Z [INFO]  waypoint: setting server configuration
2022-05-02T22:40:49.038Z [INFO]  waypoint: generating login token for runner
2022-05-02T22:40:49.044Z [INFO]  waypoint: persisting root token: secret=waypoint-server-token
2022-05-02T22:40:49.085Z [INFO]  waypoint: persisting runner token: secret=waypoint-runner-token
2022-05-02T22:40:49.094Z [INFO]  waypoint: storing on-demand runner configuration for Kubernetes
2022-05-02T22:40:49.106Z [INFO]  waypoint: bootstrap complete
2022-05-02T22:40:49.107Z [TRACE] waypoint: stopping signal listeners and cancelling the context
runner logs after init 
2022-05-02T22:42:09.404Z [INFO]  waypoint: waypoint version: full_string="v0.7.2 (6e06e1a2d+CHANGES)" version=v0.7.2 prerelease="" metadata="" revision=6e06e1a2d+CHANGES
2022-05-02T22:42:09.405Z [TRACE] waypoint: starting interrupt listener for context cancellation
2022-05-02T22:42:09.406Z [DEBUG] waypoint: home configuration directory: path=/home/waypoint/.config/waypoint
2022-05-02T22:42:09.407Z [TRACE] waypoint: interrupt listener goroutine started
2022-05-02T22:42:09.407Z [TRACE] waypoint: no API client provided, initializing connection if possible
2022-05-02T22:42:09.407Z [INFO]  waypoint.server: attempting to source credentials and connect
2022-05-02T22:42:09.407Z [DEBUG] waypoint.serverclient: connection information: address=waypoint-server:9701 tls=true tls_skip_verify=true send_auth=true has_token=true
2022-05-02T22:42:09.471Z [DEBUG] waypoint.server: connection established with sourced credentials
2022-05-02T22:42:09.471Z [TRACE] waypoint: requesting version info from server
2022-05-02T22:42:09.487Z [INFO]  waypoint: server version info: version=v0.7.2 api_min=1 api_current=1 entrypoint_min=1 entrypoint_current=1
2022-05-02T22:42:09.487Z [INFO]  waypoint: negotiated api version: version=1
2022-05-02T22:42:09.487Z [INFO]  waypoint.runner.agent: sourcing credentials and connecting to the Waypoint server

» Runner configuration:
  Server address: waypoint-server:9701

» Runner logs:
Operating as a static Runner

2022-05-02T22:42:09.546Z [INFO]  waypoint.runner.agent: initializing the runner
2022-05-02T22:42:09.546Z [DEBUG] waypoint.runner.agent.runner: Created runner: id=01G23F0C0ARBEWVC0Z4Q7CHSA3
2022-05-02T22:42:09.546Z [INFO]  waypoint.runner.agent: starting runner: id=01G23F0C0ARBEWVC0Z4Q7CHSA3
2022-05-02T22:42:09.546Z [DEBUG] waypoint.runner.agent.runner: registering runner
2022-05-02T22:42:09.547Z [DEBUG] waypoint.runner.agent.runner: runner registered, waiting for first config processing
2022-05-02T22:42:09.551Z [INFO]  waypoint.runner.agent.runner.config_recv: new configuration received
2022-05-02T22:42:09.551Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: got source config update but ignoring since there is no diff
2022-05-02T22:42:09.551Z [DEBUG] waypoint.runner.agent.runner.watch_config.watchloop: new config variables received, scheduling refresh
2022-05-02T22:42:09.552Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: calculating changes between old and new config
2022-05-02T22:42:10.052Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: refreshing app configuration
2022-05-02T22:42:10.052Z [DEBUG] waypoint.runner.agent.runner.watch_config.watchloop: new configuration computed
2022-05-02T22:42:10.053Z [TRACE] waypoint.runner.agent.runner.watch_config: received new app config
2022-05-02T22:42:10.053Z [INFO]  waypoint.runner.agent.runner: runner registered with server and ready
2022-05-02T22:42:10.058Z [DEBUG] waypoint.runner.agent.runner: opening job stream
2022-05-02T22:42:10.058Z [TRACE] waypoint.runner.agent.runner: sending job request
2022-05-02T22:42:10.059Z [INFO]  waypoint.runner.agent.runner: waiting for job assignment
2022-05-02T22:42:25.053Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: refreshing app configuration
2022-05-02T22:42:25.053Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: app configuration unchanged
2022-05-02T22:42:40.054Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: refreshing app configuration
2022-05-02T22:42:40.054Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: app configuration unchanged
2022-05-02T22:42:55.054Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: refreshing app configuration
2022-05-02T22:42:55.055Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: app configuration unchanged
2022-05-02T22:43:10.055Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: refreshing app configuration
2022-05-02T22:43:10.055Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: app configuration unchanged
2022-05-02T22:43:25.055Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: refreshing app configuration
2022-05-02T22:43:25.056Z [TRACE] waypoint.runner.agent.runner.watch_config.watchloop: app configuration unchanged

k get secret waypoint-runner-token -o json | jq -rc '.data | map_values(@base64d)'
{"token":"BCk..."}

Seems like the server should generate those tokens regardless of whether it's been previously bootstrapped with valid config on disk so that the runner can be passed those values? At least for the runner.

sergiofteixeira commented 1 year ago

i still get this issue on EKS