search

jenkins-x / jx

Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
https://jenkins-x.io/
Apache License 2.0
4.57k stars 788 forks source link

JX3 Logs are not getting stored under GCP storage #7601

Open tgelpi-bot opened 3 years ago

tgelpi-bot commented 3 years ago

Summary

JX3/GKE/GSM environment does not store logs under GCP storage even when logging is enabled.

Steps to reproduce the behavior

  1. Build an OOTB JX3/GKE/GSM using Terraform

  2. Validate that logging is enabled with Octant under config map terraform-jx-requirements (default namespace)

  3. Validate log entries under jx-requirements.yml

  4. Validate the pod jx-build-controller logs and entry: long term storage for logs is being used, bucket gs://logs-jXXXX

  5. Expected behavior

    Log files should reside under GCP buckets

    Actual behavior

    Log files do not reside under GCP buckets

Workaround steps to resolve

1, Delete the current pod jx-build-controller (another pod should automatically start)

  1. Check the log of the new jx-build-contoller pod to see the meessage: "long term storage for logs is being used, bucket gs://logs-XXXXX"
  2. Check the GCP storage under porta.

Jx version

The output of jx version is:

version: 3.1.76

Diagnostic information

The output of jx diagnose version is:

COPY OUTPUT HERE

Kubernetes cluster

Kubectl version

The output of kubectl version --client is:

Copy output here

Operating system / Environment

tgelpi-bot commented 3 years ago

This problem appears to be intermittent. I find that sometimes when creating a new environment the logs are configured. Most times I have to restart the jx-build-controller pod to set long term storage.

tgelpi-bot commented 3 years ago

Not sure if this issue was acknowledged. Each time a I build a new environment, I need to restart the jx-build-controller pod to set long term storage.

tomhobson commented 2 years ago

Hey @tgelpi-bot , is this still an issue you are seeing?

tgelpi-bot commented 2 years ago

Yes this issue is still outstanding.

I just built a new GKE environment recently and checking jx-build-controller pod logs I see:

{"timestamp":"2022-03-19T15:33:01.73282139Z","message":"long term storage for logs is not configured in cluster requirements","severity":"INFO","context":{}}

After deleting the pod when it restarts I now see in the logs:

{"timestamp":"2022-03-19T19:36:45.33661076Z","message":"long term storage for logs is being used, bucket gs://logs-jx3ggg-490a8a050b54","severity":"INFO","context":{}}