search

kyma-project / kyma-environment-broker

Kyma Environment Broker
Apache License 2.0
0 stars 22 forks source link

Timestamps inconsistent across sources (kibana, logs explorer, events table) #497

Closed jaroslaw-pieszka closed 7 months ago

jaroslaw-pieszka commented 1 year ago

While investigating one can use multiple sources of information but it is very hard to order log messages and events (from events table chronologically, since timestamps differ in value and accuracy.

Execution of Initialization step for instance_id: 096B7C10-78F4-4635-B6EE-F200502CAE89 message: "create provisioner input creator for "4deee563-e5ec-4731-b9b1-53b42d855f0c" plan ID"

For second (possibly parallel) execution of Initialization step:

Events:

  {
    "ID": "a0eb3768-d9b6-4d9d-97ef-4214db852f41",
    "Level": "info",
    "InstanceID": "096B7C10-78F4-4635-B6EE-F200502CAE89",
    "OperationID": "df044c63-f3d1-4238-8984-90ba4a04a4ba",
    "Message": "processing step: Provision_Initialization",
    "CreatedAt": "2023-04-03T13:17:27.225448Z"
  },
  {
    "ID": "f7ae3796-e149-4af8-9cf4-367e90c43bf0",
    "Level": "info",
    "InstanceID": "096B7C10-78F4-4635-B6EE-F200502CAE89",
    "OperationID": "df044c63-f3d1-4238-8984-90ba4a04a4ba",
    "Message": "processing step: Provision_Initialization",
    "CreatedAt": "2023-04-03T13:17:27.403207Z"
  },

Issues:

kyma-bot commented 1 year ago

This issue or PR has been automatically marked as stale due to the lack of recent activity. Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

You can:

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale

kyma-bot commented 1 year ago

This issue or PR has been automatically closed due to the lack of activity. Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

You can:

If you think that I work incorrectly, kindly raise an issue with the problem.

/close

kyma-bot commented 1 year ago

@kyma-bot: Closing this issue.

In response to [this](https://github.com/kyma-project/kyma-environment-broker/issues/497): >This issue or PR has been automatically closed due to the lack of activity. >Thank you for your contributions. > >This bot triages issues and PRs according to the following rules: >- After 60d of inactivity, `lifecycle/stale` is applied >- After 7d of inactivity since `lifecycle/stale` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle stale` > >If you think that I work incorrectly, kindly [raise an issue](https://github.com/kyma-project/test-infra/issues/new/choose) with the problem. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
kyma-bot commented 1 year ago

This issue or PR has been automatically marked as stale due to the lack of recent activity. Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

You can:

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale

kyma-bot commented 1 year ago

This issue or PR has been automatically closed due to the lack of activity. Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

You can:

If you think that I work incorrectly, kindly raise an issue with the problem.

/close

kyma-bot commented 1 year ago

@kyma-bot: Closing this issue.

In response to [this](https://github.com/kyma-project/kyma-environment-broker/issues/497): >This issue or PR has been automatically closed due to the lack of activity. >Thank you for your contributions. > >This bot triages issues and PRs according to the following rules: >- After 60d of inactivity, `lifecycle/stale` is applied >- After 7d of inactivity since `lifecycle/stale` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle stale` > >If you think that I work incorrectly, kindly [raise an issue](https://github.com/kyma-project/test-infra/issues/new/choose) with the problem. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
ralikio commented 10 months ago

Hint: contact SRE to customise log scraping configuration and produce timestamps that are not rounded.

szwedm commented 9 months ago

Issue should be recreated in KEB repository: https://github.com/kyma-project/kyma-environment-broker. Closing.

ralikio commented 8 months ago

Draft PR: https://github.com/kyma-project/kyma-environment-broker/pull/597

ralikio commented 8 months ago

Timestamps corrected. Will be applied in the next release.

ralikio commented 7 months ago

The PR https://github.com/kyma-project/kyma-environment-broker/pull/597 added timestamps in main logrus logger in cmd/broker/main.go. The following snippet shows usages of log libraries across all files in kyma-environment-broker repository:

github.com/sirupsen/logrus

cmd/accountcleanup/main.go cmd/archiver/main.go cmd/broker/deprovisioning.go cmd/broker/main.go cmd/broker/provisioning.go cmd/broker/update.go cmd/broker/upgrade_cluster.go cmd/broker/upgrade_kyma.go cmd/deprovisionretrigger/main.go cmd/expirator/main.go cmd/runtimereconciler/main.go cmd/trialcleanup/main.go common/deprovision/client.go common/hyperscaler/azure/client.go common/hyperscaler/azure/provider.go common/hyperscaler/azure/testing/testing.go common/orchestration/resolver.go common/orchestration/strategies/parallel.go common/setup/builder.go common/setup/error_handlers.go internal/model.go internal/avs/client.go internal/avs/delegator.go internal/avs/evaluation_manager.go internal/broker/bind_create.go internal/broker/bind_delete.go internal/broker/bind_get.go internal/broker/bind_last_operation.go internal/broker/client.go internal/broker/instance_create.go internal/broker/instance_deprovision.go internal/broker/instance_get.go internal/broker/instance_last_operation.go internal/broker/instance_update.go internal/broker/services.go internal/btpmanager/credentials/job.go internal/btpmanager/credentials/manager.go internal/cis/account_cleanup.go internal/cis/client_ver_1.go internal/cis/client.go internal/cis/e2e/tests_utils.go internal/config/reader.go internal/edp/client.go internal/environmentscleanup/service.go internal/event/pubsub.go internal/expiration/handler.go internal/health/server.go internal/httputil/response.go internal/httputil/write_response.go internal/kubeconfig/handler.go internal/logger/spy_logger.go internal/metrics/instances_collector.go internal/metrics/metrics.go internal/metrics/operations_collector.go internal/metrics/operations_db_collector.go internal/metricsv2/instances_collector.go internal/metricsv2/metrics.go internal/metricsv2/operation_duration.go internal/metricsv2/operations_result.go internal/metricsv2/operations_stats.go internal/orchestration/runtime_lister.go internal/orchestration/handlers/cancel.go internal/orchestration/handlers/cluster_handler.go internal/orchestration/handlers/cluster_retry.go internal/orchestration/handlers/handlers.go internal/orchestration/handlers/kyma_handler.go internal/orchestration/handlers/kyma_retry.go internal/orchestration/handlers/orchestration_handler.go internal/orchestration/manager/manager.go internal/orchestration/manager/upgrade_cluster.go internal/orchestration/manager/upgrade_kyma.go internal/process/operation_manager.go internal/process/queue.go nternal/process/staged_manager.go internal/process/upgrade_cluster_operation.go internal/process/upgrade_kyma_operation.go internal/process/deprovisioning/archiving_step.go internal/process/deprovisioning/avs_evaluations.go internal/process/deprovisioning/btp_operator_cleanup.go internal/process/deprovisioning/check_cluster_deregistration.go internal/process/deprovisioning/check_gardenercluster_deleted_step.go internal/process/deprovisioning/check_kyma_resource_deleted_step.go internal/process/deprovisioning/check_runtime_removal_step.go internal/process/deprovisioning/clean_step.go internal/process/deprovisioning/delete_gardener_cluster.go internal/process/deprovisioning/delete_kyma_resource_step.go internal/process/deprovisioning/deregister_cluster.go internal/process/deprovisioning/edp_deregistration.go internal/process/deprovisioning/ias_deregistration.go internal/process/deprovisioning/init.go internal/process/deprovisioning/release_subscription_step.go internal/process/deprovisioning/remove_instance_step.go internal/process/deprovisioning/remove_runtime.go internal/process/deprovisioning/utils.go internal/process/deprovisioning/automock/step.go internal/process/provisioning/apply_kyma_step.go internal/process/provisioning/btp_operator_overrides.go internal/process/provisioning/check_cluster_configuration.go internal/process/provisioning/check_runtime_step.go internal/process/provisioning/create_cluster_configuration.go internal/process/provisioning/create_runtime_for_own_cluster_step.go internal/process/provisioning/create_runtime_without_kyma_step.go internal/process/provisioning/edp_registration.go internal/process/provisioning/enable_for_trial_step.go internal/process/provisioning/external_eval_step.go internal/process/provisioning/external_eval.go internal/process/provisioning/get_kubeconfig_step.go internal/process/provisioning/initialisation.go internal/process/provisioning/inject_btp_operator_credentials_step.go internal/process/provisioning/internal_eval.go internal/process/provisioning/override_kyma_modules.go internal/process/provisioning/resolve_creds.go internal/process/provisioning/secret_cm_overrides.go internal/process/provisioning/skip_for_trial_step.go internal/process/provisioning/start_step.go internal/process/provisioning/automock/step.go internal/process/steps/gardener_cluster.go internal/process/steps/kyma_template.go internal/process/steps/lifecycle_manager_kubeconfig.go internal/process/update/apply_reconciler_configuration.go internal/process/update/btp_operator_overrides.go internal/process/update/check_step.go internal/process/update/init_kyma_step.go internal/process/update/initialisation_step.go internal/process/update/sc_migration_check_step.go internal/process/update/upgrade_shoot_step.go internal/process/upgrade_cluster/initialisation.go internal/process/upgrade_cluster/log_skipping_upgrade.go internal/process/upgrade_cluster/manager.go internal/process/upgrade_cluster/send_notification.go internal/process/upgrade_cluster/upgrade_cluster_step.go internal/process/upgrade_kyma/apply_cluster_configuration.go internal/process/upgrade_kyma/apply_kyma_step.go internal/process/upgrade_kyma/avs_maintenance.go internal/process/upgrade_kyma/btp_operator_overrides.go internal/process/upgrade_kyma/check_cluster_configuration.go internal/process/upgrade_kyma/initialisation.go internal/process/upgrade_kyma/manager.go internal/process/upgrade_kyma/secret_cm_overrides.go internal/process/upgrade_kyma/send_notification.go internal/process/upgrade_kyma/skip_for_trial_step.go internal/process/upgrade_kyma/automock/step.go internal/provisioner/graphqlizer.go internal/reconciler/client.go internal/runtimeversion/account_mapping.go internal/storage/storage_handler.go internal/storage/storage.go internal/storage/driver/postsql/instance.go internal/storage/driver/postsql/operation.go internal/storage/driver/postsql/orchestration.go internal/storage/driver/postsql/runtime_state.go internal/storage/driver/postsql/subaccount_state.go internal/storage/driver/postsql/events/events.go internal/storage/postsql/init.go internal/suspension/handler.go utils/edp-registrator/main.go

log/slog

cmd/archiver/main.go cmd/expirator/main.go cmd/subaccountsync/main.go internal/archive/service.go internal/kymacustomresource/kyma_cr_updater.go internal/syncqueues/priority_queue.go

log

cmd/broker/main.go cmd/runtimereconciler/main.go internal/docker_helper.go internal/cis/fake_server.go internal/process/staged_manager.go internal/storage/storage.go

ralikio commented 7 months ago

Examples of logs that were not corrected after the https://github.com/kyma-project/kyma-environment-broker/pull/597 PR

time="2024-04-10T06:31:50Z" level=warning msg="operation update conflict, operation ID:..." component="postsql/operation.go:1450:postsql.(*operations).update.func1"
➜  kyma-environment-broker git:(main) ✗ k logs deployments/kcp-kyma-environment-broker -n kcp-system | grep -v '^{'
compass-runtime-agent
...
time="2024-04-10T06:31:50Z" level=warning msg="operation update conflict, operation ID: ..." component="postsql/operation.go:1450:postsql.(*operations).update.func1"
...