search

kruize / autotune

Autonomous Performance Tuning for Kubernetes!
Apache License 2.0
165 stars 54 forks source link

Error: Index -1 out of bounds for length 0 observed while generating recommendations #1338

Open kusumachalasani opened 1 month ago

kusumachalasani commented 1 month ago

Describe the bug

ERROR [qtp1100399456-43][RecommendationEngine.java(299)]-Failed to create recommendation for experiment: monitor_gpu and interval_start_time: null and interval_end_time: 2024-10-09 12:27:22.0 2024-10-0912:27:31.003 ERROR [qtp1100399456-43][KruizeErrorHandler.java(71)]-{"message":"Index -1 out of bounds for length 0","httpcode":500,"documentationLink":"","status":"ERROR"}

How to reproduce it On NERC cluster, create an experiment using a following json.

[{
  "version": "v2.0",
  "experiment_name": "monitor_gpu",
  "cluster_name": "default",
  "performance_profile": "resource-optimization-local-monitoring",
  "mode": "monitor",
  "target_cluster": "local",
  "datasource": "prometheus-1",
  "kubernetes_objects": [
    {
      "type": "statefulset",
      "name": "human-eval-benchmark",
      "namespace": "unpartitioned-namespace",
      "containers": [
        {
          "container_image_name": "kruizehub/human-eval",
          "container_name": "human-eval-benchmark"
        }
      ]
    }
  ],
  "trial_settings": {
    "measurement_duration": "15min"
  },
  "recommendation_settings": {
    "threshold": "0.1"
  }
}]

While generating the recommendations for that experiment using the below curl command, following error is observed. curl -X POST http://kruize-openshift-tuning.apps.nerc-ocp-test-2.nerc.mghpcc.org/generateRecommendations?experiment_name=monitor_gpu

ERROR [qtp1100399456-43][RecommendationEngine.java(299)]-Failed to create recommendation for experiment: monitor_gpu and interval_start_time: null and interval_end_time: 2024-10-09 12:27:22.0 2024-10-0912:27:31.003 ERROR [qtp1100399456-43][KruizeErrorHandler.java(71)]-{"message":"Index -1 out of bounds for length 0","httpcode":500,"documentationLink":"","status":"ERROR"}

Expected behavior Recommendations should be generated without the error.

Relevant logs From kruize logs 

ERROR [qtp1100399456-43][RecommendationEngine.java(299)]-Failed to create recommendation for experiment: monitor_gpu and interval_start_time: null and interval_end_time: 2024-10-09 12:27:22.0
2024-10-0912:27:31.003 ERROR [qtp1100399456-43][KruizeErrorHandler.java(71)]-{"message":"Index -1 out of bounds for length 0","httpcode":500,"documentationLink":"","status":"ERROR"}

Environment: