search

grafana / promql-anomaly-detection

A framework for anomaly detection using Prometheus and PromQL
Apache License 2.0
71 stars 2 forks source link

Demo doesn't work #2

Open brankomijuskovic opened 2 days ago

brankomijuskovic commented 2 days ago

After make start, frontend-proxy is stuck in restarting state. 

# docker ps
CONTAINER ID   IMAGE                                                      COMMAND                  CREATED          STATUS                          PORTS                                                                                                           NAMES
7ddee60bbf9f   ghcr.io/open-telemetry/demo:latest-frontendproxy           "/bin/sh -c 'envsubs…"   25 minutes ago   Restarting (1) 40 seconds ago                                                                                                                   frontend-proxy
a8a0ddc866c4   ghcr.io/open-telemetry/demo:latest-loadgenerator           "/bin/sh -c 'locust …"   25 minutes ago   Up 24 minutes                   0.0.0.0:32867->8089/tcp, [::]:32867->8089/tcp                                                                   load-generator
7b6413195051   ghcr.io/open-telemetry/demo:latest-frontend                "/bin/sh -c 'npm sta…"   25 minutes ago   Up 24 minutes                   0.0.0.0:32866->8080/tcp, [::]:32866->8080/tcp                                                                   frontend
81d3dbb1878f   ghcr.io/open-telemetry/demo:latest-checkoutservice         "./checkoutservice"      25 minutes ago   Up 24 minutes                   0.0.0.0:32865->5050/tcp, [::]:32865->5050/tcp                                                                   checkout-service
edb691c6f1f0   ghcr.io/open-telemetry/demo:latest-recommendationservice   "opentelemetry-instr…"   25 minutes ago   Up 24 minutes                   0.0.0.0:32864->9001/tcp, [::]:32864->9001/tcp                                                                   recommendation-service
71bb6ec0c2ba   ghcr.io/open-telemetry/demo:latest-productcatalogservice   "./productcatalogser…"   25 minutes ago   Up 24 minutes                   0.0.0.0:32861->3550/tcp, [::]:32861->3550/tcp                                                                   product-catalog-service
82fcf1567c42   ghcr.io/open-telemetry/demo:latest-shippingservice         "/app/shippingservice"   25 minutes ago   Up 24 minutes                   0.0.0.0:32856->50050/tcp, [::]:32856->50050/tcp                                                                 shipping-service
4a2a2a8dafc8   ghcr.io/open-telemetry/demo:latest-imageprovider           "/docker-entrypoint.…"   25 minutes ago   Up 24 minutes                   80/tcp, 0.0.0.0:32859->8081/tcp, [::]:32859->8081/tcp                                                           imageprovider
afb82db373ad   ghcr.io/open-telemetry/demo:latest-cartservice             "./cartservice"          25 minutes ago   Up 24 minutes                   0.0.0.0:32862->7070/tcp, [::]:32862->7070/tcp                                                                   cart-service
c76096774bc4   ghcr.io/open-telemetry/demo:latest-quoteservice            "docker-php-entrypoi…"   25 minutes ago   Up 24 minutes                   0.0.0.0:32858->8090/tcp, [::]:32858->8090/tcp                                                                   quote-service
6cfded17bc66   ghcr.io/open-telemetry/demo:latest-currencyservice         "/bin/sh -c './usr/l…"   25 minutes ago   Up 24 minutes                   0.0.0.0:32863->7001/tcp, [::]:32863->7001/tcp                                                                   currency-service
51a3e6d61da8   ghcr.io/open-telemetry/demo:latest-adservice               "./build/install/ope…"   25 minutes ago   Up 24 minutes                   0.0.0.0:32857->9555/tcp, [::]:32857->9555/tcp                                                                   ad-service
571a6a4f2a55   ghcr.io/open-telemetry/demo:latest-emailservice            "bundle exec ruby em…"   25 minutes ago   Up 24 minutes                   0.0.0.0:32855->6060/tcp, [::]:32855->6060/tcp                                                                   email-service
f3da3f18826c   ghcr.io/open-telemetry/demo:latest-paymentservice          "npm run start"          25 minutes ago   Up 24 minutes                   0.0.0.0:32860->50051/tcp, [::]:32860->50051/tcp                                                                 payment-service
7b7aba80d198   otel/opentelemetry-collector-contrib:0.108.0               "/otelcol-contrib --…"   25 minutes ago   Up 25 minutes                   55678-55679/tcp, 0.0.0.0:32852->4317/tcp, [::]:32852->4317/tcp, 0.0.0.0:32853->4318/tcp, [::]:32853->4318/tcp   otel-col
d78cf4bcc658   ghcr.io/open-feature/flagd:v0.11.2                         "/flagd-build start …"   25 minutes ago   Up 25 minutes                   0.0.0.0:32851->8013/tcp, [::]:32851->8013/tcp                                                                   flagd
9668f8629f89   valkey/valkey:8.0-alpine                                   "docker-entrypoint.s…"   25 minutes ago   Up 25 minutes                   0.0.0.0:32854->6379/tcp, [::]:32854->6379/tcp                                                                   valkey-cart
4aa59b0a69b0   grafana/grafana:11.2.0                                     "/run.sh"                25 minutes ago   Up 25 minutes                   0.0.0.0:32848->3000/tcp, [::]:32848->3000/tcp                                                                   grafana
129401864a0e   prom/node-exporter:latest                                  "/bin/node_exporter …"   25 minutes ago   Up 25 minutes                   0.0.0.0:32850->9100/tcp, [::]:32850->9100/tcp                                                                   node-exporter
0be9bd56de52   quay.io/prometheus/prometheus:v2.54.1                      "/bin/prometheus --w…"   25 minutes ago   Up 25 minutes                   0.0.0.0:32849->9090/tcp, [::]:32849->9090/tcp                                                                   prometheus

The issue is with missing address and port for Jaeger.

...
  clusters {
    name: "jaeger"
    type: STRICT_DNS
    load_assignment {
      cluster_name: "jaeger"
      endpoints {
        lb_endpoints {
          endpoint {
            address {
              socket_address {
              }
            }
...

Full logs:

[2024-10-08 12:33:45.698][8][critical][main] [source/server/server.cc:414] error initializing config '  envoy.yaml': Proto constraint validation failed (BootstrapValidationError.StaticResources: embedded message failed validation | caused by StaticResourcesValidationError.Clusters[7]: embedded message failed validation | caused by ClusterValidationError.LoadAssignment: embedded message failed validation | caused by ClusterLoadAssignmentValidationError.Endpoints[0]: embedded message failed validation | caused by LocalityLbEndpointsValidationError.LbEndpoints[0]: embedded message failed validation | caused by LbEndpointValidationError.Endpoint: embedded message failed validation | caused by EndpointValidationError.Address: embedded message failed validation | caused by AddressValidationError.SocketAddress: embedded message failed validation | caused by SocketAddressValidationError.Address: value length must be at least 1 characters): static_resources {
  listeners {
    address {
      socket_address {
        address: "0.0.0.0"
        port_value: 8080
      }
    }
    filter_chains {
      filters {
        name: "envoy.filters.network.http_connection_manager"
        typed_config {
          [type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager] {
            stat_prefix: "ingress_http"
            route_config {
              name: "local_route"
              virtual_hosts {
                name: "frontend"
                domains: "*"
                routes {
                  match {
                    prefix: "/loadgen"
                  }
                  route {
                    cluster: "loadgen"
                    prefix_rewrite: "/"
                  }
                }
                routes {
                  match {
                    prefix: "/otlp-http/"
                  }
                  route {
                    cluster: "opentelemetry_collector_http"
                    prefix_rewrite: "/"
                  }
                }
                routes {
                  match {
                    prefix: "/jaeger"
                  }
                  route {
                    cluster: "jaeger"
                  }
                }
                routes {
                  match {
                    prefix: "/grafana"
                  }
                  route {
                    cluster: "grafana"
                  }
                }
                routes {
                  match {
                    prefix: "/images/"
                  }
                  route {
                    cluster: "imageprovider"
                    prefix_rewrite: "/"
                  }
                }
                routes {
                  match {
                    prefix: "/flagservice/"
                  }
                  route {
                    cluster: "flagservice"
                    prefix_rewrite: "/"
                    timeout {
                    }
                  }
                }
                routes {
                  match {
                    prefix: "/"
                  }
                  route {
                    cluster: "frontend"
                  }
                }
              }
            }
            http_filters {
              name: "envoy.filters.http.fault"
              typed_config {
                [type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault] {
                  delay {
                    percentage {
                      numerator: 100
                    }
                    header_delay {
                    }
                  }
                  max_active_faults {
                    value: 100
                  }
                }
              }
            }
            http_filters {
              name: "envoy.filters.http.router"
              typed_config {
                [type.googleapis.com/envoy.extensions.filters.http.router.v3.Router] {
                }
              }
            }
            tracing {
              provider {
                name: "envoy.tracers.opentelemetry"
                typed_config {
                  [type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig] {
                    grpc_service {
                      envoy_grpc {
                        cluster_name: "opentelemetry_collector_grpc"
                      }
                      timeout {
                        nanos: 250000000
                      }
                    }
                    service_name: "frontend-proxy"
                    resource_detectors {
                      name: "envoy.tracers.opentelemetry.resource_detectors.environment"
                      typed_config {
                        [type.googleapis.com/envoy.extensions.tracers.opentelemetry.resource_detectors.v3.EnvironmentResourceDetectorConfig] {
                        }
                      }
                    }
                  }
                }
              }
              spawn_upstream_span {
                value: true
              }
            }
          }
        }
      }
    }
  }
  clusters {
    name: "opentelemetry_collector_grpc"
    type: STRICT_DNS
    load_assignment {
      cluster_name: "opentelemetry_collector_grpc"
      endpoints {
        lb_endpoints {
          endpoint {
            address {
              socket_address {
                address: "otelcol"
                port_value: 4317
              }
            }
          }
        }
      }
    }
    typed_extension_protocol_options {
      key: "envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
      value {
        [type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions] {
          explicit_http_config {
            http2_protocol_options {
            }
          }
        }
      }
    }
  }
  clusters {
    name: "opentelemetry_collector_http"
    type: STRICT_DNS
    load_assignment {
      cluster_name: "opentelemetry_collector_http"
      endpoints {
        lb_endpoints {
          endpoint {
            address {
              socket_address {
                address: "otelcol"
                port_value: 4318
              }
            }
          }
        }
      }
    }
  }
  clusters {
    name: "frontend"
    type: STRICT_DNS
    load_assignment {
      cluster_name: "frontend"
      endpoints {
        lb_endpoints {
          endpoint {
            address {
              socket_address {
                address: "frontend"
                port_value: 8080
              }
            }
          }
        }
      }
    }
  }
  clusters {
    name: "imageprovider"
    type: STRICT_DNS
    load_assignment {
      cluster_name: "imageprovider"
      endpoints {
        lb_endpoints {
          endpoint {
            address {
              socket_address {
                address: "imageprovider"
                port_value: 8081
              }
            }
          }
        }
      }
    }
  }
  clusters {
    name: "flagservice"
    type: STRICT_DNS
    load_assignment {
      cluster_name: "flagservice"
      endpoints {
        lb_endpoints {
          endpoint {
            address {
              socket_address {
                address: "flagd"
                port_value: 8013
              }
            }
          }
        }
      }
    }
  }
  clusters {
    name: "loadgen"
    type: STRICT_DNS
    load_assignment {
      cluster_name: "loadgen"
      endpoints {
        lb_endpoints {
          endpoint {
            address {
              socket_address {
                address: "loadgenerator"
                port_value: 8089
              }
            }
          }
        }
      }
    }
  }
  clusters {
    name: "grafana"
    type: STRICT_DNS
    load_assignment {
      cluster_name: "grafana"
      endpoints {
        lb_endpoints {
          endpoint {
            address {
              socket_address {
                address: "grafana"
                port_value: 3000
              }
            }
          }
        }
      }
    }
  }
  clusters {
    name: "jaeger"
    type: STRICT_DNS
    load_assignment {
      cluster_name: "jaeger"
      endpoints {
        lb_endpoints {
          endpoint {
            address {
              socket_address {
              }
            }
          }
        }
      }
    }
  }
}
admin {
  address {
    socket_address {
      address: "0.0.0.0"
      port_value: 10000
    }
  }
}
layered_runtime {
  layers {
    name: "static_layer_0"
    static_layer {
      fields {
        key: "envoy"
        value {
          struct_value {
            fields {
              key: "resource_limits"
              value {
                struct_value {
                  fields {
                    key: "listener"
                    value {
                      struct_value {
                        fields {
                          key: "example_listener_name"
                          value {
                            struct_value {
                              fields {
                                key: "connection_limit"
                                value {
                                  number_value: 10000
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
kallayj commented 1 day ago

+1 for this issue

bunkrur commented 1 day ago

https://github.com/bunkrur/promql-anomaly-detection/commits/fix-demo/

this fork is working - you just need to run docker compose up --build to get it to work after these changes.

jcreixell commented 17 hours ago

@kallayj @brankomijuskovic I just merged #3 and #4 which should solve most issues, could you please pull from main and let me know if it works for you?