All grpc request fails with status code 502

[sudhirkumar05]

Bug Report

Which version of the demo you are using? (please provide either a specific [9abfe56]

Symptom

Simulating workload using http works fine but grpc fails with 502 gateway timeout.

What is the expected behavior? It should have 200 response

What is the actual behavior?

What did you see instead?

Reproduce

1. A docker image which contains locust
2. A otel-collector already running in GKE cluster - end point is - http://otel-collector.platform-opentelemetry.svc.cluster.local:4317/v1/traces
3. Running locust with below config

   apiVersion: v1
   kind: Pod
   metadata:
   creationTimestamp: null
   labels:
   run: locust
   sidecar.istio.io/inject: "true"
   name: locust
   namespace: platform-opentelemetry
   spec:
   automountServiceAccountToken: true
   serviceAccount: otel-collector-sa
   containers:
   - image: abc-docker.artifactory.gcp.def/observability/otel/loadgenerator:0.1.0
   command: ["locust"]
   ports:
   - containerPort: 8089
   name: locust
   resources: {}
   env:
   - name: LOCUST_WEB_PORT
     value: "8089"
   - name: LOCUST_USERS
     value: "10"
   - name: LOCUST_HOST
     value: "frontend:8080"
   - name: LOCUST_WEB_HOST
     value: loadgenerator
   - name: LOCUST_HEADLESS
     value: "true"
   - name: LOCUST_AUTOSTART
     value: "true"
   - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
     value: "http://otel-collector.platform-opentelemetry.svc.cluster.local:4317/v1/traces"
   - name: OTEL_RESOURCE_ATTRIBUTES
     value: "service.namespace=platform-opentelemetry"
   - name: OTEL_SERVICE_NAME
     value: "otel-loadgenerator"
   - name: PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION
     value: "python"
   - name: OTEL_EXPORTER_OTLP_TRACES_INSECURE
     value: "true"
   args:
     - "-r 5"
     - "--headless"
     - "-u 10"
   dnsPolicy: ClusterFirst
   restartPolicy: Always
   status: {}

   Output

We will close this issue if:

• The steps you provided are complex.
• If we can not reproduce the behavior you're reporting.

Additional Context

Add any other context about the problem here.

[julianocosta89]

Hello @sudhirkumar05,

The loadgenerator was implemented to use the OTLP_HTTP Exporter: https://github.com/open-telemetry/opentelemetry-demo/blob/main/src/loadgenerator/requirements.txt#L22

In order to use the OTLP_GRPC Exporter the loadgenerator needs to be re-build with the correct exporter dependency: https://pypi.org/project/opentelemetry-exporter-otlp-proto-grpc/

[sudhirkumar05]

@julianocosta89 i am installing opentelemetry-exporter-otlp-proto-grpc==1.15.0 into my container and using this code in my locustfile.py from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

sample code

import json
# import random
# import uuid
import os
from locust import HttpUser, task, between

from opentelemetry import context, baggage, trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
#from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.urllib3 import URLLib3Instrumentor

tracer_provider = TracerProvider()
trace.set_tracer_provider(tracer_provider)
tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))

# Instrumenting manually to avoid error with locust gevent monkey
RequestsInstrumentor().instrument()
URLLib3Instrumentor().instrument()

products = [
    "0PUK6V6EV0",
    "1YMWWN1N4O",
    "2ZYFJ3GM2N",
    "66VCHSJNUP",
    "6E92ZMYYFZ",
    "9SIQT8TOJO",
    "L9ECAV7KIM",
    "LS4PSXUNUM",
    "OLJCESPC7Z",
    "HQTGWGPNH4",
]

people_file = open('people.json')
people = json.load(people_file)

class WebsiteUser(HttpUser):
    wait_time = between(1, 10)

    @task(1)
    def index(self):
        with open('otel_copy.json', 'r') as f:
            sample_trace = json.load(f)

        self.client.post(url=os.getenv('OTEL_EXPORTER_OTLP_TRACES_ENDPOINT'), json=sample_trace, headers={"content-type": "application/json"})

          def on_start(self):
        ctx = baggage.set_baggage("synthetic_request", "true")
        context.attach(ctx)
        self.index()

[julianocosta89]

What are you trying to achieve here?

with open('otel_copy.json', 'r') as f:
            sample_trace = json.load(f)

        self.client.post(url=os.getenv('OTEL_EXPORTER_OTLP_TRACES_ENDPOINT'), json=sample_trace, headers={"content-type": "application/json"})

          def on_start(self):

Why are you posting something to the OTEL_EXPORTER_OTLP_TRACES_ENDPOINT? Not fully understanding the use case here

[sudhirkumar05]

@julianocosta89 i am posting a sample json here. see below I see two transaction per user in the logs. One respond with 200 and and other respond with 415. I think one is instrumented automatically and the other one causes 415. Here is the collector is running in Google cloud run The original issues posted is from the collector running in GKE.

@task(1)
    def index(self):
        sample_otel = {
            "otel": {
                "name": "o11y",
                "luckyNumber": random.choice([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
            }
        }
        self.client.post(os.getenv('OTEL_EXPORTER_OTLP_TRACES_ENDPOINT'), json=sample_otel,
                         headers={"Content-Type": "application/json", "Accept": "application/json"})

[julianocosta89]

Lets take a step back before continuing here. What are you trying to achieve?

The loadgen was implemented to send load to the frontend, and this is working.

From your last reply, it seems that the Exporter is working properly, but for some reason you are trying to send something manually to the collector, why is that?

To get further assistance on how to send custom JSON to the collector I'd raise the question either on the Collector repo, or in the Python repo, as this is not the goal of this demo.

[sudhirkumar05]

The loadgen can also be used using --headless which means without frontend. Also, the loadgen is trying to push similar custom json and i am not creating a new custom json here. Check the locustfile.py inside loadgenerator/src

[julianocosta89]

All json requests sent via the loadgen contain the data required by the endpoint called:

• self.client.post("/api/cart", json=cart_item)
• self.client.post("/api/checkout", json=checkout_person)
• self.client.post("/api/checkout", json=checkout_person)

No json payload is being sent manually to the collector.

[sudhirkumar05]

the endpoint i am passing is /v1/traces and traces can be pushed to this end point.

[julianocosta89]

From your last reply, it seems that the Exporter is working properly, but for some reason you are trying to send something manually to the collector.

To get further assistance on how to send custom JSON to the collector I'd raise the question either on the Collector repo, or in the Python repo, as this is not the goal of this demo.