pixie-io / pixie

Instant Kubernetes-Native Application Observability
https://px.dev
Apache License 2.0
5.61k stars 431 forks source link

Commands with `;` in them are rendered (wrongly) into multiple stack frames in the Pixie flamegraphs. #539

Open etep opened 2 years ago

etep commented 2 years ago

Description A command that has any ; characters in it is rendered into multiple stack frames instead of just one. After this happens, the colorization is also wrong. Most commands do not have this character in them, so we normally do not see this bug. However, we can easily construct an example by injecting the following (e.g.) into a yaml and deploying to k8s:

command: ["sh", "-c", "while $(curl --output /dev/null --silent --head --fail http://sample-app-svc); do sleep 0.02; done"]

Steps to reproduce

  1. Create an image of golang-sample-app (hereafter, referred to as the "go server app") from https://github.com/codefresh-contrib/golang-sample-app/. For convenience, there is an image here: gcr.io/pl-pixies/users/jps/go-sample-app-multi:latest.

  2. Create a pod for the go server app; you can use this yaml:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: sample-app
    labels:
    app: sample-app
    spec:
    replicas: 1
    selector:
    matchLabels:
      app: sample-app
    template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: sample-app
        image: gcr.io/pl-pixies/users/jps/go-sample-app-multi:latest
        ports:
        - containerPort: 8080
    ---
    apiVersion: v1
    kind: Service
    metadata:
    name: sample-app-svc
    spec:
    selector:
    app: sample-app
    ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  3. Create a pod for the workload, essentially a shell script that pings the server:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: sample-app-workload
    labels:
    app: sample-app-workload
    spec:
    replicas: 1
    selector:
    matchLabels:
      app: sample-app-workload
    template:
    metadata:
      labels:
        app: sample-app-workload
    spec:
      containers:
      - name: sample-app-workload
        image: curlimages/curl
        command: ["sh", "-c", "while $(curl --output /dev/null --silent --head --fail http://sample-app-svc); do sleep 0.02; done"]
  4. See error

Expected behavior We expect to see:

while $(curl --output /dev/null --silent --head --fail http://sample-app-svc); do sleep 0.02; done

rendered into a single stack frame and colorized in the darker blue hue.

Screenshots See, e.g. this flamegraph:

Screen Shot 2022-07-20 at 5 10 50 PM

Logs n/a

App information (please complete the following information):

aimichelle commented 2 years ago

Is there some way we can actually distinguish between a ; to mark a complete stackframe vs not? If not, we should probably update the underlying tables so that they use another character as a delimiter instead.