Go https requests not appearing in http_events

[tharinduwijewardane]

Describe the bug According to https://docs.px.dev/about-pixie/data-sources/#encryption-libraries tracing go TLS traffic is supported. But when a simple go client which calls an external https endpoint, is deployed and run, the request does not appear in the http_events table. If the url is changed to http, then it appears as usual.

To Reproduce

1. Write the following go program

   
   package main

import ( "fmt" "io/ioutil" "log" "net/http" )

func main() { http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {

    c := http.Client{}
    resp, err := c.Get("https://postman-echo.com/get?foo1=bar1")
    if err != nil {
        fmt.Printf("Error %s", err)
        return
    }
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Printf("Error %s", err)
        return
    }
    fmt.Printf("Body : %s", body)
})

log.Fatal(http.ListenAndServe(":9092", nil))

}

2. Build a docker image with it

ARG BUILDER_WORKSPACE="/workspace" FROM golang:1.15.14-alpine3.14 AS builder

ARG BUILDER_WORKSPACE WORKDIR ${BUILDER_WORKSPACE}

ADD service.go service.go ADD go.mod go.mod

RUN go build -o program service.go

FROM alpine:3

ARG BUILDER_WORKSPACE

COPY --from=builder ${BUILDER_WORKSPACE}/program ./

EXPOSE 9092

CMD ./program

3. Create a deployment and service in minikube cluster

apiVersion: apps/v1 kind: Deployment metadata: name: go-caller labels: app: go-caller spec: selector: matchLabels: app: go-caller replicas: 1 template: metadata: labels: app: go-caller spec: containers:

• name: go-caller-1 image: tempgocaller:echo ports:
  • name: go-caller-1 containerPort: 9092 imagePullPolicy: Never

---

apiVersion: v1 kind: Service metadata: name: go-caller spec: type: NodePort selector: app: go-caller ports:

• port: 9092 targetPort: go-caller-1

  4. Invoke the go service and observe http_events table with pod metadata

  import px

df = px.DataFrame(table='http_events', start_time='-10s')

df.pod = df.ctx['pod'] df.service = df.ctx['service']

px.display(df)

**Expected behavior**
`https://postman-echo.com/get?foo1=bar1&` request appears

**Actual behavior**
The request does not appear. It appears if the url is changed to http.

**App information (please complete the following information):**
- Pixie version : 0.7.9+Distribution.a47d77a.20220511024149.1
- K8s cluster version : Major:"1", Minor:"23"
- Node Kernel version
- Browser version

[tharinduwijewardane]

Additional observation: https requests are traced over http1. I added the following line at the start of main function to force http1. Then the https request appeared in the http_events table. os.Setenv("GODEBUG", "http2client=0")

[yzhao1012]

This is consistent with our finding that HTTP2 is used on the TLS connection. And Pixie does not support parsing http2 because of https://blog.px.dev/ebpf-http2-tracing/#when-does-wireshark-fail-to-decode-http2-headers.

But it seems we might want to bring back the http2 parsing.

This is another case where the uprobe-based http2 tracing falls short, because it seems the APIs used by http client to make http2 request does not use the same set of APIs as a native http2 client.

[yzhao1012]

So far only the official golang gRPC/HTTP2 (with and without TLS) is supported.

General HTTP2 is not supported in Pixie, all the following are not supported:

• HTTP2 with or without TLS
• HTTP-to-HTTP2 auto-upgrade with or without TLS

[htroisi]

@vihangm @yzhao1012 how should I change Pixie's documentation to make this more clear?

This is the existing table:

Possible update:

HTTP --> Supported HTTP2 --> Supported only for Golang gRPC/HTTP2 (with and without TLS). Golang apps must have debug information.