Receiving HTTP 1.1 400 response from agent

[tsloughter]

I've been trying to get a client for the agent working for the Erlang opencensus library. Oddly the client is able to publish to the collector if configured to point to it but when connecting to the agent and attempting to start the export stream I'm getting an HTTP 1.1 response:

400 Bad Request\r\nContent-Type: text/plain; charset=utf-8\r\nX-Content-Type-Options: 
nosniff\r\nDate: Thu, 07 Feb 2019 02:11:51 GMT\r\nContent-Length: 12\r\nConnection: 
close\r\n\r\nBad Request\n

Any idea how the collector and agent might be running differently?

[odeke-em]

Hello there @tsloughter!

To send observability signals to the agent you need to be speaking gRPC with Protobuf, or at least use HTTP/2.

To get you up and running, here are some resources:

Resource	URL
Go OCAgent exporter	https://github.com/census-ecosystem/opencensus-go-exporter-ocagent
OpenCensus.Metrics v1 Proto	https://github.com/census-instrumentation/opencensus-proto/tree/master/src/opencensus/proto/metrics/v1
OpenCensus.Trace v1 Proto	https://github.com/census-instrumentation/opencensus-proto/tree/master/src/opencensus/proto/trace/v1
Agent.Trace v1 Proto	https://github.com/census-instrumentation/opencensus-proto/tree/master/src/opencensus/proto/agent/trace/v1
Agent.Metrics v1 Proto	https://github.com/census-instrumentation/opencensus-proto/tree/master/src/opencensus/proto/agent/metrics/v1

Also another tip, to bootstrap your users later on, please implement the conversion from OpenCensus Stats--> OpenCensus Metrics Proto and OpenCensus Traces--> OpenCensus Trace Proto

Hope these references help!

[tsloughter]

@odeke-em Oh, sorry, should have been more clear :). I am using grpc protobufs, thats why the http 1.1 response is so odd.

The conversions are done as well, exporting the spans to the collector works and I tested the collector then exporting to jaeger.

[odeke-em]

What port on the agent are you connecting to?

[tsloughter]

55678

[odeke-em]

Alright, to try to isolate things for you, I've created for a mock agent that just runs a gRPC service and prints what it gets to stdout. If you encounter troubles with this too, then there is something else going on. The agent also has the web receiver which speaks HTTP/1.X btw.

Please run this code by the side by just go run main.go (please let me know if you don't have Go and I can compile for you a binary).

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "net"
    "os"
    "os/signal"

    "google.golang.org/grpc"

    agentmetricspb "github.com/census-instrumentation/opencensus-proto/gen-go/agent/metrics/v1"
    agenttracepb "github.com/census-instrumentation/opencensus-proto/gen-go/agent/trace/v1"
)

func main() {
    // Then create the echo-agents.
    srv := grpc.NewServer()
    agenttracepb.RegisterTraceServiceServer(srv, new(echoTraceAgent))
    agentmetricspb.RegisterMetricsServiceServer(srv, new(echoMetricsAgent))

    agentAddress := "localhost:55678"
    ln, err := net.Listen("tcp", agentAddress)
    if err != nil {
        log.Fatalf("Failed to serve the agent port: %v", err)
    }
    defer func() {
        srv.Stop()
        ln.Close()
    }()

    go func() {
        if err := srv.Serve(ln); err != nil {
            log.Fatalf("Failed to serve gRPC server: %v", err)
        }
    }()

    stopCh := make(chan os.Signal)
    signal.Notify(stopCh, os.Interrupt)
    log.Printf("Echo agent running at: %q", agentAddress)
    <-stopCh
}

type echoTraceAgent int

var _ agenttracepb.TraceServiceServer = (*echoTraceAgent)(nil)

func (eta *echoTraceAgent) Config(tscs agenttracepb.TraceService_ConfigServer) error {
    in, err := tscs.Recv()
    if err != nil {
        return err
    }
    if in == nil || in.Node == nil {
        return fmt.Errorf("the first message must contain the node identifier")
    }

    // Just for the sake of draining any configs
    // that the client-side exporter is still sending.
    for {
        blob, _ := json.MarshalIndent(in, "", "  ")
        log.Printf("Config:\n\033[33m%s\033[00m\n\n", blob)

        // Just send back this config.
        out := &agenttracepb.UpdatedLibraryConfig{
            Node:   in.Node,
            Config: in.Config,
        }

        if err := tscs.Send(out); err != nil {
            return err
        }
        in, err = tscs.Recv()
        if err != nil {
            return err
        }
    }
}

func (eta *echoTraceAgent) Export(tses agenttracepb.TraceService_ExportServer) error {
    in, err := tses.Recv()
    if err != nil {
        return err
    }

    // The first trace message should contain the node identifier.
    if in == nil || in.Node == nil {
        return fmt.Errorf("the first message must contain the node identifier")
    }

    // Now that we have the node identifier, let's start receiving spans.
    for {
        req, err := tses.Recv()
        if err != nil {
            return err
        }
        blob, _ := json.MarshalIndent(req, "", "  ")
        log.Printf("Trace.Export:\n\033[32m%s\033[00m\n\n", blob)
    }
}

type echoMetricsAgent int

var _ agentmetricspb.MetricsServiceServer = (*echoMetricsAgent)(nil)

func (ema *echoMetricsAgent) Export(mses agentmetricspb.MetricsService_ExportServer) error {
    in, err := mses.Recv()
    if err != nil {
        return err
    }

    // The first trace message should contain the node identifier.
    if in == nil || in.Node == nil {
        return fmt.Errorf("the first message must contain the node identifier")
    }

    // Now that we have the node identifier, let's start receiving spans.
    for {
        req, err := mses.Recv()
        if err != nil {
            return err
        }
        blob, _ := json.MarshalIndent(req, "", "  ")
        log.Printf("Metrics.Export:\n\033[33m%s\033[00m\n\n", blob)
    }
}

If the code above works alright, then the next isolation of the problem will point to the web-agent route.

[odeke-em]

@tsloughter please get the latest from master and recompile the agent.