Test/Debug story http proxy

[codefromthecrypt]

The minimum viable deployment of spark-streaming is likely

• an embedded http stream factory #23
• either a storage consumer (ex mysql) or a http consumer (ex reporter to a zipkin server)

In this case, the starter arch could be a single sparkstreaming process run in dev mode, pointed at a single in-memory zipkin server. you post spans to the spark job and view or download the json from the zipkin server,

[POST localhost:19411/api/v1/spans]
    -> [sparkstreaming adjustment happens]
        -> [POST localhost:9411/api/v1/spans] 

wdyt? cc @openzipkin/devops-tooling

[naoman]

either a storage consumer (ex mysql) or a http consumer (ex reporter to a zipkin server)

It think its a good idea to add zipkin server as a sparkstreaming consumer. I thought it's already part of zipkin-storage. Is there a reason we didn't built a storage module for zipkin server? or was it not needed before?

[codefromthecrypt]

It think its a good idea to add zipkin server as a sparkstreaming consumer. I thought it's already part of zipkin-storage https://github.com/openzipkin/zipkin/tree/master/zipkin-storage. Is there a reason we didn't built a storage module for zipkin server?

We have a different package for reporting to zipkin https://github.com/openzipkin/zipkin-reporter-java

or was it not needed before?

When we had a discussion before about an http "storage", it boiled down to a proxy use case. A normal http proxy is simpler as it requires no code, and is more flexible, too. The stackdriver storage module is http (well grpc), but that's because its storage layer only has an http api.

At any rate, I think it is fair game to put a Reporter consumer, as that would let us slap http or any other supported transport on the other side of it. For example, this could be a way to accomplish more powerful sampling, tee'ing off to vizceral, etc, as well the simple arch setup where this snaps into a well understood basic zipkin deployment.