Document how to make a custom standalone collector

[virtuald]

This involves some spring magic and isn't particularly obvious on how to do it. I did it earlier today, but I'll have to look at it tomorrow to remember what exactly is needed. Here's what I remember so far:

POM modifications:

• Lots of copy/paste from zipkin-server POM
  • Need to add the correct zipkin autoconfigure dependencies to your POM (so for example, if you're writing to cassandra, add zipkin-autoconfigure-storage-cassandra... or just add all of them)
  • Add spring boot configuration parser to POM
  • Add sprint boot starter to POM

Creating the collector:

• Lots of copy/paste from the kafka collector
  • Two pieces: configuration and stream collector, it's fairly straightforward copy/pasting
  • Need to add a spring autoconfigure thing to your resources

Creating a main function to run:

• Create a main class, see the main zipkin-server class
  • This class should have the sampler @Bean in it
• If you want to use environment variables to configure the app like everything else in zipkin, then you need to copy/paste a bunch of stuff from zipkin-server.yml, and add that to your own app's resources

Finally, make sure that you don't mess up the annotation compilation, otherwise you'll get really weird errors and it won't work.

At some point, it might be useful to make a sample project with all of these.

Execution is however one would normally execute a spring boot application.

[codefromthecrypt]

fodder for examples, here's the only custom collector I'm aware of on github https://github.com/spring-cloud/spring-cloud-sleuth/tree/master/spring-cloud-sleuth-zipkin-stream

[codefromthecrypt]

another example, is a standalone server with no collector or UI components: https://github.com/openzipkin/zipkin/tree/master/zipkin-server/src/it/minimal-dependencies

[codefromthecrypt]

1233 would be a great example for this. an example of how to make an FQ collector kills two birds with one stone.

[codefromthecrypt]

Since last time we discussed how to deal with bootstrapping third parties and now, there's something brewing from @dsyer called https://github.com/dsyer/spring-boot-thin-launcher

The idea is that you could mix and match components this way, for example, adding the core zipkin server with elasticsearch storage and azure collector autoconfiguration deps.

A stable docker image could still be created by doing a "dry run" which just downloads the dependencies listed in the manifest during the build.

What do you all think about trying this? @aliostad @shakuzen @marcingrzejszczak @llinder @devinsba @anuraaga @kevinmdavis @denyska

[aliostad]

I am happy to follow the lead from you guys. As I have said, I am new to this.

Initially it seemed a stand-alone collector is cleaner but frankly since it has to have access to storage, it seems that needs zipkin-server (core bits) anyway. UI obviously is not necessary.

[llinder]

The thin launcher thing sounds interesting. Reading the docs it seems like it would be about the same amount of work as making a new fat jar. To the thin launchers advantage it sounds like it should avoid the extra bloat in the docker container since the thin wrapper could make informed decisions to only layer in new jars. Also it would hopefully maintain easy local testing/debugging like the fat jar provides.

[codefromthecrypt]

one subtle plus to the thin launcher is that it would not have the problem of composing fat jars (classpath duplication). Ex unless I misunderstand, the classpath would resolve to a single version on conflict, so you don't accidentally have mixed version of zipkin core jars.

[codefromthecrypt]

copied from #1488

Update: figured out how to get modularity working. It is a little constrained, but it works now.

Here's how.

Step 1: Add spring boot plugin to your autoconfig module, adding a "module" jar

The classifier name "module" is arbitrary, but I think it works. Take extreme care to not duplicate jars already in zipkin-server's exec jar. Here's an example configuration for the SQS collector:

  <dependencyManagement>
    <dependencies>
      <dependency>
        <!-- Import dependency management from Spring Boot -->
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-dependencies</artifactId>
        <version>${spring-boot.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <build>
    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>repackage</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <layout>MODULE</layout>
          <classifier>module</classifier>
          <!-- https://github.com/spring-projects/spring-boot/issues/3426 transitive exclude doesn't work -->
          <excludeGroupIds>io.zipkin.java,org.springframework.boot,org.springframework,commons-codec,com.fasterxml.jackson.core,com.fasterxml.jackson.dataformat,org.apache.httpcomponents,commons-logging,joda-time,software.amazon.ion</excludeGroupIds>
          <!-- already packaged in zipkin-server -->
          <excludeArtifactIds>aws-java-sdk-core,aws-java-sdk-sts,jmespath-java</excludeArtifactIds>
        </configuration>
      </plugin>
    </plugins>
  </build>

This will make a file like.. autoconfigure/collector-sqs/target/zipkin-autoconfigure-collector-sqs-0.0.4-SNAPSHOT-module.jar

Step 2: Extract this module jar when composing a server layer

I cannot get the PropertiesLauncher configuration to accept the module-jar directly. However, the following does work.

1. extract your module jar somewhere (Ex to an extensions folder)
2. explicitly start zipkin using PropertiesLauncher with loader.path set to include that directory

Example:

$ java  -Dloader.path=/tmp/extensions -cp zipkin.jar org.springframework.boot.loader.PropertiesLauncher --zipkin.collector.sqs.queue-url=http://foobar

Note: the module jar is self-contained.. when doing your devopsian things, feel free to just curl it from jcenter, like we do for the server jar.

[codefromthecrypt]

this works fine now. both in aws and azure. we just need to polish instructions