Formatting incoming data for publishing

[marcintao]

What format should be applied to published data?

Having looked at a few publishers it struck me that publishers don't yet have a common way to handle incoming metric data types. I believe we should guide developers in the choice of proper conversion strategy.

Here's breakdown of publishers by the data conversion approach:

Approach	Second Header	Remarks
Specialized	snap-plugin-publisher-heka	Delivery via protobuf, reference
Json serialization	snap-plugin-publisher-rabbitmq	call to json.Marshal(), reference
"	snap-plugin-publisher-kairosdb	", reference
"	snap-plugin-publisher-kafka	", reference
"	snap-plugin-publisher-file	", reference
No conversion	snap-plugin-publisher-opentsdb	Data passed as is, reference
"	snap-plugin-publisher-influxdb	", reference
Shallow string conversion	snap-plugin-publisher-graphite	Call to Sprintf, reference
"	snap-plugin-publisher-etcd	", reference
Custom Routine	snap-plugin-publisher-postgresql	Routine handling a handful of cases (each type individually), no support for structured types, reference
"	snap-plugin-publisher-mysql	", reference
"	snap-plugin-publisher-hana	", reference

I understand that some product-specific publishers will need a specific handling (ie: heka). But for other cases we might propose a selection guide.

[mbbroberg]

This is an awesome proposal. Hey @jcooklin and other @intelsdi-x/snap-maintainers, I'd like to get a discussion on this one.

[IRCody]

Having looked at a few publishers it struck me that publishers don't yet have a common way to handle incoming metric data types. I believe we should guide developers in the choice of proper conversion strategy.

It seems like you aren't referring to incoming data types but outgoing. i.e What format should data be going out of the publisher. This seems like it would necessarily depend on where you are publishing to.