search

dylanmei / logspout-kafka

Logspout adapter for writing Docker container logs to Kafka topics
MIT License
52 stars 41 forks source link

Why doesn't this write json to kafka? #9

Open rocketraman opened 7 years ago

rocketraman commented 7 years ago

I'm not sure why this doesn't write json to kafka? We basically take our nice structured log output, and destructure it.

gavinmcnair commented 7 years ago

Agree completely. Looking into solutions for this now.

rocketraman commented 6 years ago

Any updates on this? Thanks!

gavinmcnair commented 6 years ago

My solution was ugly

Put it back into JSON.

https://github.com/gavinmcnair/logspout-kafka

OneCricketeer commented 6 years ago

Could just format the template to do that yourself 

KAFKA_TEMPLATE="{\"time\":\"{{.Time}}\",\"container_name\":\"{{.Container.Name}}\",\"source\":\"{{.Source}}\",\"data\":\"{{.Data}}\"}"

Or more simply 

KAFKA_TEMPLATE="{\"msg\":\"{{.Data}}\"}"
rocketraman commented 6 years ago

Could just format the template to do that yourself

This is quite likely to result in invalid JSON output, given that it does no escaping whatsoever of the data fields.

OneCricketeer commented 6 years ago

Probably a way to escape it. Not too familiar with Go templates.

https://golang.org/pkg/text/template/

rocketraman commented 6 years ago

Well, yes, I mean, that is the obvious workaround. The point of the issue, however, is why we need to do:

JSON -> destructured -> JSON

when we could just output the original JSON in the first place.

dylanmei commented 6 years ago

We seem to be struggling with the way Logspout is designed: applying templates to Docker log objects.

Interestingly, the stock raw adapter does have a toJSON helper to aid with extracting values from the object from within the template. https://github.com/gliderlabs/logspout#raw-format

OneCricketeer commented 6 years ago

If the message is already JSON, then I think you would just do KAFKA_TEMPLATE="{{.Data}}". Otherwise, if you need to get the other fields of the container, source, and time, then you need a workaround or use the raw formatting.