codec doesn't seem to do anything.

[lesaux]

I'm using this plugin to output transformed log lines to GCS.

My goal is to be able to import those files into BQ at a later time - (streaming to BQ is $$$ right now).

When I use

output_format => "json"

I run into a problem because BQ doesn't like fields starting with @ (i.e. @timestamp)

So now I'm trying to use

output_format => "plain"

But the default format

@timestamp %{host} %{message}

doesn't suit me. Therefore I am adding a codec.

codec => plain  { format =>  "%{[myfield]}" }

but it doesn't seem to do anything.

Would any one have recommendations on how I could achieve this?

[pdehlke]

+1

Same issue here. Anybody listening to issues on this repo?

[josephlewis42]

This plugin isn't one of them that supports codecs. Until the feature is added, can you try using something like the mutate plugin in the Logstash pipeline to remove the fields with @ in them?

[josephlewis42]

Migration plan: Currently, the plugin supports json and plain output_formats. By default, the output format is plain.

Step 1

Mark output_format as :deprecated with the note that users should use :codec moving forward. Set the default output_format to be nil.

If the user has set a non-nil codec in their configuration, continue using the existing logic for conversion.

Step 2

Mark output_format as obsolete with the next major version bump with info about what to use when upgrading; this will prevent inadvertent regressions.

[josephlewis42]

I have a fix in for this, let me know if anyone still has desire to test it and I'll push a gemfile to github.

[nandanrao]

I'd be keen to test it! Would be very helpful.

[colinsurprenant]

codec support has been added in #42 and v4.1.0 has been published with that. Closing.