manage templates per source ip

[robcowart]

This PR adds the necessary changes to allow Netflow and IPFIX templates to be managed per source IP (i.e. per device sending flow records).

In order for this PR to work it requires that metadata be passed to the codec by the UDP input, which is pending merging of this PR (https://github.com/logstash-plugins/logstash-input-udp/pull/46).

In the event that metadata is not provided by the UDP input, the behavior falls back to the legacy handling of templates. This will prevent any "surprises" if used with the current or older UDP input, or if the metadata => true option is not set for the input.

All testing was performed for both Netflow v9 and IPFIX flow records. Conflicting flows were intentionally selected from available sample data, and confirmed to be problematic with current functionality. The above mentioned UDP input PR was then applied along with the changes to the Netflow codec in this PR. The same data was them tested again in the exact same order. The result was that there was no longer any conflict of data from different devices.

[robcowart]

I have added the source port to the template key in https://github.com/logstash-plugins/logstash-codec-netflow/pull/187/commits/c3126bdb150a8a2d9b52f3d69df9f5e0afb3f082 due to these passages in RFC7011...

Regarding template management in general...

A Collecting Process MUST NOT use Templates from one Transport Session to decode Data Sets in a subsequent Transport Session.

And specific to UDP...

As Template IDs are unique per UDP session and per Observation Domain, at any given time, the Collecting Process SHOULD maintain the following for all the current Template Records and Options Template Records: <IPFIX Device, Exporter source UDP port, Collector IP address, Collector destination UDP port, Observation Domain ID, Template ID, Template Definition, Last Received>.

A review of available sample data and live devices indicates that the source port used by a flow exporter is consistent for all records, until for example the flow accounting process is restarted.

[robcowart]

@colinsurprenant As you have been active on this plugin recently, could you perhaps review this PR, and the related PR (logstash-plugins/logstash-input-udp#46) for the UDP input? As you can see above, there are a lot of issues that have been raised by users which require this fix.

I would also be willing to take on the role of maintainer of this plugin if it would help fixes to move more quickly through the process.

[harrygit944]

Does "logstash-codec-netflow (4.2.1)" have the change ?

[robcowart]

This PR must first be merged by the maintainers of this repo... which at this time seems to be the Elastic team.

It is a bit frustrating that an issue that affects so many users apparently doesn’t even merit a comment.

[colinsurprenant]

Thanks for you contribution and sorry about slow followup @robcowart - this is not by lack of interest; we just have a really hard time keeping up with the flow of issues in our plugins ecosystem.

We will followup on this and get back to you for your maintainer offer!

[nightah]

@colinsurprenant any luck with following this up? Is there anything we can do to support you and the team to get this through?

[colinsurprenant]

@robcowart A few observations:

• what if metadata is not nil but either or both host and port are not part of the metadata?
• could that introduce a change in behaviour and if so could we make that optional?
• I would like to see specs added

[robcowart]

@colinsurprenant thanks for the review. As this PR depends on the changes to the UDP input, I am going to hold on any additional work here until that is sorted.

[robcowart]

Unfortunately the Elastic team declined to merge UDP input changes (see... logstash-plugins/logstash-input-udp#46). This makes this PR irrelevant.

[sgreszcz]

@robcowart , any recommendations on the workaround here. We were pinning our hopes on using Logstash to handle our large netflow consumption and sending the data to ES to use elastiflow and other custom processing. Because of template collision errors, I guess we need to look at something else, or build our own netflow decoder into another open-source tool.

Since they use the same Elasticsearch netflow codec, I guess this affects filebeat too?

[colinsurprenant]

To clarify; after looking into the PRs we realized that the proposed solution here and in the udp input is not compatible with the API and introduce a one-off and incompatible behaviour with the rest of the ecosystem. We also want to avoid having codecs manage per-stream state and for this we introduced the IdentityMapCodec some time ago which manages codec instances per stream. We would prefer to have a solution that leverages the IdentityMapCodec which will require a bit more work at the udp input but should work as-is with the netflow codec. We will be happy to provide assistance in moving this forward.