Filter IPs

[SamuelCarroll]

In an upcoming release of IPFIX-RITA we want to limit the number of connections we store. This reduces the size of the conn table in RITA and makes analyzing data faster

This isn't an urgent concern yet, need to wait for it to be implemented in RITA first

• [ ] Filter external IP to external IP
• [ ] Filter internal IP to internal IP
• [ ] Ensure connections on "alwaysinclude" list are present (nameservers, possible internal http proxies)

[SamuelCarroll]

This is more pressing now that the RITA changes are nearing completion and I'll need to ensure it doesn't break anything.

[Zalgo2462]

We'll want to copy the network config thats in RITA 2 and implement that here.

https://github.com/activecm/rita/blob/d39bbd02df7a6a6160ecca59befc3552d751cf57/etc/rita.yaml#L60

Since we throwout the data without caring which host is the sender or receiver, we can do all of the filtering early in the pipeline.

[Zalgo2462]

After discussing with the team, we agreed that we don't see a need for filtering based on which host is the source and which is the destination at this time or shortly into the future.

Therefore, the filter step can be implemented before stitching the flows together.