IPFIX templates sent by Citrix Netscaler are not cached

[tkuronen]

• Version: Logstash (7.7.1) / logstash-codec-netflow (4.2.1)
• Operating System: RHEL 7.8
• Input config:

  input {
  udp {
      id => "logstash_netscaler_input"
      port => 12208
      codec => netflow {
           versions => [10]
           target => ipfix
        cache_save_path => "/tmp"
        ipfix_definitions => "/etc/logstash/conf.d/ipfix.yaml"
        include_flowset_id => true
      }
      type => ipfix
   }
  }

  File ipfix.yaml is exactly this one: https://github.com/logstash-plugins/logstash-codec-netflow/blob/master/lib/logstash/codecs/netflow/ipfix.yaml

Traffic is received from Netscaler to Logstash but it seems that the IPFIX template can not be read. We get the following Warnings to logstash-plain.log from id's 256 - 332:

Can't (yet) decode flowset id 256 from observation domain id 0, because no template to decode it with has been received. This message will usually go away after 1 minute.

... and this Warning from enterprise field: Unsupported enterprise field {:type=>638, :enterprise=>5951, :length=>2}

However it can be confirmed by taking a tcpdump capture from Logstash server that templates are received:

When starting Logstash, it warns that: Template Cache does not exist {:file_path=>"/tmp/ipfix_templates.cache"} However the cache file is not generated even if pipeline is running for hours, although templates are sent by Netscaler every 60 seconds.

[TheVastyDeep]

That is correct. When the caching code was refactored in PR #158, the call to save_templates_cache was removed, and its functionailty was packaged as the persist method, but persist is never called.

[tkuronen]

Ok, thanks for the clarification. Could the persist() call be added and is there any workarounds until that ? And is the case so that even if ipfix_definitions file is defined in the config, it does not affect because of missing persist() call ?

[TheVastyDeep]

In terms of workarounds, the code will load JSON from the cache if it exists, so if you can populate the cache file manually that would work.

It is true that persist() is never called, but do_persist() is called from do_register() if it catches :invalid_template. That is described as "throwing this symbol causes the template to not be cached", but that appears to be the wrong way around. It looks like the template is ONLY cached if it is invalid.

The fix is going to involve some very minor changes to do_register, but I do not understand the code well enough to attempt a PR.

[tkuronen]

Thanks for the answers. I think that the workaround you suggested, could be worth a try. I did not find any documentation on the "schema" of the json cache file. Is is maybe the same than in Graylog's input: https://docs.graylog.org/en/3.3/pages/integrations/inputs/ipfix_input.html

{
  "enterprise_number": PRIVATE ENTERPRISE NUMBER,
  "information_elements": [
    {
      "element_id": ELEMENT ID NUMBER,
      "name": "NAME OF DEFINITION",
      "data_type": "ABSTRACT DATA TYPE"
    },
    ...
    ...
    ...
  {
    "element_id": ELEMENT ID NUMBER,
    "name": "NAME OF DEFINITIONt",
    "data_type": "ABSTRACT DATA TYPE"
  }
  ]
}