IPFIX/V9 templates are not cached correctly causing to data corruption/wrong parsing/memory leaks/infinite loops.

[avimas]

Currently V9 and IPFIX templates are cached using template id and exporter address as a key. Rfc3954 implies that v9 templates should be cached using the following fields:

msg header Source ID exporter source IP Template id

As stated below:

Source ID: A 32-bit value that identifies the Exporter Observation Domain. NetFlow Collectors SHOULD use the combination of the source IP address and the Source ID field to separate different export streams originating from the same Exporter.

As an example of how serious the problem is consider the attached short pcap taken from a Cisco device with multiple exporter instances(source id's):

Packet number 5 should be parsed using the template in packet number 2 i.e template id 257 and sourceID 513 instead it's being parsed using the template from packet 4 i.e template id 257 and sourceID 517, this leads to parsing valid data sets into empty templates which in turn leads to infinite loop in the decoder loop which leads to a memory leak (appending empty data to the data sets infinitely)

I made a local fix in memcache to cache the templates using the additional srcID/domain ID, however I am not sure it's not affecting the memcache_rpc.go

func (m MemCache) getShard(templateId uint16, addr net.IP, srcId uint32) (*TemplatesShard, uint32) { var key []byte hash := fnv.New32() sId := make([]byte, 4) tID := make([]byte, 2) binary.LittleEndian.PutUint32(sId, srcId) binary.BigEndian.PutUint16(tID, templateId) key = append(key, addr...) key = append(key, sId...) key = append(key, tID...)

hash.Write(key)
hSum32 := hash.Sum32()

return m[uint32(hSum32)%uint32(shardNo)], hSum32

}

The same behavior applies to IPFIX as well using domain id instead of source id.

To see all the issues caused by this behavior replay the attached full pcap to vflow.

pcaps.zip

[fredriklin]

Hello,

I'm seeing this issue as well with IPFIX data from a Nokia device. The symptoms are the same, empty sets being produced and and the process goes into an infinite loop until the it crashes because of running out of memory.

I can provide PCAPs if needed, however from looking at the ones provided above I'm certain I'm seeing the same issue.

[chrispassas]

For what its worth we are also seeing the same issue.

[avimas]

The issue is fixed in https://github.com/alexeigr/vflow it also contains fixes to other related security bugs. As I wrote in the case description I am not sure it's not affecting the memcache_rpc.go.

In my environment it works perfectly using many different exporters vendors. Please test it.

[glowa001]

Why don't you make a pull request instead of link to your own repository with the master as the only branch with a fix, that mixes with others and is split into several commits?

[avimas]

This fix might break the rpc template mechanism which I don't use in my environment, and don't have the time to check. I consulted the author but haven't got any answer yet regarding this.

[chrispassas]

@avimas @glowa001 I've created a pull request here https://github.com/VerizonDigital/vflow/pull/103

I took the good work from @avimas and applied it to both V9 and IPFix.

I had a similar issue in my environment and confirmed this fixed our issue.