Open flexd opened 6 years ago
Yeah, not active on slack. I'm not sure about options templates either, never had a use for them. You're welcome to investigate.
I'm looking into it, but I have never done any kind of parsing like this. It does not look that complicated, and the format is well documented in the RFC.
Looks like there are two types of messages, that either contain a Template Set and a Data Set, or a Option Template Set and a Data set? Or could a IPFIX Message contain any number of these combinations?
IPFIX Message:
+--------+------------------------------------------. . .
| | +--------------+ +------------------+
|Message | | Template | | Data |
| Header | | Set | | Set | . . .
| | | (1 Template) | | (3 Data Records) |
| | +--------------+ +------------------+
+--------+------------------------------------------. . .
. . .-------------------------------------------+
+------------------+ +------------------+ |
| Options | | Data | |
. . . | Template Set | | Set | |
| (1 Template) | | (2 Data Records) | |
+------------------+ +------------------+ |
. . .-------------------------------------------+
The Options Template Set format is also described
The example in Figure O shows an Options Template Set with mixed
IANA-assigned and enterprise-specific Information Elements. It
consists of a Set Header, an Options Template Header, and several
Field Specifiers.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 258 | Field Count = N + M |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field Count = N |0| Scope 1 Infor. Element id. |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 1 Field Length |0| Scope 2 Infor. Element id. |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 2 Field Length | ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |1| Scope N Infor. Element id. |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope N Field Length | Scope N Enterprise Number ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
... Scope N Enterprise Number |1| Option 1 Infor. Element id. |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Option 1 Field Length | Option 1 Enterprise Number ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
... Option 1 Enterprise Number | ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |0| Option M Infor. Element id. |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Option M Field Length | Padding (optional) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
And with some examples from the RFC
A.4. Options Template Set Examples
A.4.1. Options Template Set Using IANA Information Elements
Per line card (the router being composed of two line cards), we want
to report the following Information Elements:
- Total number of IPFIX Messages: exportedMessageTotalCount
[IANA-IPFIX], with a length of 2 octets
- Total number of exported Flows: exportedFlowRecordTotalCount
[IANA-IPFIX], with a length of 2 octets
The line card, which is represented by the lineCardId Information
Element [IANA-IPFIX], is used as the Scope Field.
Therefore, the Options Template Set will be:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 24 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID 258 | Field Count = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field Count = 1 |0| lineCardId = 141 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 1 Field Length = 4 |0|exportedMessageTotalCount=41 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 |0|exportedFlowRecordTotalCo.=42|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
A.4.2. Options Template Set Using Enterprise-Specific Information
Elements
Per line card (the router being composed of two line cards), we want
to report the following Information Elements:
- Total number of IPFIX Messages: exportedMessageTotalCount
[IANA-IPFIX], with a length of 2 octets
- An enterprise-specific number of exported Flows, with a type of 42
and a length of 4 octets
The line card, which is represented by the lineCardId Information
Element [IANA-IPFIX], is used as the Scope Field.
The format of the Options Template Set is as follows:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 28 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID 259 | Field Count = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field Count = 1 |0| lineCardId = 141 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 1 Field Length = 4 |0|exportedMessageTotalCount=41 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 |1|Information Element id. = 42 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 | Enterprise number ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
... Enterprise number | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
A.4.3. Options Template Set Using an Enterprise-Specific Scope
In this example, we want to export the same information as in the
example in Appendix A.4.1:
- Total number of IPFIX Messages: exportedMessageTotalCount
[IANA-IPFIX], with a length of 2 octets
- Total number of exported Flows: exportedFlowRecordTotalCount
[IANA-IPFIX], with a length of 2 octets
But this time, the information pertains to a proprietary scope,
identified by enterprise-specific Information Element number 123.
The format of the Options Template Set is now as follows:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 28 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID 260 | Field Count = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field Count = 1 |1|Scope 1 Infor. El. id. = 123 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 1 Field Length = 4 | Enterprise Number ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
... Enterprise Number |0|exportedMessageTotalCount=41 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 |0|exportedFlowRecordTotalCo.=42|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
A Data Set is composed of Data Record(s). No Template Record is
included. A Template Record or an Options Template Record defines
the Data Record.
A Template Set contains only Template Record(s).
An Options Template Set contains only Options Template Record(s).
Following are some examples of IPFIX Messages:
1. An IPFIX Message consisting of interleaved Template, Data, and
Options Template Sets, as shown in Figure C. Here, Template and
Options Template Sets are transmitted "on demand", before the
first Data Set whose structure they define.
+--------+--------------------------------------------------------+
| | +----------+ +---------+ +-----------+ +---------+ |
|Message | | Template | | Data | | Options | | Data | |
| Header | | Set | | Set | ... | Template | | Set | |
| | | | | | | Set | | | |
| | +----------+ +---------+ +-----------+ +---------+ |
+--------+--------------------------------------------------------+
Figure C: IPFIX Message: Example 1
2. An IPFIX Message consisting entirely of Data Sets, sent after the
appropriate Template Records have been defined and transmitted to
the Collecting Process, as shown in Figure D.
+--------+----------------------------------------------+
| | +---------+ +---------+ +---------+ |
|Message | | Data | | Data | | Data | |
| Header | | Set | ... | Set | ... | Set | |
| | +---------+ +---------+ +---------+ |
+--------+----------------------------------------------+
Figure D: IPFIX Message: Example 2
3. An IPFIX Message consisting entirely of Template and Options
Template Sets, as shown in Figure E. Such a message can be used
to define or redefine Templates and Options Templates in bulk.
+--------+-------------------------------------------------+
| | +----------+ +----------+ +----------+ |
|Message | | Template | | Template | | Options | |
| Header | | Set | ... | Set | ... | Template | |
| | | | | | | Set | |
| | +----------+ +----------+ +----------+ |
+--------+-------------------------------------------------+
I'm kind of using this issue as a notepad for useful information about it. At some point I'll understand this enough to implement it, and/or someone else is capable and willing.
I see here https://github.com/calmh/ipfix/blob/master/parser.go#L296 that option templates are not handled at all.
I am not very familiar with the IPFIX format. How hard would it be to implement support for Option Templates?
I am trying to grab statistics from yaf flow files. Relevant docs embedded below in this post, see https://tools.netsa.cert.org/yaf/yaf.html and search for "Statistics Option Template" for the source.
I tried contacting you on the gophers slack to talk, but I figured I'd post here as well :)