exceptions with default example file

[ddraveh]

Hi, I was trying to use VERMONT on Ubuntu 14.04. Following the instruction, building and installation passed without errors but trying to run with the default example generate some exceptions, for example:

$ ./vermont -f configs/example.xml -d
24/11 11:07:46 VERMONT: message debug level is 3
24/11 11:07:46 INFO   : SensorManager: hertz jiffy value=100, hostname=ubuntu
24/11 11:07:46 INFO   : SensorManager started with following parameters:
24/11 11:07:46 INFO   :   - outputfilename=sensor_output.xml
24/11 11:07:46 INFO   :   - checkInterval=2 seconds
24/11 11:07:46 INFO   :   - append=0
24/11 11:07:46 FATAL  : InfoElementCfg: unknown information element name revflowStartMilliSeconds
terminate called after throwing an instance of 'std::runtime_error'
  what():  : InfoElementCfg: unknown information element name revflowStartMilliSeconds
Aborted (core dumped)  

Removing the 'problematic' elements names from example.xml, I finally got this error, which I was not able to overcome:

$ sudo ./vermont -f configs/example.xml -d
24/11 11:15:06 VERMONT: message debug level is 3
24/11 11:15:06 INFO   : SensorManager: hertz jiffy value=100, hostname=ubuntu
24/11 11:15:06 INFO   : SensorManager started with following parameters:
24/11 11:15:06 INFO   :   - outputfilename=sensor_output.xml
24/11 11:15:06 INFO   :   - checkInterval=2 seconds
24/11 11:15:06 INFO   :   - append=0
24/11 11:15:06 INFO   : Exporter: using maximum rate of 0 records/second
24/11 11:15:06 INFO   : Connecting module observer[Id = 1] -> packetQueue[Id = 2]
24/11 11:15:06 INFO   : Finding devices
24/11 11:15:06 INFO   : pcap opening interface=eth0, promisc=1, snaplen=128, timeout=100
24/11 11:15:06 INFO   : pcap seems to run on netmask 255.255.255.0
24/11 11:15:06 INFO   : Connecting module packetQueue[Id = 2] -> packetAggregator[Id = 3]
24/11 11:15:06 INFO   : IPFIX IE protocolIdentifier is contained in template 998, accepting all protocol types for this template
24/11 11:15:06 INFO   : Hashtable initialized with following parameters:
24/11 11:15:06 INFO   :   - minBufferTime=1
24/11 11:15:06 INFO   :   - maxBufferTime=1
24/11 11:15:06 INFO   :   - htableBits=17
24/11 11:15:06 INFO   : Done. Parsed 1 rules; minBufferTime 1, maxBufferTime 1
24/11 11:15:06 INFO   : Connecting module packetAggregator[Id = 3] -> ipfixAggregator[Id = 4]
24/11 11:15:06 INFO   : IPFIX IE protocolIdentifier is contained in template 888, accepting all protocol types for this template
24/11 11:15:06 INFO   : Hashtable initialized with following parameters:
24/11 11:15:06 INFO   :   - minBufferTime=5
24/11 11:15:06 INFO   :   - maxBufferTime=10
24/11 11:15:06 INFO   :   - htableBits=17
24/11 11:15:06 INFO   : IPFIX IE protocolIdentifier is contained in template 0, accepting all protocol types for this template
24/11 11:15:06 INFO   : Hashtable initialized with following parameters:
24/11 11:15:06 INFO   :   - minBufferTime=5
24/11 11:15:06 INFO   :   - maxBufferTime=10
24/11 11:15:06 INFO   :   - htableBits=17
24/11 11:15:06 FATAL  : did not find reverse element for flowStartMilliseconds (id=152, length=8)
terminate called after throwing an instance of 'std::runtime_error'
  what():  : did not find reverse element for flowStartMilliseconds (id=152, length=8) 

Please advise, Many thanks,

Danny

[nickbroon]

Strangely reveflowStartMilliSeconds appears in a lot of the examples, yet does not appear in the code base anywhere.

10:06 $ grep -r -i flowStartMilliSeconds * | grep rev
configs/offlineprinter.xml:             <ieName>revflowStartMilliSeconds</ieName>
configs/interop/simple_linux.xml:               <ieName>revflowStartMilliSeconds</ieName>
configs/interop/simple_freebsd.xml:             <ieName>revflowStartMilliSeconds</ieName>
configs/interop/csnet_udp_export.xml:               <ieName>revflowStartMilliSeconds</ieName>
configs/interop/simple_sctp_export_linux.xml:               <ieName>revflowStartMilliSeconds</ieName>
configs/interop/simple_udp_export_linux.xml:                <ieName>revflowStartMilliSeconds</ieName>
configs/mongo/mongow.xml:               <ieName>revflowStartMilliSeconds</ieName>
configs/sctp_exporter.xml:              <ieName>revflowStartMilliSeconds</ieName>
configs/db/dbwriter.xml:                <ieName>revflowStartMilliSeconds</ieName>
configs/analysis/autofocus_vermont.xml:         <ieName>revflowStartMilliSeconds</ieName>
configs/analysis/rbsworm_vermont.xml:           <ieName>revflowStartMilliSeconds</ieName>
configs/analysis/autofocus_eval.xml:            <ieName>revflowStartMilliSeconds</ieName>
configs/oracle/oxewriter.xml:               <ieName>revflowStartMilliSeconds</ieName>
configs/example.xml:                <ieName>revflowStartMilliSeconds</ieName>
configs/example.xml:                <ieName>revflowStartMilliSeconds</ieName>
src/modules/ipfix/Connection.cpp:       fi = record->templateInfo->getFieldInfo(IPFIX_TYPEID_flowStartMilliseconds, IPFIX_PEN_reverse);
src/modules/ipfix/database/IpfixDbWriterSQL.cpp:    {   CN_revFirstSwitched,    IPFIX_TYPEID_flowStartMilliseconds, "", IPFIX_PEN_reverse, 0}, // default value is invalid/not used for this entry
src/modules/ipfix/database/IpfixDbWriterSQL.cpp:                // look for alternative (revFlowStartMilliseconds/1000)
src/modules/ipfix/database/IpfixDbWriterSQL.cpp:                    if(dataTemplateInfo->fieldInfo[k].type == InformationElement::IeInfo(IPFIX_TYPEID_flowStartMilliseconds, IPFIX_PEN_reverse)) {
src/modules/ipfix/database/IpfixDbWriterMongo.cpp:  {CN_revFirstSwitchedMillis, 0, IPFIX_TYPEID_flowStartMilliseconds, IPFIX_PEN_reverse},
src/modules/ipfix/database/IpfixDbWriterMongo.cpp:                              // look for alternative (revFlowStartMilliseconds/1000)
src/modules/ipfix/database/IpfixDbWriterMongo.cpp:                                  if(dataTemplateInfo.fieldInfo[k].type == InformationElement::IeInfo(IPFIX_TYPEID_flowStartMilliseconds, IPFIX_PEN_reverse)) {
src/modules/ipfix/IpfixCsExporter.cpp:  uint64_t revtimestart = retrieveTime(record, IPFIX_TYPEID_flowStartNanoseconds, IPFIX_TYPEID_flowStartMilliseconds,

[nickbroon]

8da0d26785987749c4cce97d1724ed0e83f7060e appears to have removed this:

- { IPFIX_TYPEID_flowStartMilliSeconds, IPFIX_LENGTH_flowStartMilliSeconds, IPFIX_PEN_reverse, "revFlowStartMilliSeconds" },

from src/common/ipfixlolib/ipfix_names.c

but unlike most other fields which were added to src/common/ipfixlolib/ipfix_iana.c the revFlow... fields do appear to have been added anywhere.

Perhaps @ogasser, who made this change can comment? Are these field missing and need added somewhere? Or is there nolonger support for them, and all the config example need updated?

[ogasser]

Yes, this was removed as part of the automatic IPFIX IE generation from IANA sources.

This should be integrated again into the code, but instead of using the rev prefix I suggest to use a reverse attribute: <ieName reverse="true">flowStartMilliSeconds</ieName>.

@evintila will have a look at implementing this. After the implementation is done, we should also update the example config files and the wiki.

[ogasser]

Or maybe even better:

<nonFlowKey>
    <ieName>flowStartMilliSeconds</ieName>
    <reverse>true</reverse>
</nonFlowKey>

[leandrov96]

Hey,

I still can't seem to get revflowStartMilliSeconds, even with the solutions provided by the last two comments by @ogasser. Wireshark doesn't show these two fields, not even in the templates. Any help please?

[nickbroon]

The last two comments discuss how a solution for this could be implemented. That implementation has not yet been done. For the moment there are no rev... element supported, so any modules that require these to operate will not function. For example biflowAggregation is not possible without them, so <biflowAggregation>1</biflowAggregation> would need removed from configs/example.xml. <trwPortscanDetector> also requires biflows so it will also not work, so should probably be removed from configs/example.xml