Issue with SSL and Kafka

[anindyar]

We were trying to enable SSL on kafka and were using the documentation at this url https://www.ntop.org/guides/nProbe/case_study/exporting_to_kafka.html

but seems like the kafka config switches are not working for us somehow. can you please look into this urgently? The setup is now in production and we are stuck with the boxes not pushing data over SSL.

Here is a sample config that we were using and the output error message is in the screenshot.

--kafka-conf="ssl.ca.location=/home/nbox/trustkey.pem" --kafka-config="security.protocol=ssl"

The error we are seeing is this.

image.png

We, in fact, tried with a couple of switches and we are invariably getting the same error saying

ERROR: Unable to set Kafka property "": [No such configuration property: ""]

Regards, Anindya

[simonemainardi]

Please, send me the full nProbe configuration used. If you are using a test CAs and certs, please also send me them so I can check if they've been properly generated.

[anindyar]

Dear Simone,

Please find below the full configuration file that we used last. we had done a varity of combination of the below parameters. and have tried with both .jks keystore file with "ssl.truststore.location" parameter and with pem certificate file using "ssl.ca.location" but both are givng same errors.

Also I am afread the CA certs are not test certs and are generated by cloudera engineers, so we wont be able to share them here. you can please spceify if there is any limitation of the certificate format type. I will try to work around that.

----------Configuration File---------------------

This configuration for nProbe exposes many more options than is available in the file provided

when downloading nProbe.

Most likely you will only need to change the -n and -i options for your environment. However, a

number of other options are documented in case you need them.

#

The provided template works very well with ElastiFlow - A solution to collect and analyze network

flow data using the Elastic Stack. ElastiFlow is available from:

https://github.com/robcowart/elastiflow

-V: flow export version

It is used to specify the flow version for exported flows. Supported versions are 5 (v5), 9 (v9)

and 10 (IPFIX).

-V=9

-T: flow template definition

Contrary to NetFlow v5 where the flow format is fixed, NetFlow V9 and IPFIX flows have a custom

format that can be specified at runtime using this option.

###########################

Optimized Template Test 1

###########################

-T="%IPV4_SRC_ADDR %IPV4_DST_ADDR %PROTOCOL %L4_SRC_PORT %L4_DST_PORT %FIRST_SWITCHED %LAST_SWITCHED %IN_BYTES %IN_PKTS %PROTOCOL %PROTOCOL_MAP %L4_SRC_PORT %L4_SRC_PORT_MAP %IPV4_SRC_ADDR %IPV4_SRC_MASK %L4_DST_PORT %L4_SRV_PORT %IPV4_DST_ADDR %OUT_BYTES %OUT_PKTS %IN_SRC_MAC %SRC_VLAN %DST_VLAN %DIRECTION %OUT_DST_MAC %L7_PROTO_NAME %DHCP_CLIENT_MAC %DHCP_REMOTE_ID %DHCP_SUBSCRIBER_ID %DHCP_MESSAGE_TYPE %DIAMETER_REQ_MSG_TYPE %DIAMETER_RSP_MSG_TYPE %DIAMETER_REQ_ORIGIN_HOST %DIAMETER_RSP_ORIGIN_HOST %DIAMETER_REQ_USER_NAME %DIAMETER_RSP_RESULT_CODE %DIAMETER_EXP_RES_VENDOR_ID %DIAMETER_EXP_RES_RESULT_CODE %DIAMETER_HOP_BY_HOP_ID %DNS_QUERY %DNS_QUERY_TYPE %DNS_RESPONSE %FTP_LOGIN %FTP_COMMAND %FTP_COMMAND_RET_CODE %HTTP_URL %HTTP_METHOD %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST %HTTP_SITE %NETBIOS_QUERY_NAME %NETBIOS_QUERY_TYPE %NETBIOS_QUERY_RSP"

##################

all attributes

######################

#################################

Preferred attributes Template

#################################

-T="%IN_BYTES %IN_PKTS %PROTOCOL %PROTOCOL_MAP %SRC_TOS %TCP_FLAGS %L4_SRC_PORT %L4_SRC_PORT_MAP %IPV4_SRC_ADDR %IPV4_SRC_MASK %INPUT_SNMP %L4_DST_PORT %L4_DST_PORT_MAP %L4_SRV_PORT %L4_SRV_PORT_MAP %IPV4_DST_ADDR %IPV4_DST_MASK %OUTPUT_SNMP %IPV4_NEXT_HOP %SRC_AS %DST_AS %LAST_SWITCHED %FIRST_SWITCHED %OUT_BYTES %OUT_PKTS %IPV6_SRC_ADDR %IPV6_DST_ADDR %IPV6_SRC_MASK %IPV6_DST_MASK %ICMP_TYPE %SAMPLING_INTERVAL %SAMPLING_ALGORITHM %FLOW_ACTIVE_TIMEOUT %FLOW_INACTIVE_TIMEOUT %ENGINE_TYPE %ENGINE_ID %TOTAL_BYTES_EXP %TOTAL_PKTS_EXP %TOTAL_FLOWS_EXP %MIN_TTL %MAX_TTL %DST_TOS %IN_SRC_MAC %SRC_VLAN %DST_VLAN %DOT1Q_SRC_VLAN %DOT1Q_DST_VLAN %IP_PROTOCOL_VERSION %DIRECTION %IPV6_NEXT_HOP %MPLS_LABEL_1 %MPLS_LABEL_2 %MPLS_LABEL_3 %MPLS_LABEL_4 %MPLS_LABEL_5 %MPLS_LABEL_6 %MPLS_LABEL_7 %MPLS_LABEL_8 %MPLS_LABEL_9 %MPLS_LABEL_10 %OUT_DST_MAC %APPLICATION_ID %PACKET_SECTION_OFFSET %SAMPLED_PACKET_SIZE %SAMPLED_PACKET_ID %EXPORTER_IPV4_ADDRESS %EXPORTER_IPV6_ADDRESS %FLOW_ID %FLOW_START_SEC %FLOW_END_SEC %FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS %BIFLOW_DIRECTION %OBSERVATION_POINT_TYPE %OBSERVATION_POINT_ID %SELECTOR_ID %IPFIX_SAMPLING_ALGORITHM %SAMPLING_SIZE %SAMPLING_POPULATION %FRAME_LENGTH %PACKETS_OBSERVED %PACKETS_SELECTED %SELECTOR_NAME %APPLICATION_NAME %USER_NAME %FRAGMENTS %CLIENT_NW_LATENCY_MS %SERVER_NW_LATENCY_MS %APPL_LATENCY_MS %NUM_PKTS_UP_TO_128_BYTES %NUM_PKTS_128_TO_256_BYTES %NUM_PKTS_256_TO_512_BYTES %NUM_PKTS_512_TO_1024_BYTES %NUM_PKTS_1024_TO_1514_BYTES %NUM_PKTS_OVER_1514_BYTES %CUMULATIVE_ICMP_TYPE %SRC_IP_COUNTRY %SRC_IP_CITY %DST_IP_COUNTRY %DST_IP_CITY %FLOW_PROTO_PORT %UPSTREAM_TUNNEL_ID %UPSTREAM_SESSION_ID %LONGEST_FLOW_PKT %SHORTEST_FLOW_PKT %RETRANSMITTED_IN_BYTES %RETRANSMITTED_IN_PKTS %RETRANSMITTED_OUT_BYTES %RETRANSMITTED_OUT_PKTS %OOORDER_IN_PKTS %OOORDER_OUT_PKTS %UNTUNNELED_PROTOCOL %UNTUNNELED_IPV4_SRC_ADDR %UNTUNNELED_L4_SRC_PORT %UNTUNNELED_IPV4_DST_ADDR %UNTUNNELED_L4_DST_PORT %L7_PROTO %L7_PROTO_NAME %DOWNSTREAM_TUNNEL_ID %DOWNSTREAM_SESSION_ID %FLOW_USER_NAME %FLOW_SERVER_NAME %PLUGIN_NAME %UNTUNNELED_IPV6_SRC_ADDR %UNTUNNELED_IPV6_DST_ADDR %NUM_PKTS_TTL_EQ_1 %NUM_PKTS_TTL_2_5 %NUM_PKTS_TTL_5_32 %NUM_PKTS_TTL_32_64 %NUM_PKTS_TTL_64_96 %NUM_PKTS_TTL_96_128 %NUM_PKTS_TTL_128_160 %NUM_PKTS_TTL_160_192 %NUM_PKTS_TTL_192_224 %NUM_PKTS_TTL_224_255 %IN_SRC_OSI_SAP %OUT_DST_OSI_SAP %DURATION_IN %DURATION_OUT %TCP_WIN_MIN_IN %TCP_WIN_MAX_IN %TCP_WIN_MSS_IN %TCP_WIN_SCALE_IN %TCP_WIN_MIN_OUT %TCP_WIN_MAX_OUT %TCP_WIN_MSS_OUT %TCP_WIN_SCALE_OUT %PAYLOAD_HASH %SRC_AS_MAP %DST_AS_MAP %SRC_AS_PATH_1 %SRC_AS_PATH_2 %SRC_AS_PATH_3 %SRC_AS_PATH_4 %SRC_AS_PATH_5 %SRC_AS_PATH_6 %SRC_AS_PATH_7 %SRC_AS_PATH_8 %SRC_AS_PATH_9 %SRC_AS_PATH_10 %DST_AS_PATH_1 %DST_AS_PATH_2 %DST_AS_PATH_3 %DST_AS_PATH_4 %DST_AS_PATH_5 %DST_AS_PATH_6 %DST_AS_PATH_7 %DST_AS_PATH_8 %DST_AS_PATH_9 %DST_AS_PATH_10 %DHCP_CLIENT_MAC %DHCP_CLIENT_IP %DHCP_CLIENT_NAME %DHCP_REMOTE_ID %DHCP_SUBSCRIBER_ID %DHCP_MESSAGE_TYPE %DIAMETER_REQ_MSG_TYPE %DIAMETER_RSP_MSG_TYPE %DIAMETER_REQ_ORIGIN_HOST %DIAMETER_RSP_ORIGIN_HOST %DIAMETER_REQ_USER_NAME %DIAMETER_RSP_RESULT_CODE %DIAMETER_EXP_RES_VENDOR_ID %DIAMETER_EXP_RES_RESULT_CODE %DIAMETER_HOP_BY_HOP_ID %DNS_QUERY %DNS_QUERY_ID %DNS_QUERY_TYPE %DNS_RET_CODE %DNS_NUM_ANSWERS %DNS_TTL_ANSWER %DNS_RESPONSE %FTP_LOGIN %FTP_PASSWORD %FTP_COMMAND %FTP_COMMAND_RET_CODE %HTTP_URL %HTTP_METHOD %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST %HTTP_SITE %IMAP_LOGIN %MYSQL_SERVER_VERSION %MYSQL_USERNAME %MYSQL_DB %MYSQL_QUERY %MYSQL_RESPONSE %MYSQL_APPL_LATENCY_USEC %NETBIOS_QUERY_NAME %NETBIOS_QUERY_TYPE %NETBIOS_QUERY_RSP %ORACLE_USERNAME %ORACLE_QUERY %ORACLE_RSP_CODE %ORACLE_RSP_STRING %ORACLE_QUERY_DURATION %POP_USER %SRC_PROC_PID %SRC_PROC_NAME %SRC_PROC_UID %SRC_PROC_USER_NAME %SRC_FATHER_PROC_PID %SRC_FATHER_PROC_NAME %SRC_PROC_ACTUAL_MEMORY %SRC_PROC_PEAK_MEMORY %SRC_PROC_AVERAGE_CPU_LOAD %SRC_PROC_NUM_PAGE_FAULTS %SRC_PROC_PCTG_IOWAIT %DST_PROC_PID %DST_PROC_NAME %DST_PROC_UID %DST_PROC_USER_NAME %DST_FATHER_PROC_PID %DST_FATHER_PROC_NAME %DST_PROC_ACTUAL_MEMORY %DST_PROC_PEAK_MEMORY %DST_PROC_AVERAGE_CPU_LOAD %DST_PROC_NUM_PAGE_FAULTS %DST_PROC_PCTG_IOWAIT %RADIUS_REQ_MSG_TYPE %RADIUS_RSP_MSG_TYPE %RADIUS_USER_NAME %RADIUS_CALLING_STATION_ID %RADIUS_CALLED_STATION_ID %RADIUS_NAS_IP_ADDR %RADIUS_NAS_IDENTIFIER %RADIUS_USER_IMSI %RADIUS_USER_IMEI %RADIUS_FRAMED_IP_ADDR %RADIUS_ACCT_SESSION_ID %RADIUS_ACCT_STATUS_TYPE %RADIUS_ACCT_IN_OCTETS %RADIUS_ACCT_OUT_OCTETS %RADIUS_ACCT_IN_PKTS %RADIUS_ACCT_OUT_PKTS %RTP_SSRC %RTP_FIRST_SEQ %RTP_FIRST_TS %RTP_LAST_SEQ %RTP_LAST_TS %RTP_IN_JITTER %RTP_OUT_JITTER %RTP_IN_PKT_LOST %RTP_OUT_PKT_LOST %RTP_IN_PKT_DROP %RTP_OUT_PKT_DROP %RTP_IN_PAYLOAD_TYPE %RTP_OUT_PAYLOAD_TYPE %RTP_IN_MAX_DELTA %RTP_OUT_MAX_DELTA %RTP_SIP_CALL_ID %RTP_MOS %RTP_IN_MOS %RTP_OUT_MOS %RTP_R_FACTOR %RTP_IN_R_FACTOR %RTP_OUT_R_FACTOR %RTP_IN_TRANSIT %RTP_OUT_TRANSIT %RTP_RTT %RTP_DTMF_TONES %SIP_CALL_ID %SIP_CALLING_PARTY %SIP_CALLED_PARTY %SIP_RTP_CODECS %SIP_INVITE_TIME %SIP_TRYING_TIME %SIP_RINGING_TIME %SIP_INVITE_OK_TIME %SIP_INVITE_FAILURE_TIME %SIP_BYE_TIME %SIP_BYE_OK_TIME %SIP_CANCEL_TIME %SIP_CANCEL_OK_TIME %SIP_RTP_IPV4_SRC_ADDR %SIP_RTP_L4_SRC_PORT %SIP_RTP_IPV4_DST_ADDR %SIP_RTP_L4_DST_PORT %SIP_RESPONSE_CODE %SIP_REASON_CAUSE %SIP_C_IP %SIP_CALL_STATE %SMTP_MAIL_FROM %SMTP_RCPT_TO %SSDP_HOST %SSDP_USN"

-T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %SRC_AS %DST_AS %HTTP_URL %HTTP_METHOD %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST %HTTP_SITE %DHCP_CLIENT_MAC %DHCP_CLIENT_IP %DHCP_CLIENT_NAME %DHCP_REMOTE_ID %DHCP_SUBSCRIBER_ID %DHCP_MESSAGE_TYPE %DNS_QUERY %DNS_QUERY_ID %DNS_QUERY_TYPE %DNS_RET_CODE %DNS_NUM_ANSWERS %DNS_TTL_ANSWER %DNS_RESPONSE"

-U: flow template id

NetFlow v9 and IPFIX flows format is specified in a template whose definition is sent by nProbe

before to start sending flows. The flow format is defined by --T, where --U is used to set the

template identifier. This option should not be used unless the default template value (257) needs

to be changed. As based on -T nProbe can define several templates, this value is the one used for

the first defined template.

-U=888

-n|--collector: collector addresses

This specifies the NetFlow collectors addresses to which nProbe will send the flows. If more than

one is specified, they need to be separated with a comma or the --n flag can be repeated several

times (e.g. -n 172.22.3.4:33,172.22.3.4:34 and -n 172.22.3.4:33 --n 172.22.3.4:34 are

equivalent). When multiple collectors are defined, you can control the way flows are exported

using the --a option (see below); if on a collector address the destination port is omitted,

flows are sent to 2055 port and whereas if all the option is not specified, by default, flows are

sent to the loop back interface (127.0.0.1) on port 2055. If this parameter is used, nProbe

exports flows towards collector running at 127.0.0.1:2055. By default the UDP protocol is used

but also TCP and SCTP (Linux only when nProbe is compiled with SCTP support and the kernel

supports it). In this case you can specify the collector address as udp://:,

tcp:// :, and sctp://:,

-n=none

-3|--collector-port: specifies the NetFlow collector port

It is now possible to use the nProbe as NetFlow proxy. With --collector-port we can se the

incoming NetFlow port on which flows are received instead of sniffing packets. nProbe is able to

convert flows from various versions. For instance “nprobe --collector-port 2055 --i

192.168.0.1:2056 --V 10” converts each flow received on port 2055 to IPFIX and sends them to

192.168.0.1:2056.

-i|--interface: interface name

It specifies the interface from which packets are captured. If -i is not used, nProbe will use

the default interface (if any). In case a user needs to activate nProbe on two different

interfaces, then he/she needs to activate multiple nProbe instances once per interface. For

debugging purposes it is possible to pass nProbe a .pcap file from which packets will be read. If

nProbe is compiled and activated with PF_RING support, you can specify multiple interfaces from

which packets are captured. For example, “-i eth0,eth1”, will merge packets received on eth0 and

eth1 into a single traffic stream. This configuration is particularly useful when merging the two

directions (TX and RX) of a network TAP.

-i=nt:stream1

-t: maximum flow lifetime

Regardless of the flow duration, a flow that has been active for more that the specified maximum

lifetime is considered expired and it will be emitted. Further packets belonging to the same

flow will be accounted on a new flow.

-t=60

-d: maximum flow idle lifetime

A flow is over when the last packet received is older that the maximum flow idle lifetime. This

means that whenever applicable, (e.g. SNMP walk) UDP flows will not be accounted on 1 packet/1

flow basis, but on one global flow that accounts all the traffic. This has a benefit on the total

number of generated flows and on the overall collector performance.

-d=15

-l: maximum queue timeout

It specifies the maximum amount of time that a flow can be queued waiting to be exported. Use

this option in order to try to pack several flows into fewer packets, but at the same time have

an upper bound timeout for queuing flows into the probe.

-l=60

-s: snaplen

This flag specifies the portion of the packet (also called snaplen) that will be captured by

nProbe. By default nprobe sets the snaplen automatically according to its configuration, but you

can override its value using thia flag.

-p: flow aggregation

Flows can be aggregated both at collector and probe side. However probe allocation is much more

effective as it reduces significantly the number of emitted flows hence the work that the

collector has to carry on. nProbe supports various aggregation levels that can be selected

specifying with the --p flag. The aggregation format is /////

where each option can be set to 0 (ignore) or 1 (take care). Ignored fields are set to a null

value. For instance the value 0/0/1/0/0/0 is useful for creating a map of who’s talking to who

(network conversation matrix).

-a: select flow export policy

When multiple collectors are defined (see --n option), nProbe sends them flows in round robin.

However it is possible to send the same flow to all collectors as a flow redirector does if the

--a option is used.

-O: set the number of threads that fetch packets out of the network interface.

In general: the more threads are available, the better is the performance. However it is not

suggested to have too many threads as in some platforms this can slow down the probe. Start with

1 and increase it if necessary. We suggest to run nprobe as single threaded application and

distribute the traffic across multiple probes using PF_RING (e.g. PF_RING cluster or libzero). In

fact adding threads you will end up spending a lot of time on synchronization without improving

the performance. Please refer to this post

http://www.ntop.org/nprobe/10-gbit-line-rate-netflow-traffic-analysis-using-nprobe-and-dna/ for

more information.

-u: input device index

The NetFlow specification contains a numeric index in order to identify flows coming from

different interfaces of the same probe. As multiple nProbe instances can be started on the same

host but on different devices, the collector can use this flag to divide flows according to the

interface number. If --u is not used, then nprobe will use 0 as interface index. Alternatively,

if -1 is used then the last two bytes of the mac address of the flow sender are used as index.

-Q: output device index

Similar to --u but for the output interface.

--vlanid-as-iface-idx <mode: inner | outer | single | double>

nProbe can use the VLAN tag as interface identifier. Using this flag you enable this feature. As

VLAN tags can be stacked you need to specify if the inner or outer tag will be used for the

interface identifier. Since VLAN fields are included in the template, this is option is likely

not necessary.

--discard-unknown-flows <mode:0 | 1 | 2>

nProbe includes nDPI support for analyzing packet contents in order to detect application

protocol. The mode value can be used to:

• 0: Export all known (i.e. those whose application protocol has been detected) and unknown (i.e.

the application protocol is unknown)

• 1: Export only known flows, discarding unknown flows.

• 2: Export only unknown flows, discarding known flows.

--discard-unknown-flows=mode:0

-w: size of the hash that stores the flows

The default size is 131072 and it should be enough for most of networks. In case flows are not

emitted often and with strong traffic conditions it would be necessary to increase the hash. See

later in this manual for knowing more about nProbe tuning.

-W: Discard IPv6 traffic

Use this flag if you want nProbe not to account IPv6 traffic.

-e: flow export delay

Some collectors cannot keep up with nProbe export speed. This flag allows flows to be slow down

by adding a short delay (specified in ms) between two consecutive exports. The maximum allowed

delay is 1000 ms.

-B: packet count delay

It specified how many flow packets need to be sent before --e is applied,

-z: minimum TCP flow size

Peer-to-peer applications, attacks or misconfigured applications often generate a lot of tiny TCP

flows that can cause significant load on the collector side. As most collector setups often

discarded those flows, it is possible to instrument nProbe via the --z flag not to emit such

flows. Note that the --z flag affects only the TCP protocol (i.e. UDP, ICMP and other protocols

are not affected).

-M: maximum number of active flows

It is used to limit the maximum number of concurrent flows that the probe can sustain. This is

useful for preventing the probe from creating as many flows as needed and hence to take over all

the available resources.

-E: netflow engine

Specify the netflow engineType:engineId into the generated flows.

-m: minimum number of flows per packet

In order to minimize the number of emitted packets containing flows, it is possible to specify

the minimum number of flows that necessarily need to be contained in a packet. This means that

the packet is not emitted until the specified number of flows is reached.

-m=2

-q: :[] flow sender address and port

This option is used to specify the address -- and, optionally, the port -- that will be used by

nProbe to emit the flows towards the destination indicated with -n. In practice, nProbe will

create a socket and bind it to :[port], thus allowing the user to choose the interface

taken by the emitted flows when leaving the host. It is not possible to specify an arbitrary IP

address. Only local interface addresses are allowed. An error is raised when trying to specify a

non-allowed address.

-S: sample rate :

nProbe uses all the captured packets for calculating flows. In some situations (e.g. strong

traffic conditions) it is necessary to reduce the number of packets that need to be handled by

nProbe. This option specifies the sampling rate, i.e. the number of packets that are discarded

between two packets used to produce flows. You can also specify the flow sample rate that reduce

the egress flow rate thus lowering the load on collectors. The default value is 1:1 (no packet

sample, no flow sample).

-g|--pid-file:

It specifies the path where nProbe will save the process PID.

-o: intra templates packet export.

It specifies the number of flow packets that are exported between two templates export.

-o=50

--aggregate-gtp-tunnels

Aggregates traffic flowing in each GTP tunnel based in tunnel id.

[--collector-sample-rate]

Specifies the sample rate of the collector (either in bytes of packets).

--tunnel:

Let the probe decode tunneled traffic (e.g. GTP or GRE traffic) and thus extract traffic

information from such traffic rather than from the external envelope.

--no-promisc:

With this option nProbe does not use promiscuous mode to capture packets.

--smart-udp-frags:

Ignore UDP fragmented packets with fragment offset greater than zero, and compute the fragmented

packet length on the initial fragment header. This flag might lead to inaccuracy in measurement

but it speeds us operations with fragmented traffic.

--ipsec-auth-data-len

Length of the authentication data of IPSec in tunnel mode. If not set, IPSec will not be decoded

but just accounted.

--black-list

With this option you can specify a list of networks or hosts from which all the incoming packets

will be discarded by the probe. The accepted notation can be CIDR format or the classical

network/netmask format.

--biflows-export-policy

Bi-directional flows are such when there is traffic in both direction of the flow (i.e.

source->dest and dest->source). As mono-directional flows might indicate suspicious activities,

this flag is used to determine the export policy:

• 0: Export all known (i.e. mono and bi-directional flows)

• 1: Export only bi-directional flows, discarding mono-directional flows.

• 2: Export only mono-directional flows, discarding bi-directional flows.

--biflows-export-policy=0

--dont-drop-privileges

Do not drop root privileges to user ‘nobody’ when this option is specified.

--dont-drop-privileges

--unprivileged-user

When nprobe drops privileges (unless --dont-drop-privileges is used) the user nobody is used. It

is possible to use another user by using this option.

--account-l2

NetFlow accounts IP traffic only, not counting layer 2 headers. Using this option the layer 2

headers are also accounted in flow traffic statistics.

--tcp

Delivers flows in JSON format via TCP to the specified pair server:port.

--ndpi-proto-ports

Read the nDPI custom protocol and ports configuration from the specified file. Please refer to

the nDPI manual for further information about the format of this file.

--disable-l7-protocol-guess

When nDPI is unable to detect a protocol, nProbe uses the port information to guess the protocol.

This flag prevents nProbe from doing that, so protocols are detected only by nDPI without relying

on default ports.

--disable-cache

nProbe implements a flow cache for merging packets belonging to the same flow. In proxy/collector

mode, nProbe can disable this feature so that incoming flows are not put in cache but immediately

exported.

--kafka="10.xxx.xx.xx:90923,10.xxx.xx.xx:90923,10.xxx.xx.xx:90923;NTOP-Incoming" --kafka-conf="security.protocol=ssl"

--kafka-conf="ssl.truststore.location=/home/nbox/truststore.jks"

--kafka-conf="ssl.ca.location=/home/nbox/trustkey.pem

[simonemainardi]

the default librdkafka version shipped with ubuntu16 doesn't have support for SSL so it ignores SSL-related configuration options (reference: https://github.com/edenhill/librdkafka/issues/896#issuecomment-259537327):

simone@devel:~/nProbe$ sudo apt-cache show librdkafka1
Package: librdkafka1
Priority: optional
Section: universe/libs
Installed-Size: 221
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Faidon Liambotis <paravoid@debian.org>
Architecture: amd64
Source: librdkafka
Version: 0.8.6-1.1

In order to enable SSL support you need version 0.9.x.

Personally, to make it work with SSL, I've cloned and compiled librdkafka from sources (https://github.com/edenhill/librdkafka), and then told nProbe to use the manually compiled version.

As you can see from the following output, SSL option is now properly recognized.

simone@devel:~/nProbe$ LD_LIBRARY_PATH=/home/simone/librdkafka/src/ ./nprobe -i none -n none --collector-port 2055 --kafka "192.168.2.129:9092;test7;none;0" --disable-cache --kafka-conf debug=msg --kafka-conf queue.buffering.max.ms=1000 --kafka-conf topic.auto.commit.interval.ms=2000 --kafka-conf security.protocol=ssl
29/Jan/2019 10:22:48 [plugin.c:187] Loading 25 plugins [.so] from ./plugins
29/Jan/2019 10:22:48 [nprobe.c:6107] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
29/Jan/2019 10:22:48 [nprobe.c:6110] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
29/Jan/2019 10:22:48 [nprobe.c:6197] Welcome to nProbe Pro v.8.7.190116 ($Revision: 6369 $) for x86_64-pc-linux-gnu with native PF_RING acceleration
29/Jan/2019 10:22:48 [nprobe.c:6207] Running on Ubuntu 16.04.5 LTS
29/Jan/2019 10:22:48 [nprobe.c:6285] Sample rate [packet: 1][flow collection/export: 1/1]
29/Jan/2019 10:22:48 [exportPlugin.c:719] Kafka property "debug" set to "msg"
29/Jan/2019 10:22:48 [exportPlugin.c:719] Kafka property "queue.buffering.max.ms" set to "1000"
29/Jan/2019 10:22:48 [exportPlugin.c:719] Kafka property "topic.auto.commit.interval.ms" set to "2000"
29/Jan/2019 10:22:48 [exportPlugin.c:719] Kafka property "security.protocol" set to "ssl"

You can also install the compiled librdkafka system-wide so you don't have to specify the LD_LIBRARY_PATH. Fundamental is to use a version 0.9.x.

[simonemainardi]

Solution given. Closing.