search

ntop / ntopng

Web-based Traffic and Security Network Traffic Monitoring
http://www.ntop.org
GNU General Public License v3.0
6.23k stars 654 forks source link

Couldn't load the plugins and also cannot recognize the --elastic #977

Closed VenkatH closed 7 years ago

VenkatH commented 7 years ago

OS: centos 6.8 final nprobe version: nProbe Pro v.7.4.170109 ($Revision: 5334 $)

nprobe is the proxy to collect NetFlow lite from the cisco switch and export to Elasticsearch which running in the same machine. I have demo license under the /etc/nprobe.license. when i start the nprobe it prints the below statements in the log.

[root@NetFlowLiteELK nprobe]# cat nprobe-udp\@0.log
10/Feb/2017 15:52:32 [nprobe.c:2078] WARNING: [LICENSE] Your time-limited license will expire on Sat Mar 11 16:11:02 2017
10/Feb/2017 15:52:32 [nprobe.c:3407] Valid nProbe Pro license found
10/Feb/2017 15:52:32 [plugin.c:168] No plugins found in ./plugins
10/Feb/2017 15:52:32 [plugin.c:176] Loading 24 plugins [.so] from /usr/local/lib/nprobe/plugins
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin DHCP Protocol [/etc/nprobe.license.dhcp]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin Diameter Protocol [/etc/nprobe.license.diameter]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin DNS/LLMNR Protocol [/etc/nprobe.license.dns]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin Export Plugin [/etc/nprobe.license.export]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin FTP Protocol [/etc/nprobe.license.ftp]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin GTPv0 Signaling Protocol [/etc/nprobe.license.gtpv0]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin GTPv1 Signaling Protocol [/etc/nprobe.license.gtpv1]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin GTPv2 Signaling Protocol [/etc/nprobe.license.gtpv2]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin HTTP Protocol [/etc/nprobe.license.http]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin IMAP Protocol [/etc/nprobe.license.email]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin Netflow-Lite Plugin [/etc/nprobe.license.nflite]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin Oracle Protocol [/etc/nprobe.license.oracle]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin POP3 Protocol [/etc/nprobe.license.email]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin System process information [/etc/nprobe.license.process]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin Radius Protocol [/etc/nprobe.license.radius]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin RTP Plugin [/etc/nprobe.license.voip]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin S1AP Protocol [/etc/nprobe.license.S1AP]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin SIP Plugin [/etc/nprobe.license.voip]: License Ok
10/Feb/2017 15:52:32 [plugin.c:778] Unable to enable plugin SMTP Protocol [/etc/nprobe.license.email]: License Ok
10/Feb/2017 15:52:32 [nprobe.c:4168] WARNING: If you want to preserve the -M value, please specify -w before -M
nprobe: unrecognized option '--elastic'
10/Feb/2017 15:52:32 [nprobe.c:4807] WARNING: Unrecognized option '???'
10/Feb/2017 15:52:32 [nprobe.c:4872] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
10/Feb/2017 15:52:32 [nprobe.c:4875] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
10/Feb/2017 15:52:32 [nprobe.c:4975] Welcome to nProbe Pro v.7.4.170109 ($Revision: 5334 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration
10/Feb/2017 15:52:32 [nprobe.c:4985] Running on CentOS release 6.8 (Final)
10/Feb/2017 15:52:32 [nprobe.c:4996] [LICENSE] nProbe SystemId: 68A1AE1E76066ACF
10/Feb/2017 15:52:32 [nprobe.c:7308] Welcome to nProbe v.7.4.170109 for x86_64-unknown-linux-gnu
10/Feb/2017 15:52:32 [nprobe.c:6415] Using NetFlow Packet Payload Len: 1472
10/Feb/2017 15:52:32 [nprobe.c:6475] WARNING: Your template lacks some important fields
10/Feb/2017 15:52:32 [nprobe.c:6476] WARNING: Unless you know what you are doing, make sure
10/Feb/2017 15:52:32 [nprobe.c:6477] WARNING: your template (-T) contains at least
10/Feb/2017 15:52:32 [nprobe.c:6478] WARNING: %IPV4_SRC_ADDR %IPV4_DST_ADDR %PROTOCOL
10/Feb/2017 15:52:32 [nprobe.c:6479] WARNING: %L4_SRC_PORT %L4_DST_PORT
10/Feb/2017 15:52:32 [nprobe.c:6537] WARNING: Protocol will be ignored (your template lacks %PROTOCOL)
10/Feb/2017 15:52:32 [plugin.c:1045] 0 plugin(s) enabled
10/Feb/2017 15:52:32 [nprobe.c:6816] Each flow is 114 bytes long
10/Feb/2017 15:52:32 [nprobe.c:6817] The # packets per flow has been set to 11
10/Feb/2017 15:52:32 [nprobe.c:6836] Non IPv4/v6 traffic is discarded according to the template
10/Feb/2017 15:52:32 [util.c:434] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
10/Feb/2017 15:52:32 [util.c:445] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
10/Feb/2017 15:52:32 [nprobe.c:5495] Using packet capture length 128
10/Feb/2017 15:52:32 [nprobe.c:7631] Not capturing packet from interface (collector mode)
10/Feb/2017 15:52:32 [collect.c:147] Flow collector listening on port 9995 (IPv4/v6)
10/Feb/2017 15:52:32 [nprobe.c:7856] nProbe started successfully

here is my nprobe.conf -n=none -i=none -t=60 -d=60 -a= -e=1 -B=10 -w=128000 -z=0 -S=1:1 -E=0:0 -g=/var/run/nprobe-none.pid --collector-port=9995 -V=9 -T ="%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS%IN_BYTES %OUT_PKTS %OUT_BYTES %L4_SRV_PORT_MAP %L4_DST_PORT_MAP %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %PROTOCOL_MAP %SRC_TOS %SRC_AS %DST_AS %SRC_MASK %DST_MASK %WHOIS_DAS_DOMAIN %HTTP_URL %HTTP_METHOD %HTTP_HOST %HTTP_FBOOK_CHAT %HTTP_SITE %DNS_QUERY %DNS_QUERY_ID %DNS_QUERY_TYPE %DNS_RET_CODE %DNS_NUM_ANSWERS %DNS_TTL_ANSWER %LONGEST_FLOW_PKT %SHORTEST_FLOW_PKT %DST_IP_CITY %DST_IP_COUNTRY %SRC_IP_CITY %SRC_IP_COUNTRY %PACKETS_OBSERVED %FLOW_START_SEC %FLOW_END_SEC %OUT_DST_MAC %IPV6_NEXT_HOP %DIRECTION %BIFLOW_DIRECTION %IP_PROTOCOL_VERSION %IN_SRC_MAC %IPV6_DST_MASK %IPV6_SRC_MASK %IPV6_DST_ADDR %IPV6_SRC_ADDR" --elastic = ntopng;nprobe-%Y.%m.%d;http://localhost:9200/_bulk; --dump-stats=/home/nprobe/log/none-0_flows_stats.txt

lucaderi commented 7 years ago

This is correct. Plugins such as DNS require packets to be dissected in order to operate. If you send nProbe flows this cannot happen and thus they are disabled.

VenkatH commented 7 years ago

-- nprobe.conf -n=none -i=none -t=60 -d=60 -a= -e=1 -B=10 -w=128000 -z=0 -S=1:1 -E=0:0 -g=/var/run/nprobe-none.pid --collector-port=9995 -V=9 -T ="%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS%IN_BYTES %OUT_PKTS %OUT_BYTES %L4_SRV_PORT_MAP %L4_DST_PORT_MAP %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %PROTOCOL_MAP %SRC_TOS %SRC_AS %DST_AS %SRC_MASK %DST_MASK %WHOIS_DAS_DOMAIN %HTTP_URL %HTTP_METHOD %HTTP_HOST %HTTP_FBOOK_CHAT %HTTP_SITE %LONGEST_FLOW_PKT %SHORTEST_FLOW_PKT %DST_IP_CITY %DST_IP_COUNTRY %SRC_IP_CITY %SRC_IP_COUNTRY %PACKETS_OBSERVED %FLOW_START_SEC %FLOW_END_SEC %OUT_DST_MAC %IPV6_NEXT_HOP %DIRECTION %BIFLOW_DIRECTION %IP_PROTOCOL_VERSION %IN_SRC_MAC %IPV6_DST_MASK %IPV6_SRC_MASK %IPV6_DST_ADDR %IPV6_SRC_ADDR" --elastic = ntopng;nprobe-%Y.%m.%d;http://localhost:9200/_bulk; --dump-stats=/home/nprobe/log/none-0_flows_stats.txt

-- nprobe.log 12/Feb/2017 12:52:39 [nprobe.c:2078] WARNING: [LICENSE] Your time-limited license will expire on Sat Mar 11 16:11:02 2017 12/Feb/2017 12:52:39 [nprobe.c:3407] Valid nProbe Pro license found 12/Feb/2017 12:52:39 [plugin.c:168] No plugins found in ./plugins 12/Feb/2017 12:52:39 [plugin.c:176] Loading 24 plugins [.so] from /usr/local/lib/nprobe/plugins 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin DHCP Protocol [/etc/nprobe.license.dhcp]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin Diameter Protocol [/etc/nprobe.license.diameter]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin DNS/LLMNR Protocol [/etc/nprobe.license.dns]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin Export Plugin [/etc/nprobe.license.export]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin FTP Protocol [/etc/nprobe.license.ftp]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin GTPv0 Signaling Protocol [/etc/nprobe.license.gtpv0]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin GTPv1 Signaling Protocol [/etc/nprobe.license.gtpv1]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin GTPv2 Signaling Protocol [/etc/nprobe.license.gtpv2]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin HTTP Protocol [/etc/nprobe.license.http]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin IMAP Protocol [/etc/nprobe.license.email]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin Netflow-Lite Plugin [/etc/nprobe.license.nflite]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin Oracle Protocol [/etc/nprobe.license.oracle]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin POP3 Protocol [/etc/nprobe.license.email]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin System process information [/etc/nprobe.license.process]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin Radius Protocol [/etc/nprobe.license.radius]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin RTP Plugin [/etc/nprobe.license.voip]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin S1AP Protocol [/etc/nprobe.license.S1AP]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin SIP Plugin [/etc/nprobe.license.voip]: License Ok 12/Feb/2017 12:52:39 [plugin.c:778] Unable to enable plugin SMTP Protocol [/etc/nprobe.license.email]: License Ok 12/Feb/2017 12:52:39 [nprobe.c:4168] WARNING: If you want to preserve the -M value, please specify -w before -M nprobe: unrecognized option '--elastic' 12/Feb/2017 12:52:39 [nprobe.c:4807] WARNING: Unrecognized option '???' 12/Feb/2017 12:52:39 [nprobe.c:4872] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ? 12/Feb/2017 12:52:39 [nprobe.c:4875] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ? 12/Feb/2017 12:52:39 [nprobe.c:4975] Welcome to nProbe Pro v.7.4.170109 ($Revision: 5334 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration 12/Feb/2017 12:52:39 [nprobe.c:4985] Running on CentOS release 6.8 (Final) 12/Feb/2017 12:52:39 [nprobe.c:4996] [LICENSE] nProbe SystemId: 68A1AE1E76066ACF 12/Feb/2017 12:52:39 [nprobe.c:7308] Welcome to nProbe v.7.4.170109 for x86_64-unknown-linux-gnu 12/Feb/2017 12:52:39 [nprobe.c:6415] Using NetFlow Packet Payload Len: 1472 12/Feb/2017 12:52:39 [nprobe.c:6475] WARNING: Your template lacks some important fields 12/Feb/2017 12:52:39 [nprobe.c:6476] WARNING: Unless you know what you are doing, make sure 12/Feb/2017 12:52:39 [nprobe.c:6477] WARNING: your template (-T) contains at least 12/Feb/2017 12:52:39 [nprobe.c:6478] WARNING: %IPV4_SRC_ADDR %IPV4_DST_ADDR %PROTOCOL 12/Feb/2017 12:52:39 [nprobe.c:6479] WARNING: %L4_SRC_PORT %L4_DST_PORT 12/Feb/2017 12:52:39 [nprobe.c:6537] WARNING: Protocol will be ignored (your template lacks %PROTOCOL) 12/Feb/2017 12:52:39 [plugin.c:1045] 0 plugin(s) enabled 12/Feb/2017 12:52:39 [nprobe.c:6816] Each flow is 114 bytes long 12/Feb/2017 12:52:39 [nprobe.c:6817] The # packets per flow has been set to 11 12/Feb/2017 12:52:39 [nprobe.c:6836] Non IPv4/v6 traffic is discarded according to the template 12/Feb/2017 12:52:39 [util.c:434] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat 12/Feb/2017 12:52:39 [util.c:445] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat 12/Feb/2017 12:52:39 [nprobe.c:5495] Using packet capture length 128 12/Feb/2017 12:52:39 [nprobe.c:7631] Not capturing packet from interface (collector mode) 12/Feb/2017 12:52:39 [collect.c:147] Flow collector listening on port 9995 (IPv4/v6) 12/Feb/2017 12:52:39 [nprobe.c:7856] nProbe started successfully

Note: in the above config i removed the fields related to the DNS

  1. Is there a config parameter to mention use only required plugins, so that i dont want to worry about sending packets for DNS plugin.
  2. Do I need to explicitly mention the flow coming from the switch is NetFlow Lite
  3. Why it couldn't enable the any of those plugins
  4. even after using demo license why its throws unrecognized option '--elastic' or --elastic parameter passed wrongly in my config
  5. I am trying to receive Netflow Lite from the switch and export the flows to the elasticsearch within the machine without using logstash, so couldn't recommend any changes that i should make in my config or nprobe setup