search

ntop / nProbe

Open source components and extensions for nProbe
http://ntop.org
GNU General Public License v2.0
1.62k stars 44 forks source link

nProbe Kubernetes container fails to load with cloud license #613

Open jmessenger51 opened 1 month ago

jmessenger51 commented 1 month ago

I have a Kubernetes cluster where ntopng is running as (1) container and nprobe is running as another container. ntopng is running successfully with the cloud license.

nprobe starts but fails to associate with the cloud license. The error messages are:

<omit>@<omit>:/# nprobe -v

Welcome to nProbe v.10.5.240522 for x86_64-pc-linux-gnu
with native PF_RING acceleration.
Built with nDPI 4.9.0-4706-fd02baa

Copyright 2002-24 ntop.org

Version:       10.5.240522
Build OS:      Ubuntu 22.04.4 LTS
SystemID:      <omit>
GIT rev:       dev:5206c92b16e66b6b91f2893ac1b90688a16342f0:20240522
License:       Invalid license (/etc/nprobe.license) [License mismatch (check systemId, product version, or host date/time)]

nProbe is subject to the terms and conditions defined in
the LICENSE and EULA files that are part of this package.

nProbe also contains third party code:
Radix tree code - (C) The Regents of the University of Michigan
                      ("The Regents") and Merit Network, Inc.
sFlow collector - (C) InMon Inc.

When nprobe starts it shows:

23/May/2024 18:46:30 [nprobe.c:5803] Reading configuration file /etc/nprobe/nprobe.conf
23/May/2024 18:46:30 [plugin.c:178] No plugins found in ./plugins
23/May/2024 18:46:30 [plugin.c:186] Loading 23 plugins [.so] from /usr/lib/nprobe/plugins
23/May/2024 18:46:30 [nprobe.c:6140] Disabling flow cache during collection
23/May/2024 18:46:30 [nprobe.c:6393] Using Engine Type/Id 0:248 [NetFlow, 8 bit]
23/May/2024 18:46:30 [nprobe.c:6396] Using ObservationPoint Type/Id  <omit> [IPFIX, 16 bit]
23/May/2024 18:46:31 [cloud.c:105] Successfully connected to ntop cloud
23/May/2024 18:46:31 [cloud.c:119] Unique id ntop/<omit>
23/May/2024 18:46:31 [nprobe.c:8118] Using ZMQ sourceId <omit>
23/May/2024 18:46:31 [nprobe.c:8195] Flow cache is disabled in flow collection mode
23/May/2024 18:46:31 [nprobe.c:8198] Welcome to nProbe v.10.5.240522 for x86_64-pc-linux-gnu with native PF_RING acceleration
23/May/2024 18:46:31 [nprobe.c:8220] Pro Edition running on Ubuntu 22.04.4 LTS
23/May/2024 18:46:31 [nprobe.c:8228] Current limits [4 ZMQ exporters][4 collector devices]
23/May/2024 18:46:31 [nprobe.c:8243] SystemId: <omit>
23/May/2024 18:46:31 [nprobe.c:8336] Sample rate [packet: 1][flow collection/export: 1/1]
23/May/2024 18:46:31 [nprobe.c:11711] WARNING: ***************************************************************
23/May/2024 18:46:31 [nprobe.c:11712] WARNING: * NOTE: This is a DEMO version limited to:                    *
23/May/2024 18:46:31 [nprobe.c:11713] WARNING: *       - flows export: 5000 (live), 512 (pcap).              *
23/May/2024 18:46:31 [nprobe.c:11714] WARNING: *       - 300 seconds.                                        *
23/May/2024 18:46:31 [nprobe.c:11715] WARNING: ***************************************************************
23/May/2024 18:46:31 [exportPlugin.c:670] WARNING: Kafka support requires nprobe Enterprise M or better: disabled
23/May/2024 18:46:31 [nprobe.c:10197] Adding %OBSERVATION_POINT_ID to the template

the cloud.config file is mounted to /etc/ntop/cloud.conf the nprobe license is mounted to /etc/nprobe.license

I'm pulling - docker.io/ntop/nprobe.dev:latest so it shouldn't be related to the stable release not supporting the cloud license.

iPenguin commented 1 month ago

I'm having the same problem. Any help would be greatly appreciated.

Thanks!

jmessenger51 commented 1 month ago

After troubleshooting more, the container is able to recognize the license if its launched with a shell script that initiates the CLI command to start nprobe. However, if its launched with the config file it fails.

cardigliano commented 1 month ago

@jmessenger51 could you show how you run it exactly in both cases? We need to reproduce this. Thank you.

jmessenger51 commented 1 month ago

Here is the Manifest using the CLI which works file that fails

`

apiVersion: v1 kind: ConfigMap metadata: name: nprobe-config namespace: ntopng labels: app: nprobe data: startup.sh: |-

!/bin/bash

  # config options: https://www.ntop.org/guides/nprobe/cli_options.html
  nprobe --ntopng zmq://<IP Removed>:5556 -i none -n none -3 6343 --zmq-probe-mode \
    -E 0:<Observation Point Removed> -Q 1 -u 1

apiVersion: v1 kind: Secret metadata: name: nprobe-cloud-license namespace: ntopng labels: app: nprobe stringData: cloud.conf: |- < Cloud license removed>

apiVersion: v1 kind: Secret metadata: name: nprobe-license namespace: ntopng labels: app: nprobe stringData: nprobe.license: |- < nProbe license removed>

apiVersion: v1 kind: Service metadata: name: nprobe-svc namespace: ntopng labels: app: nprobe spec: type: LoadBalancer loadBalancerIP: selector: app: nprobe ports:

And here is the failing manifest that uses the config file:

`

apiVersion: v1 kind: ConfigMap metadata: name: nprobe-config namespace: ntopng labels: app: nprobe data: nprobe.conf: |

config options: https://www.ntop.org/guides/nprobe/cli_options.html

  --ntopng=zmq://<ntopng IP removed>:5556
  -i=none
  -n=none
  -3=6343
  --zmq-probe-mode
  -E=0:248
  -Q=1
  -u=1

apiVersion: v1 kind: Secret metadata: name: nprobe-cloud-license namespace: ntopng labels: app: nprobe stringData: cloud.conf: |-

--- apiVersion: v1 kind: Secret metadata: name: nprobe-license namespace: ntopng labels: app: nprobe stringData: nprobe.license: |- --- apiVersion: v1 kind: Service metadata: name: nprobe-svc namespace: ntopng labels: app: nprobe spec: type: LoadBalancer loadBalancerIP: selector: app: nprobe ports: - name: sflow port: 6343 targetPort: 6343 protocol: UDP --- apiVersion: apps/v1 kind: Deployment metadata: name: nprobe namespace: ntopng labels: app: nprobe spec: replicas: 1 selector: matchLabels: app: nprobe template: metadata: labels: app: nprobe spec: containers: - name: nprobe image: docker.io/ntop/nprobe.dev:latest imagePullPolicy: IfNotPresent ports: - name: nprobe containerPort: 3000 protocol: TCP env: - name: NTOP_CONFIG value: /etc/nprobe/nprobe.conf volumeMounts: - name: config mountPath: /etc/nprobe/ readOnly: true - name: license mountPath: /etc/nprobe.license subPath: nprobe.license readOnly: true - name: cloud-license mountPath: /etc/ntop readOnly: true volumes: - name: config configMap: name: nprobe-config - name: license secret: secretName: nprobe-license - name: cloud-license secret: secretName: nprobe-cloud-license `
jmessenger51 commented 1 month ago

the CLI method is not reliablly running, it starts and runs for 6-8 hours, then fails. still investigating.

jmessenger51 commented 1 month ago

after investigating further, it appears that the containers systemID even if on the same host will change, even though the cloud.config is in place its dropping to demo mode.

(2) system ID's generated during test: LF612D37600A10F21--UF612D37618E7FF9B--OL L336E92A900A10F21--U336E92A918E7FF9B--OL

and log details:

31/May/2024 20:24:04 [plugin.c:178] No plugins found in ./plugins 31/May/2024 20:24:04 [plugin.c:186] Loading 23 plugins [.so] from /usr/lib/nprobe/plugins 31/May/2024 20:24:04 [nprobe.c:6154] Disabling flow cache during collection 31/May/2024 20:24:04 [nprobe.c:6407] Using Engine Type/Id 0:248 [NetFlow, 8 bit] 31/May/2024 20:24:04 [nprobe.c:6410] Using ObservationPoint Type/Id 0:248 [IPFIX, 16 bit] 31/May/2024 20:24:05 [ntop_cloud.c:179] ERROR: Login failed [user: 2816936171][host: cloud.ntop.org cloud-hsol.ntop.org][port: 8883]: Network error 31/May/2024 20:24:05 [nprobe.c:8132] Using ZMQ sourceId 1883812335 31/May/2024 20:24:05 [nprobe.c:8209] Flow cache is disabled in flow collection mode 31/May/2024 20:24:05 [nprobe.c:8212] Welcome to nProbe v.10.5.240531 for x86_64-pc-linux-gnu with native PF_RING acceleration 31/May/2024 20:24:05 [nprobe.c:8234] Pro Edition running on Ubuntu 22.04.4 LTS 31/May/2024 20:24:05 [nprobe.c:8242] Current limits [4 ZMQ exporters][4 collector devices] 31/May/2024 20:24:05 [nprobe.c:8257] SystemId: L336E92A900A10F21--U336E92A918E7FF9B--OL 31/May/2024 20:24:05 [nprobe.c:8350] Sample rate [packet: 1][flow collection/export: 1/1] 31/May/2024 20:24:05 [nprobe.c:11725] WARNING: 31/May/2024 20:24:05 [nprobe.c:11726] WARNING: NOTE: This is a DEMO version limited to: 31/May/2024 20:24:05 [nprobe.c:11727] WARNING: - flows export: 5000 (live), 512 (pcap). 31/May/2024 20:24:05 [nprobe.c:11728] WARNING: - 300 seconds. 31/May/2024 20:24:05 [nprobe.c:11729] WARNING: 31/May/2024 20:24:05 [exportPlugin.c:670] WARNING: Kafka support requires nprobe Enterprise M or better: disabled 31/May/2024 20:24:05 [nprobe.c:10211] Adding %OBSERVATION_POINT_ID to the template 31/May/2024 20:24:05 [nprobe.c:10398] Using template %IN_SRC_MAC %OUT_DST_MAC %INPUT_SNMP %OUTPUT_SNMP %SRC_VLAN %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %SRC_TOS %DST_TOS %IP_PROTOCOL_VERSION %PROTOCOL %L7_PROTO %L7_CONFIDENCE %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %CLIENT_TCP_FLAGS %SERVER_TCP_FLAGS %L7_PROTO_RISK %L7_RISK_SCORE %EXPORTER_IPV4_ADDRESS %DIRECTION %SAMPLING_INTERVAL %TOTAL_FLOWS_EXP %NPROBE_IPV4_ADDRESS %NPROBE_INSTANCE_NAME %FLOW_SOURCE %JA3C_HASH %JA3S_HASH %JA4C_HASH %POST_NAT_SRC_IPV4_ADDR %POST_NAT_DST_IPV4_ADDR %POST_NAPT_SRC_TRANSPORT_PORT %POST_NAPT_DST_TRANSPORT_PORT %OBSERVATION_POINT_ID 31/May/2024 20:24:05 [nprobe.c:10400] Using NetFlow Packet Payload Len: 1472 31/May/2024 20:24:05 [template.c:3582] WARNING: Unable to locate template 'NPROBE_IPV6_ADDRESS': Pro version does not include plugins. 31/May/2024 20:24:05 [template.c:3582] WARNING: Unable to locate template 'POST_NAT_SRC_IPV6_ADDR': Pro version does not include plugins. 31/May/2024 20:24:05 [template.c:3582] WARNING: Unable to locate template 'POST_NAT_DST_IPV6_ADDR': Pro version does not include plugins. 31/May/2024 20:24:05 [template.c:3582] WARNING: Unable to locate template 'NPROBE_IPV6_ADDRESS': Pro version does not include plugins. 31/May/2024 20:24:05 [template.c:3582] WARNING: Unable to locate template 'POST_NAT_SRC_IPV6_ADDR': Pro version does not include plugins. 31/May/2024 20:24:05 [template.c:3582] WARNING: Unable to locate template 'POST_NAT_DST_IPV6_ADDR': Pro version does not include plugins. 31/May/2024 20:24:05 [plugin.c:1205] 0 plugin(s) enabled 31/May/2024 20:24:05 [nprobe.c:10945] Each flow is 265 bytes long 31/May/2024 20:24:05 [nprobe.c:10946] The # flows per packet has been set to 4 31/May/2024 20:24:05 [nprobe.c:10949] IP TOS is accounted 31/May/2024 20:24:05 [nprobe.c:11801] Flow export type (-T): bidirectional flows 31/May/2024 20:24:05 [nprobe.c:11995] Flows ASs will not be computed (no GeoDB files loaded with --as-list) 31/May/2024 20:24:05 [nprobe.c:12027] Flows will be exported in NetFlow 9 format 31/May/2024 20:24:05 [nprobe.c:12073] Learning the public IP address.. Disable it with --disable-startup-checks 31/May/2024 20:24:05 [util.c:6491] Initializing ZMQ as client 31/May/2024 20:24:05 [util.c:6522] Exporting flows towards ZMQ endpoint

cardigliano commented 1 month ago

Actually you should not care about the system id when running a cloud license, that's one of the advantages of using this licensing model. It seems there is some connectivity issues, that is probably causing the license check failure:

31/May/2024 20:24:05 [ntop_cloud.c:179] ERROR: Login failed [user: 2816936171][host: cloud.ntop.org cloud-hsol.ntop.org][port: 8883]: Network error

cardigliano commented 1 month ago

Do you have any special network configuration in place here (e.g. proxy)?

jmessenger51 commented 1 month ago

Do you have any special network configuration in place here (e.g. proxy)?

The cluster sits behind a firewall. However, we have a policy that allows 8883 & 8884 out to:

We see the sessions allowed through the firewall with TCP syn / syn-ack / ack flags so the session starts successfully.

jmessenger51 commented 4 weeks ago

when I restart the container, sometimes it says it successfully connected to the cloud, but then still has a demo license:

04/Jun/2024 15:45:07 [plugin.c:178] No plugins found in ./plugins 04/Jun/2024 15:45:07 [plugin.c:186] Loading 23 plugins [.so] from /usr/lib/nprobe/plugins 04/Jun/2024 15:45:07 [nprobe.c:6154] Disabling flow cache during collection 04/Jun/2024 15:45:07 [nprobe.c:6407] Using Engine Type/Id 0:248 [NetFlow, 8 bit] 04/Jun/2024 15:45:07 [nprobe.c:6410] Using ObservationPoint Type/Id 0:248 [IPFIX, 16 bit] 04/Jun/2024 15:45:07 [cloud.c:105] Successfully connected to ntop cloud 04/Jun/2024 15:45:07 [cloud.c:119] Unique id ntop/2816936171/L201E689A00A10F21--U201E689A18E7FF9B--OL/nprobe/7 04/Jun/2024 15:45:07 [nprobe.c:8132] Using ZMQ sourceId 772815004 04/Jun/2024 15:45:07 [nprobe.c:8209] Flow cache is disabled in flow collection mode 04/Jun/2024 15:45:07 [nprobe.c:8212] Welcome to nProbe v.10.5.240531 for x86_64-pc-linux-gnu with native PF_RING acceleration 04/Jun/2024 15:45:07 [nprobe.c:8234] Pro Edition running on Ubuntu 22.04.4 LTS 04/Jun/2024 15:45:07 [nprobe.c:8242] Current limits [4 ZMQ exporters][4 collector devices] 04/Jun/2024 15:45:07 [nprobe.c:8257] SystemId: L201E689A00A10F21--U201E689A18E7FF9B--OL 04/Jun/2024 15:45:07 [nprobe.c:8350] Sample rate [packet: 1][flow collection/export: 1/1] 04/Jun/2024 15:45:07 [nprobe.c:11725] WARNING: 04/Jun/2024 15:45:07 [nprobe.c:11726] WARNING: NOTE: This is a DEMO version limited to: 04/Jun/2024 15:45:07 [nprobe.c:11727] WARNING: - flows export: 5000 (live), 512 (pcap). 04/Jun/2024 15:45:07 [nprobe.c:11728] WARNING: - 300 seconds. 04/Jun/2024 15:45:07 [nprobe.c:11729] WARNING: 04/Jun/2024 15:45:07 [exportPlugin.c:670] WARNING: Kafka support requires nprobe Enterprise M or better: disabled 04/Jun/2024 15:45:07 [nprobe.c:10211] Adding %OBSERVATION_POINT_ID to the template 04/Jun/2024 15:45:07 [nprobe.c:10398] Using template %IN_SRC_MAC %OUT_DST_MAC %INPUT_SNMP %OUTPUT_SNMP %SRC_VLAN %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %SRC_TOS %DST_TOS %IP_PROTOCOL_VERSION %PROTOCOL %L7_PROTO %L7_CONFIDENCE %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %CLIENT_TCP_FLAGS %SERVER_TCP_FLAGS %L7_PROTO_RISK %L7_RISK_SCORE %EXPORTER_IPV4_ADDRESS %DIRECTION %SAMPLING_INTERVAL %TOTAL_FLOWS_EXP %NPROBE_IPV4_ADDRESS %NPROBE_INSTANCE_NAME %FLOW_SOURCE %JA3C_HASH %JA3S_HASH %JA4C_HASH %POST_NAT_SRC_IPV4_ADDR %POST_NAT_DST_IPV4_ADDR %POST_NAPT_SRC_TRANSPORT_PORT %POST_NAPT_DST_TRANSPORT_PORT %OBSERVATION_POINT_ID 04/Jun/2024 15:45:07 [nprobe.c:10400] Using NetFlow Packet Payload Len: 1472 04/Jun/2024 15:45:07 [template.c:3582] WARNING: Unable to locate template 'NPROBE_IPV6_ADDRESS': Pro version does not include plugins. 04/Jun/2024 15:45:07 [template.c:3582] WARNING: Unable to locate template 'POST_NAT_SRC_IPV6_ADDR': Pro version does not include plugins. 04/Jun/2024 15:45:07 [template.c:3582] WARNING: Unable to locate template 'POST_NAT_DST_IPV6_ADDR': Pro version does not include plugins. 04/Jun/2024 15:45:07 [template.c:3582] WARNING: Unable to locate template 'NPROBE_IPV6_ADDRESS': Pro version does not include plugins. 04/Jun/2024 15:45:07 [template.c:3582] WARNING: Unable to locate template 'POST_NAT_SRC_IPV6_ADDR': Pro version does not include plugins. 04/Jun/2024 15:45:07 [template.c:3582] WARNING: Unable to locate template 'POST_NAT_DST_IPV6_ADDR': Pro version does not include plugins. 04/Jun/2024 15:45:07 [plugin.c:1205] 0 plugin(s) enabled

cardigliano commented 4 weeks ago

@jmessenger51 what version are you currently running?

jmessenger51 commented 4 weeks ago

Welcome to nProbe v.10.5.240531 for x86_64-pc-linux-gnu with native PF_RING acceleration. Built with nDPI 4.9.0-4718-81e42b7

Copyright 2002-24 ntop.org

Version: 10.5.240531 Build OS: Ubuntu 22.04.4 LTS SystemID: L201E689A00A10F21--U201E689A18E7FF9B--OL GIT rev: dev:49971d892ff03082a32851a8a753d9dfa54fe737:20240531 License: Invalid license (/etc/nprobe.license) [License mismatch (check systemId, product version, or host date/time)]

jmessenger51 commented 4 weeks ago

When we launch the container in K8s the manifest tells it to pull: docker.io/ntop/nprobe.dev:latest

jmessenger51 commented 2 weeks ago

I'm able to get nprobe to launch when a host level license is installed. However, it continues to fail with the cloud license.