search

ntop / nProbe

Open source components and extensions for nProbe
http://ntop.org
GNU General Public License v2.0
1.65k stars 44 forks source link

In nProbe —json-labels seems to be working with the default template, but not with custom ones #616

Open xavibonet opened 3 months ago

xavibonet commented 3 months ago

We're sending data from nprobe in collector mode to an ELK node.

When we're using standard templates in conjunction with --json-labels, not using -T parameter, everything works fine. Data is received and seen ok on ELK.

If we use a custom template with -T, and not using --json-labels, we receive all the info in ELK but we don't see the field names, just the IDs.

If we send custom templates, -T parameter, in conjuction with --json-labels data is not seen in ELK. It seems that custom templates with --json-labels have some weird behaviour. I must say that data seems to be sent to ELK because we can see traffic flowing through the FW from nProbe to ELK.

Our nprobe config is :

Version: 10.4.240307 Build OS: Ubuntu 22.04.3 LTS Edition: nProbe Enterprise L

Thanks

xavibonet commented 3 months ago

So far our workaround is to modify Kibana Index Patterns

lucaderi commented 3 months ago

Can you please provide an example for reproducing the defect?

xavibonet commented 3 months ago

It all started when we tried to add some new fields to the @NTOPNG@ template. We wanted to get nflow not just from the switches but from our Paloalto FW too. We needed three more fields and I'll try to explain what we found.

The problem arises when we use the -T @NTOPNG@ in our config. If we don't use "-T " parameter then we can use add "--json-labels" and it works, just there's not all the info that we need there because the default template is used. See image1.

imagen

If we use "-T @NTOPNG@" we stop seeing the name of the fields, just the ID, but all the fields are there, even the new ones from paloalto because we add the file with the definition, we use the following config:

-T "@NTOPNG@ %FIREWALL_EVENT %PALOALTO_APPID %PALOALTO_USERID %PALOALTO_FLOWID" --load-custom-fields /etc/nprobe/newfields_netflow.txt

We can see all the fields here in image2:

imagen

And finally, if now we add the "--json-labels" we stop seeing info in the ELK node, just look at the records in the timeline that's become flat at 0 (although we can see in our FW that there's information flowing from nprobe to ELK):

imagen

I don't know if my explanation use clear or just a bunch of images 😄