search

netsampler / goflow2

High performance sFlow/IPFIX/NetFlow Collector
BSD 3-Clause "New" or "Revised" License
452 stars 107 forks source link

ipfx postNATDestinationIPv4Address #329

Closed Skop-22 closed 2 months ago

Skop-22 commented 3 months ago

Hello, I am a junior programmer. I am testing goflow2 with ipfx for MikroTik, but I don't know how to make the data returned by goflow2 include

225: "postNATSourceIPv4Address",
226: "postNATDestinationIPv4Address",
227: "postNAPTSourceTransportPort",
228: "postNAPTDestinationTransportPort".

I found it in the repository, but I don't know much about Go. Can someone help me?

What does this error mean? 2055 NetFlowV9 [version:9 type:Decode obsDomainId:0: templateId:256] Error template not found"

lspgn commented 2 months ago

Hello @Skop-22

You need to configure a mapping file, there's an example for flow_direction:

https://github.com/netsampler/goflow2/blob/7265159473ecb80dae984d375682a132f963f270/cmd/goflow2/mapping.yaml#L51-L52

https://github.com/netsampler/goflow2/blob/7265159473ecb80dae984d375682a132f963f270/cmd/goflow2/mapping.yaml#L31-L33

https://github.com/netsampler/goflow2/blob/7265159473ecb80dae984d375682a132f963f270/cmd/goflow2/mapping.yaml#L7

The error template not found, when you're using NetFlow v9/IPFIX, indicates the decoder have not received the template in order to decode the data payloads. It may take a minute in order to be able to decode those messages.

Skop-22 commented 2 months ago

Hello, thank you for answering my question. I have many questions, but I will start with the ones that have been bothering me the most. I still don't understand how to solve it.

  1. In MikroTik, there is a graph showing the upload and download consumption of each router. In all the returned flows, they follow the same pattern:

    cpe -> isp
isp -> internet
isp -> cpe

I consider the in-bytes going from cpe to isp if their time is practically the same. How can I calculate the in_bytes returned by each flow to represent the consumption?

  1. I use Apache Kafka to communicate with another project, but if for some reason the second project goes down, Apache Kafka stores it in the cache, and the server receiving the flow information crashes. How can I improve the performance to connect it with the second project?

  2. In the second project, I insert data in batches, but when creating the tables and inserting the data, the RAM usage increases significantly. Do you have any advice on how I can manage the data to insert it into MySQL and free up RAM?

I have been trying for quite some time to make it work correctly. If you can help me, I will be eternally grateful.

lspgn commented 2 months ago

I'm not sure I understand your questions

Inserting into MySQL will require a different tool. Possibly like Kafka Connect or something you have to develop. As a side note, you may want to avoid relational database and look more at OLAP and other log databases.

GoFlow2 has a very limited buffer. Usually, for the other tools, buffer (Kafka for instance) can be configured: like limiting the age of messages or size on disk. It is meant as protection against loss.