Slack link is broken in README

[punisherVX]

I have a question about logging in the docker container and was going to use the slack link, but it redirects and says the shared join link is no longer valid. Can you please fix so I can ask my question there?

Thanks.

[mehrdadrad]

We prefer to have your question here but if you want to chat, we can chat at Linkedin (mehrdadrad)

[punisherVX]

Sure, will do it here. Thanks.

I am wanting to send IPFIX to vflow and then use the kafka producer to send it to a topic. Using tcpdump (port 4739), I can see that the IPFIX packets are hitting the server, but I can't figure out how to see if they are actually hitting vflow.

I am using the container with the following command

 docker run -d -p 4739:4739 -p 4729:4729 -p 6343:6343 -v ~/tools/mq.conf:/etc/vflow/mq.conf -v ~/tools/vflow.conf:/etc/vflow/vflow.conf -v ~/tools/vflow.hosts:/etc/hosts --name=vflow_ipfix mehrdadrad/vflow

Looking at the docker logs I get this (and only this, nothing else):

docker logs -f vflow_ipfix
vFlow running with PID 12.
[vflow] 2020/10/21 00:07:14 Welcome to vFlow v.0.7.0 Apache License 2.0
[vflow] 2020/10/21 00:07:14 Copyright (C) 2018 Verizon. github.com/VerizonDigital/vflow
[vflow] 2020/10/21 00:07:14 starting prometheus http server ...
[vflow] 2020/10/21 00:07:14 sFlow is running (UDP: listening on [::]:6343 workers#: 200)
[vflow] 2020/10/21 00:07:14 ipfix is running (UDP: listening on [::]:4739 workers#: 300)
[vflow] 2020/10/21 00:07:14 netflow v5 is running (UDP: listening on [::]:9996 workers#: 200)
[vflow] 2020/10/21 00:07:14 ipfix RPC enabled
[vflow] 2020/10/21 00:07:14 netflow v9 is running (UDP: listening on [::]:4729 workers#: 200)
[vflow] 2020/10/21 00:07:14 start producer: Kafka, brokers: [10.29.75.151:9092 10.29.75.152:9092 10.29.75.153:9092], topic: my.ipfix
[vflow] 2020/10/21 00:07:14 start producer: Kafka, brokers: [10.29.75.151:9092 10.29.75.152:9092 10.29.75.153:9092], topic: vflow.netflow9
[vflow] 2020/10/21 00:07:14 start producer: Kafka, brokers: [10.29.75.151:9092 10.29.75.152:9092 10.29.75.153:9092], topic: vflow.sflow
[vflow] 2020/10/21 00:07:14 start producer: Kafka, brokers: [10.29.75.151:9092 10.29.75.152:9092 10.29.75.153:9092], topic: vflow.netflow5

which looks correct, brokers are correct, changed the ipfix topic name, etc. The hosts file I am mounting has the entries for the advertised.listeners in kafka, hence why I put that in there.

However, I am not getting anything on the topic. I have looked around the container for any other logs and have found /var/log/vflow.log but that is empty.

Here are the two conf files if you need those. vflow.conf

ipfix-workers: 300
ipfix-topic: my.ipfix

mq.conf

brokers:
    - 10.29.75.151:9092
    - 10.29.75.152:9092
    - 10.29.75.153:9092
retry-max: 1
retry-backoff: 30

Any suggestions on where to look or some troubleshooting steps? Thanks

[mehrdadrad]

Once you added the below to vflow.conf, vflow prints out the ipfix samples on the terminal.

verbose: true

[punisherVX]

I added that and it didn't change anything, tho it does have this in the log now, so I know it took: [vflow] 2020/10/22 01:13:56 the full logging enabled However, I am still not getting anything. Is this line in the log correct, meaning that it is bound to all interfaces in the container? [vflow] 2020/10/22 01:13:56 ipfix.go:116: ipfix is running (UDP: listening on [::]:4739 workers#: 300)

[punisherVX]

OK - got past that part. Built a new VM and followed the instructions to install via deb pkg. Works to start and I can see the packets, but I am getting this error in the log:

[vflow] 2020/10/22 03:56:16 Multiple errors:
- IPFIX element key (513) not exist
- IPFIX element key (513) not exist

Kafka topic is created (sweet!) but nothing is being put on the topic.

I am guessing that I need to let vflow know about our specific keys? Where do I put the template info for our specific element keys? I have it (I think - still new to IPFIX) but not sure where it goes?

Really appreciate the help.

[mehrdadrad]

looks there is proprietary ipfix information element, you can add it at /etc/vflow/ipfix.elements (if you installed it by deb pkg then it should be exist otherwise you can download from https://raw.githubusercontent.com/VerizonDigital/vflow/master/scripts/ipfix.elements

[punisherVX]

Thanks. We are close. I am still getting the same error tho.

I did install via deb package, but that file did not exist. I also checked the file-system to make sure it wasn't some place else, couldn't find it. So, I downloaded the one provided, put it in /etc/vflow/ipfix.elements and then appended the elements below, stopped and started vflow but it is still throwing the error IPFIX element key (513) not exist

Is there a vflow.conf setting that I can set to make sure it is looking in the right place? I don't see it in the configuration document.

51886:
 0:
 - :skip
 234:
 - :uint32
 - :ingressVRFID
 8:
 - :uint32
 - :sourceIPv4Address
 12:
 - :uint32
 - :destinationIPv4Address
 4:
 - :uint8
 - :protocolIdentifier
 7:
 - :uint16
 - :sourceTransportPort
 11:
 - :uint16
 - :destinationTransportPort
 513:
 - :string
 - :flow-state-index
 514:
 - :string
 - :role
 192:
 - :uint8
 - :ipTTL
 32:
 - :uint16
 - :icmpTypeCodeIPv4
 184:
 - :uint32
 - :tcpSequenceNumber
 185:
 - :uint32
 - tcpAcknowledgementNumber
 186:
 - :unint16
 - :tcpWindowSize
 238:
 - :uint8
 - :tcpWindowScale
 515:
 - :uint8
 - :tcp-state
 516:
 - :uint16
 - :tcp-win-mss
 517:
 - :uint32
 - :tcp-exceptions
 210:
 - :uint16
 - :paddingOctets
 518:
 - :uint32
 - :tcp-rtt
 148:
 - :uint32
 - :flowId
 1282:
 - :uint16
 - :egress-logical-interface
 1281:
 - :uint16
 - :egress-interface-type
 86:
 - :uint64
 - :packetTotalCount
 85:
 - :uint64
 - :octetTotalCount
 2:
 - :uint32
 - :packetDeltaCount
 1:
 - :uint32
 - :octetDeltaCount
 135:
 - :uint64
 - :droppedPacketTotalCount
 134:
 - :uint64
 - :droppedOctetTotalCount
 133:
 - :uint32
 - :droppedPacketDeltaCount
 132:
 - :uint32
 - :droppedOctetDeltaCount
 152:
 - :uint64
 - :flowStartMilliseconds
 153:
 - :uint64
 - :flowEndMilliseconds
 1286:
 - :uint64
 - :pen1
 27:
 - :uint128
 - :sourceIPv6Address
 28:
 - :uint128
 - :destinationIPv6Address
 56:
 - :uint48
 - :sourceMacAddress
 80:
 - :uint48
 - :destinationMacAddress
 256:
 - :uint16
 - :ethernetType

[mehrdadrad]

-config set configuration path put the ipfix.elements and vflow.conf at the same directory like /etc/vflow then run:

vflow -config /etc/vflow/vflow.conf

[punisherVX]

Problem still exists with that flag. It's seeing the vflow.conf and mq.conf (even without the -config option) but not picking up the ipfix.elements file.

[punisherVX]

Thanks to @mehrdadrad was able to get this fixed.
Needed to specify only 1 Kafka broker (for now) in the mq.conf file and the topic was populated.