search

Edgio / vflow

Enterprise Network Flow Collector (IPFIX, sFlow, Netflow)
http://www.verizonmedia.com
Apache License 2.0
1.1k stars 223 forks source link

No data written to Kafka topic #138

Open saloni-vunet opened 3 years ago

saloni-vunet commented 3 years ago

I am testing vFlow in Docker with Confluent Kafka. Creating a test netflow v5 flow with nerdalert/nflow-generator. The topic is created in Kafka but no data is written to the topic. I see the following error in the logs:

kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused

I am running vFlow with the command docker run -d -p 4739:4739 -p 4729:4729 -p 6343:6343 -p 9996:9996 -e VFLOW_KAFKA_BROKERS="172.21.0.3:9092" mehrdadrad/vflow

mehrdadrad commented 3 years ago

interesting! can you send me the full log?

saloni-vunet commented 3 years ago 
vFlow running with PID 9.
[vflow] 2020/12/01 13:48:46 open /etc/vflow/vflow.conf: no such file or directory
[vflow] 2020/12/01 13:48:46 Welcome to vFlow v.0.7.0 Apache License 2.0
[vflow] 2020/12/01 13:48:46 Copyright (C) 2018 Verizon. github.com/VerizonDigital/vflow
[vflow] 2020/12/01 13:48:47 starting prometheus http server ...
[vflow] 2020/12/01 13:48:47 ipfix is running (UDP: listening on [::]:4739 workers#: 200)
[vflow] 2020/12/01 13:48:47 sFlow is running (UDP: listening on [::]:6343 workers#: 200)
[vflow] 2020/12/01 13:48:47 ipfix RPC enabled
[vflow] 2020/12/01 13:48:47 open /etc/vflow/mq.conf: no such file or directory
[vflow] 2020/12/01 13:48:47 open /etc/vflow/mq.conf: no such file or directory
[vflow] 2020/12/01 13:48:47 start producer: Kafka, brokers: [172.21.0.3:9092], topic: vflow.sflow
[vflow] 2020/12/01 13:48:47 netflow v9 is running (UDP: listening on [::]:4729 workers#: 200)
[vflow] 2020/12/01 13:48:47 open /etc/vflow/mq.conf: no such file or directory
[vflow] 2020/12/01 13:48:47 netflow v5 is running (UDP: listening on [::]:9996 workers#: 200)
[vflow] 2020/12/01 13:48:47 open /etc/vflow/mq.conf: no such file or directory
[vflow] 2020/12/01 13:48:47 start producer: Kafka, brokers: [172.21.0.3:9092], topic: vflow.ipfix
[vflow] 2020/12/01 13:48:47 start producer: Kafka, brokers: [172.21.0.3:9092], topic: vflow.netflow9
[vflow] 2020/12/01 13:48:47 start producer: Kafka, brokers: [172.21.0.3:9092], topic: vflow.netflow5
[vflow] 2020/12/01 13:49:58 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:09 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:15 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:15 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:15 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:16 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:21 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:22 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:22 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:24 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:24 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:25 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:26 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:27 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:27 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:28 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:29 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:30 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:31 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
[vflow] 2020/12/01 13:50:32 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused

This is the log for vFlow container.

I modified the run command and added the --network option to run vFlow in the same overlay network of Confluent Kafka, but it still results in the exact same error log.

mehrdadrad commented 3 years ago

can you try w/ docker-compose?

docker-compose up
saloni-vunet commented 3 years ago

I tried with docker-compose but still the same error. I believe it is just picking the default broker for producer and not the specified configuration

vflow_1       | [vflow] 2020/12/02 07:43:51 open /etc/vflow/vflow.conf: no such file or directory
vflow_1       | [vflow] 2020/12/02 07:43:51 Welcome to vFlow v.0.7.0 Apache License 2.0
vflow_1       | [vflow] 2020/12/02 07:43:51 Copyright (C) 2018 Verizon. github.com/VerizonDigital/vflow
vflow_1       | [vflow] 2020/12/02 07:43:51 starting prometheus http server ...
vflow_1       | [vflow] 2020/12/02 07:43:51 ipfix is running (UDP: listening on [::]:4739 workers#: 200)
vflow_1       | [vflow] 2020/12/02 07:43:51 netflow v5 is running (UDP: listening on [::]:9996 workers#: 200)
vflow_1       | [vflow] 2020/12/02 07:43:51 sFlow is running (UDP: listening on [::]:6343 workers#: 200)
vflow_1       | [vflow] 2020/12/02 07:43:51 netflow v9 is running (UDP: listening on [::]:4729 workers#: 200)
vflow_1       | [vflow] 2020/12/02 07:43:51 ipfix RPC enabled
vflow_1       | [vflow] 2020/12/02 07:43:51 open /etc/vflow/mq.conf: no such file or directory
vflow_1       | [vflow] 2020/12/02 07:43:51 open /etc/vflow/mq.conf: no such file or directory
vflow_1       | [vflow] 2020/12/02 07:43:51 open /etc/vflow/mq.conf: no such file or directory
vflow_1       | [vflow] 2020/12/02 07:43:51 open /etc/vflow/mq.conf: no such file or directory
vflow_1       | [vflow] 2020/12/02 07:43:51 start producer: Kafka, brokers: [broker:9092], topic: vflow.sflow
vflow_1       | [vflow] 2020/12/02 07:43:51 start producer: Kafka, brokers: [broker:9092], topic: vflow.netflow5
vflow_1       | [vflow] 2020/12/02 07:43:51 start producer: Kafka, brokers: [broker:9092], topic: vflow.netflow9
vflow_1       | [vflow] 2020/12/02 07:43:51 start producer: Kafka, brokers: [broker:9092], topic: vflow.ipfix
vflow_1       | [vflow] 2020/12/02 07:52:33 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:35 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:35 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:37 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:37 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:38 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:38 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:40 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:41 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:41 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:43 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused
vflow_1       | [vflow] 2020/12/02 07:52:44 kafka: Failed to produce message to topic vflow.netflow5: dial tcp 127.0.0.1:9092: connect: connection refused