Empty bytes and packets in graphs, but present in flow records

[v-bashaev]

Hello! Maybe someone has stumbled upon the same issue? I have installed Elastiflow 4 in docker for Cisco IOS netflow (2 rather low-traffic exporters). Whatever changes I make to Logstash part of docker-compose.yml (originally to reduce memory consumption I experimented with Java heap, workers count and qeue), I get the same result: no traffic graphs (replaced by message 'No results displayed because all values equal 0'). Yet I see clients & servers IP and numbers of flows and if I look at 'Flow records' section I see flow records which appear complete with fields:

netflow.in_bytes | 82B netflow.in_pkts | 1

What could be causing no visualization?

Here is my yml file:

version: '3'

services:
  elastiflow-elasticsearch-oss:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: elastiflow4-elastic
    restart: 'no'
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 131072
        hard: 131072
      nproc: 8192
      fsize: -1
    network_mode: host
    volumes:
      - /var/lib/elastiflow_es:/usr/share/elasticsearch/data
    environment:
      # JVM Heap size
      #   - this should be at least 2GB for simple testing, receiving only a few flows per second.
      #   - for production environments upto 31GB is recommended.
      ES_JAVA_OPTS: '-Xms2g -Xmx2g'

      cluster.name: elastiflow

      bootstrap.memory_lock: 'true'

      network.host: 0.0.0.0
      http.port: 9201
      discovery.type: 'single-node'

      indices.query.bool.max_clause_count: 8192
      search.max_buckets: 100000

      action.destructive_requires_name: 'true'

  elastiflow-kibana-oss:
    image: docker.elastic.co/kibana/kibana:7.8.1
    container_name: elastiflow4-kibana
    restart: 'no'
    depends_on:
      - elastiflow-elasticsearch-oss
    network_mode: host
    environment:
      SERVER_HOST: 0.0.0.0
      SERVER_PORT: 5601

      ELASTICSEARCH_HOSTS: "http://127.0.0.1:9201"

      KIBANA_DEFAULTAPPID: "dashboard/653cf1e0-2fd2-11e7-99ed-49759aed30f5"

      LOGGING_QUIET: 'true'

  elastiflow-logstash-oss:
    image: robcowart/elastiflow-logstash:4.0.0
    container_name: elastiflow4-logstash
    restart: 'no'
    depends_on:
      - elastiflow-elasticsearch-oss
    network_mode: host
    ports:        #added by me
      - 9998:9998 #added by me
    environment:
      # JVM Heap size - this MUST be at least 3GB (4GB preferred)
      LS_JAVA_OPTS: '-Xms3g -Xmx3g'

      # ElastiFlow global configuration
      ELASTIFLOW_DEFAULT_APPID_SRCTYPE: "__UNKNOWN"
      ELASTIFLOW_GEOIP_LOOKUP: 'false'
      ELASTIFLOW_ASN_LOOKUP: 'false'

      # Name resolution option
      ELASTIFLOW_RESOLVE_IP2HOST: "true"
      ELASTIFLOW_NAMESERVER: "192.168.10.39"
      ELASTIFLOW_ES_HOST: "127.0.0.1:9201"   #added by me
      ELASTIFLOW_NETFLOW_IPV4_PORT: 9998
      ELASTIFLOW_SFLOW_IPV4_PORT: 6343
      ELASTIFLOW_IPFIX_TCP_IPV4_PORT: 4739

      ELASTIFLOW_NETFLOW_UDP_WORKERS: 4
      ELASTIFLOW_NETFLOW_UDP_QUEUE_SIZE: 4096

[robcowart]

Verify that both the server where ElastiFlow is installed and the devices sending flows are configured for UTC time. You may be receiving data that is simply has a time stamp outside the window you are viewing.

Also, what kind of system is this running on?

[v-bashaev]

Verify that both the server where ElastiFlow is installed and the devices sending flows are configured for UTC time. You may be receiving data that is simply has a time stamp outside the window you are viewing.

Also, what kind of system is this running on?

I have verified that clock is set to local timezone: 18:37:11.757 MSK Wed Dec 9 2020 - on the router Wed Dec 9 18:34:54 MSK 2020 - on the virtual machine

The system runs on a virtual machine in docker, virtual machine has 8GB RAM and 4 cores. I know this is small but it works with only two routers.

What is really strange is that I had Elastiflow 3.5.1 on this very virtual machine and it worked. I constructed the yml from the file I used in 3.5.1 by changing versions and also disabling geo ip and asn.

[robcowart]

If your devices are set for MSK and the Elastic components are expecting everything is UTC, that will be problematic. You will be off by 3 hours.

You mention 3.5.1. Did you just update the containers or did you actually do a completely fresh setup procedure? For example, did you import the 4.0.x dashboards or are you trying to use what was already there from 3.5.1?

[v-bashaev]

What I currently observe is that for instance I want to see some dashboard for 'last 15 minutes'. I open 'flows' and I get No results displayed because all values equal 0, but I also get lists of IPs and flows distribution, it's just they are all zero packets and bytes. If I go to 'flow records' section, I see flows with correct bytes and packets. So I don't understand how a time shift somewhere could be causing this. Literally I can pick an individual flow and it will show 0 bytes in some graph and yet show non-zero packets in flow record details. In 3.5.1 the timezone settings were the same as now and it showed everything just right timewise. As for containers, I deleted the old ones and created the new ones. I also deleted 'nodes' directory in elastiflow_es as I read in documentation that data structures changed between 3.x and 4.0. In Kibana I imported saved objects from file that is currently on Elastiflow project page, which is named for 7.8.x, so it should fit in my scenario I suppose.

[robcowart]

Sorry... I missed the detail of ONLY bytes and packets being zero. You must be receiving Netflow v5. That is related to a bug in 4.0.0 and 4.0.1. I need to merge those fixes and do a new release. Sorry for the confusion.

[v-bashaev]

Thank you very much for letting me know! I switched to netflow 9 for now and bytes/packets are now displayed.