No option for netflow.fwd_flow_delta_bytes

[dfdalamar]

This is the oddest issue I have seen with this tool. When I am tracking the cflow in tshark, it reports as initiator octets and destination octets. When I look at the flow in kibana under discover it lists it as netflow.fwd_flow_delta_bytes and netflow.rev_flow_delta_bytes. It is not converting it to flow.bytes for some reason. Where I used to have an option to redo the visualization as fwd_flow that is gone.

Also, in my overview I am getting this error. I am not sure if it is related or not. I have tried specifying the new flow since I renamed it. I can change the name back easy enough.

Visualize: Request to Elasticsearch failed: {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"","phase":"fetch","grouped":true,"failed_shards":[],"caused_by":{"type":"aggregation_execution_exception","reason":"Merging/Reducing the aggregations failed when computing the aggregation [2] because the field you gave in the aggregation query existed as two different types in two different indices"}},"status":503}

[robcowart]

The error is related to the older indices from the previous version having a slightly different schema. This is an example of the "anomalies" I spoke of in the other issue.

I am not sure what you mean by "redo the visualization". If you want me to take a look at the PCAP, you can send it to me... rob{at)koiossian(dot)com

[dfdalamar]

In the previous version, I could use the field fwd_flow_delta_bytes to make my visualizations. Now that is not an option because of the template change.

There is something not processing right, I just don't know what it is. Let me email you my conf files. The flows are coming in and whatnot, just labeled wrong based on what tool you look at.

[robcowart]

Closing this issue as it was related to using a "modified" deployment of ElastiFlow which broke things.

[dfdalamar]

Rob,

I have done the reinstall and it is not gathering flows. It cannot bind the port. The only change I have made since having it work is that I disabled the netflow index with that conf file and enabled the elastiflow index. I am changing the port now to see if that makes any difference.

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 3:47 AM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

Closed #97https://github.com/robcowart/elastiflow/issues/97.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#event-1631302192, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyRNuHAV6c92IM2jy5r70q7Rc59k4ks5tzSsbgaJpZM4UABcK.

[robcowart]

What do you mean by "I disabled the netflow index with that conf file and enabled the elastiflow index"?

[dfdalamar]

Rob,

In my pipelines.yml file. It was configured for netflow with a netflow- index while I was testing the base install. You had told me that the netflow configuration file was what was causing my issues.

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 9:01 AM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

What do you mean by "I disabled the netflow index with that conf file and enabled the elastiflow index"?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389858124, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyaZP3vaf5pr4T-Py9-YYZ8uRjcTJks5tzXSrgaJpZM4UABcK.

[dfdalamar]

Rob,

I turned the original netflow index back on and flows are coming through. Does your environment have multiple pipelines, the netflow and the elastiflow? I did have to make a change to the configuration files. In the instructions it says the default directory is /etc/logstash/dictionary, geoipdbs etc. But in the conf files it is /etc/logstash/elastiflow/dictionary etc. So I removed the reference in the conf files to the elastiflow folder.

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 9:01 AM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

What do you mean by "I disabled the netflow index with that conf file and enabled the elastiflow index"?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389858124, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyaZP3vaf5pr4T-Py9-YYZ8uRjcTJks5tzXSrgaJpZM4UABcK.

[dfdalamar]

Rob,

After installing the netflow codec, do you run?

sudo ./logstash --modules netflow --setup -M "netflow.var.kibana.host=http://10.15.41.10:5601" -M "netflow.var.input.udp.port=9595" -M "netflow.var.elasticsearch.hosts=http://10.15.41.10:9200"

Thanks, Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 9:01 AM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

What do you mean by "I disabled the netflow index with that conf file and enabled the elastiflow index"?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389858124, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyaZP3vaf5pr4T-Py9-YYZ8uRjcTJks5tzXSrgaJpZM4UABcK.

[robcowart]

Can you send me a copy of your /etc/systemd/system/logstash.service.d/elastiflow.conf ?

[robcowart]

sudo ./logstash --modules netflow --setup -M "netflow.var.kibana.host=http://10.15.41.10:5601" -M "netflow.var.input.udp.port=9595" -M "netflow.var.elasticsearch.hosts=http://10.15.41.10:9200"

No don't do this.

[robcowart]

I did have to make a change to the configuration files. In the instructions it says the default directory is /etc/logstash/dictionary, geoipdbs etc. But in the conf files it is /etc/logstash/elastiflow/dictionary etc. So I removed the reference in the conf files to the elastiflow folder.

The default directory in the README was wrong. They should all start with /etc/logstash/elastiflow. But that shouldn't matter. Even if your files were in a different place, you change that by setting the environment variable to the correct location. On a recent RedHat/CentOS or Ubuntu release that would be in /etc/systemd/system/logstash.service.d/elastiflow.conf. A starter file for this is available in the repo.

[dfdalamar]

Rob,

I have not been able to completely undo that command. I have completely uninstalled logstash and starting from scratch. I was able to show through using a temp conf file stdout that it was binding. But yeah. Reinstalling fresh and will not run that command. Will update shortly.

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 11:59 AM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

sudo ./logstash --modules netflow --setup -M "netflow.var.kibana.host=http://10.15.41.10:5601" -M "netflow.var.input.udp.port=9595" -M "netflow.var.elasticsearch.hosts=http://10.15.41.10:9200"

No don't do this.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389918245, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyRgyp5xIAVFG1eM66TUmzSD8N5fAks5tzZ5XgaJpZM4UABcK.

[dfdalamar]

Rob,

I have done the reinstall. I did not use the command for netflow. I am still unable to bind the port. Here is my /etc/systemd/system/logstash.services.d/elastiflow.conf

compliance with the License. You may obtain a copy of the License at:

#

http://www.koiossian.com/public/robert_cowart_public_license.txt

#

Software distributed under the License is distributed on an "AS IS" basis,

WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for

the specific language governing rights and limitations under the License.

#

The Original Source Code was developed by Robert Cowart. Portions created by

Robert Cowart are Copyright (C)2018 Robert Cowart. All Rights Reserved.

------------------------------------------------------------------------------

[Service]

ElastiFlow global configuration

Environment="ELASTIFLOW_DICT_PATH=/etc/logstash/elastiflow/dictionaries" Environment="ELASTIFLOW_TEMPLATE_PATH=/etc/logstash/elastiflow/templates" Environment="ELASTIFLOW_GEOIP_DB_PATH=/etc/logstash/elastiflow/geoipdbs" Environment="ELASTIFLOW_GEOIP_CACHE_SIZE=8192" Environment="ELASTIFLOW_GEOIP_LOOKUP=true" Environment="ELASTIFLOW_ASN_LOOKUP=true" Environment="ELASTIFLOW_KEEP_ORIG_DATA=true"

Name resolution option

Environment="ELASTIFLOW_RESOLVE_IP2HOST=false" Environment="ELASTIFLOW_NAMESERVER=127.0.0.1" Environment="ELASTIFLOW_DNS_HIT_CACHE_SIZE=25000" Environment="ELASTIFLOW_DNS_HIT_CACHE_TTL=900" Environment="ELASTIFLOW_DNS_FAILED_CACHE_SIZE=75000" Environment="ELASTIFLOW_DNS_FAILED_CACHE_TTL=3600"

Elasticsearch connection settings

Environment="ELASTIFLOW_ES_HOST=10.15.41.10"

Netflow - IPv4

Environment="ELASTIFLOW_NETFLOW_IPV4_HOST=192.168.200.1" Environment="ELASTIFLOW_NETFLOW_IPV4_PORT=9596"

Netflow - IPv6

Environment="ELASTIFLOW_NETFLOW_IPV6_HOST=[::]"

Environment="ELASTIFLOW_NETFLOW_IPV6_PORT=52055"

Netflow - UDP input options

Environment="ELASTIFLOW_NETFLOW_UDP_WORKERS=4" Environment="ELASTIFLOW_NETFLOW_UDP_QUEUE_SIZE=4096"

Netflow timestamp options

Environment="ELASTIFLOW_NETFLOW_LASTSW_TIMESTAMP=false" Environment="ELASTIFLOW_NETFLOW_TZ=UTC"

sFlow - IPv4

Environment="ELASTIFLOW_SFLOW_IPV4_HOST=0.0.0.0"

Environment="ELASTIFLOW_SFLOW_IPV4_PORT=6343"

sFlow - IPv6

Environment="ELASTIFLOW_SFLOW_IPV6_HOST=[::]"

Environment="ELASTIFLOW_SFLOW_IPV6_PORT=56343"

sFlow - UDP input options

Environment="ELASTIFLOW_SFLOW_UDP_WORKERS=4"

Environment="ELASTIFLOW_SFLOW_UDP_QUEUE_SIZE=4096"

IPFIX - IPv4

Environment="ELASTIFLOW_IPFIX_TCP_IPV4_HOST=0.0.0.0"

Environment="ELASTIFLOW_IPFIX_TCP_IPV4_PORT=4739"

Environment="ELASTIFLOW_IPFIX_UDP_IPV4_HOST=0.0.0.0"

Environment="ELASTIFLOW_IPFIX_UDP_IPV4_PORT=4739"

IPFIX - IPv6

Environment="ELASTIFLOW_IPFIX_TCP_IPV6_HOST=[::]"

Environment="ELASTIFLOW_IPFIX_TCP_IPV6_PORT=54739"

Environment="ELASTIFLOW_IPFIX_UDP_IPV6_HOST=[::]"

Environment="ELASTIFLOW_IPFIX_UDP_IPV6_PORT=54739"

IPFIX - UDP input options

Environment="ELASTIFLOW_IPFIX_UDP_WORKERS=4"

Environment="ELASTIFLOW_IPFIX_UDP_QUEUE_SIZE=4096"

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 12:02 PM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

I did have to make a change to the configuration files. In the instructions it says the default directory is /etc/logstash/dictionary, geoipdbs etc. But in the conf files it is /etc/logstash/elastiflow/dictionary etc. So I removed the reference in the conf files to the elastiflow folder.

The default directory in the README was wrong. They should all start with /etc/logstash/elastiflow. But that shouldn't matter. Even if your files were in a different place, you change that by setting the environment variable to the correct location. On a recent RedHat/CentOS or Ubuntu release that would be in /etc/systemd/system/logstash.service.d/elastiflow.conf. A starter file for this is available in the repo.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389919476, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyeAhpQk8c4_a_aaN4-OyRvrxxe3dks5tzZ8pgaJpZM4UABcK.

[robcowart]

What is the error you are getting?

[dfdalamar]

Rob,

[2018-05-17T13:07:23,836][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"192.168.200.1:9596"} [2018-05-17T13:07:23,839][WARN ][logstash.inputs.udp ] UDP listener died {:exception=>#<Errno::EADDRNOTAVAIL: Cannot assign requested address - bind - Cannot assign requested address>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:190:in bind'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.3/lib/logstash/inputs/udp.rb:102:inudp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.3/lib/logstash/inputs/udp.rb:58:in run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514:ininputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507:in `block in start_input'"]}

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 1:07 PM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

What is the error you are getting?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389939544, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyQKFPsGQ3JzAh2cvLMsluxFMdnOXks5tza40gaJpZM4UABcK.

[robcowart]

run ifconfig and send the output

[dfdalamar]

Rob,

librenms@librenms:~$ ifconfig ens34 Link encap:Ethernet HWaddr 00:50:56:9e:0e:d8 inet addr:10.15.41.10 Bcast:10.15.43.255 Mask:255.255.252.0 inet6 addr: fe80::250:56ff:fe9e:ed8/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3663889 errors:0 dropped:0 overruns:0 frame:0 TX packets:2517759 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1696276841 (1.6 GB) TX bytes:469750071 (469.7 MB)

lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:7514065 errors:0 dropped:0 overruns:0 frame:0 TX packets:7514065 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1 RX bytes:2066075751 (2.0 GB) TX bytes:2066075751 (2.0 GB)

librenms@librenms:~$

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 1:13 PM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

run ifconfig and send the output

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389941595, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyaydrirVYHFvWddW3bxuB9jCbjiAks5tza_PgaJpZM4UABcK.

[robcowart]

You don't have an IP address 192.168.200.1, which is what you told ElastiFlow to listen on.

[robcowart]

It is easiest just to leave the default, which was 0.0.0.0 (all IPs)

[dfdalamar]

Rob,

I thought the listen IP address was the one to listen for configs. Let me adjust that.

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 1:15 PM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

You don't have an IP address 192.168.200.1, which is what you told ElastiFlow to listen on.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389942096, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyWCDGQogoXoNmWKgAEp_x_BmjT-Qks5tzbA4gaJpZM4UABcK.

[dfdalamar]

Rob,

Got some new things in the log. I have never seen the dictionary file being refreshed.

May 17 13:28:25 librenms logstash[21441]: [2018-05-17T13:28:25,516][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 261 from source id 0, because no template to decode it with has been receiv May 17 13:28:26 librenms logstash[21441]: [2018-05-17T13:28:26,454][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 263 from source id 0, because no template to decode it with has been receiv May 17 13:28:26 librenms logstash[21441]: [2018-05-17T13:28:26,455][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 263 from source id 0, because no template to decode it with has been receiv May 17 13:32:52 librenms logstash[21441]: [2018-05-17T13:32:52,577][INFO ][logstash.filters.translate] refreshing dictionary file May 17 13:33:07 librenms logstash[21441]: [2018-05-17T13:33:07,191][INFO ][logstash.filters.translate] refreshing dictionary file May 17 13:33:08 librenms logstash[21441]: [2018-05-17T13:33:08,019][INFO ][logstash.filters.translate] refreshing dictionary file May 17 13:33:08 librenms logstash[21441]: [2018-05-17T13:33:08,047][INFO ][logstash.filters.translate] refreshing dictionary file May 17 13:33:08 librenms logstash[21441]: [2018-05-17T13:33:08,201][INFO ][logstash.filters.translate] refreshing dictionary file May 17 13:33:08 librenms logstash[21441]: [2018-05-17T13:33:08,209][INFO ][logstash.filters.translate] refreshing dictionary file May 17 13:33:08 librenms logstash[21441]: [2018-05-17T13:33:08,332][INFO ][logstash.filters.translate] refreshing dictionary file

Is this normal? I know the flowset stuff is waiting for the template.

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 1:18 PM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

It is easiest just to leave the default, which was 0.0.0.0 (all IPs)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389943069, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyXOvYWf8pKwgnKHuwQ1r3Tb2rTBoks5tzbDsgaJpZM4UABcK.

[dfdalamar]

Rob,

Got some things working. It was the 0.0.0.0 and the netflow junk. In my visualizations I am getting this error. I have checked permissions and all look good.

The aggregations key is missing from the response, check your permissions for this request.

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 1:18 PM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

It is easiest just to leave the default, which was 0.0.0.0 (all IPs)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389943069, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyXOvYWf8pKwgnKHuwQ1r3Tb2rTBoks5tzbDsgaJpZM4UABcK.

[robcowart]

Yes, dictionary refresh every 10 mins by default. This allows you to update dictionary files without a restart.

I have never seen this error before. Where do you see it?

[robcowart]

A quick google of that error, indicates it is related to a lack of Elasticsearch resources. What kind of system are you running on? CPUs?, RAM?, type of Storage? How much JVM heap space did you give to Elasticsearch and Logstash? Is everything on the same server? How many flows/sec is your device sending?

[dfdalamar]

Rob,

That is in the visualizations area of kibana. I am running everything on the same server. I have 58 gigs of free space. I need to delete old indexes

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 2:07 PM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

A quick google of that error, indicates it is related to a lack of Elasticsearch resources. What kind of system are you running on? CPUs?, RAM?, type of Storage? How much JVM heap space did you give to Elasticsearch and Logstash? Is everything on the same server? How many flows/sec is your device sending?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389957983, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TycTUfJw2IzyeQY_0H0LGX7lxweBBks5tzbx4gaJpZM4UABcK.

[dfdalamar]

Rob,

I am sending between 8 and 10k flows every 30 seconds.

Stats

librenms@librenms:/etc/logstash$ mpstat Linux 4.4.0-124-generic (librenms) 17/05/18 _x8664 (4 CPU)

14:32:27 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 14:32:27 all 38.40 17.70 10.06 3.09 0.00 0.83 0.00 0.00 0.00 29.92

librenms@librenms:/etc/logstash$ top top - 14:33:41 up 5:56, 1 user, load average: 12.22, 10.58, 8.22 Tasks: 369 total, 3 running, 366 sleeping, 0 stopped, 0 zombie %Cpu(s): 18.2 us, 10.8 sy, 69.7 ni, 0.7 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st KiB Mem : 8174896 total, 147344 free, 5114436 used, 2913116 buff/cache KiB Swap: 1044476 total, 675360 free, 369116 used. 2640232 avail Mem

I gave the jvm.options the recommended 2g.

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 2:07 PM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

A quick google of that error, indicates it is related to a lack of Elasticsearch resources. What kind of system are you running on? CPUs?, RAM?, type of Storage? How much JVM heap space did you give to Elasticsearch and Logstash? Is everything on the same server? How many flows/sec is your device sending?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389957983, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TycTUfJw2IzyeQY_0H0LGX7lxweBBks5tzbx4gaJpZM4UABcK.

[robcowart]

How much JVM space did you give to Elasticsearch?

You can already see that your CPU is pegged... 0.7 id. Is this a VM or physical HW?

At 300-400 flows per second, and everything on the same system, you should start out with something like 8 cores and 32GB. Give Elasticsearch 8GB JVM heap. Give Logstash 2GB of JVM heap. You may need to bump up the heap size of both, but you should be OK with this, which will leave the remaining RAM for the OS to use a page cache.

At 10K flows per 30sec you will write approx 12GB per day. You will need to size storage accordingly for the volume of data you want to retain.

[dfdalamar]

Rob,

Thank you!!! I will boost my RAM. The will get mad at me for boosting the cores. This is a VM. I have been averaging about 6 gig per day per flow.

Are you on Venmo or Patreon etc? Your help has been amazing.

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 3:00 PM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

How much JVM space did you give to Elasticsearch?

You can already see that your CPU is pegged... 0.7 id. Is this a VM or physical HW?

At 300-400 flows per second, and everything on the same system, you should start out with something like 8 cores and 32GB. Give Elasticsearch 8GB JVM heap. Give Logstash 2GB of JVM heap. You may need to bump up the heap size of both, but you should be OK with this, which will leave the remaining RAM for the OS to use a page cache.

At 10K flows per 30sec you will write approx 12GB per day. You will need to size storage accordingly for the volume of data you want to retain.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389972585, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyUeL9KwQV06po8f32H6DmnvKCka5ks5tzcjQgaJpZM4UABcK.

[dfdalamar]

Rob,

I boosted elasticsearch to 5 gigs. Which is nuts because I have not needed to do this before. Also deleted all indices but the new ones. I am getting these errors.

Could not locate that index-pattern-field (id: flow.tcp_flags)

TypeError: "field" is a required parameter at FieldParamTypeProvider.FieldParamType.write (http://10.15.41.10:5601/bundles/commons.bundle.js?v=16627:1:1266731) at http://10.15.41.10:5601/bundles/commons.bundle.js?v=16627:1:229815 at AggParams.forEach () at AggParams.AggTypesAggParamsProvider.AggParams.write (http://10.15.41.10:5601/bundles/commons.bundle.js?v=16627:1:229773) at AggConfig.VisAggConfigProvider.AggConfig.write (http://10.15.41.10:5601/bundles/commons.bundle.js?v=16627:1:50265) at AggConfig.VisAggConfigProvider.AggConfig.toDsl (http://10.15.41.10:5601/bundles/commons.bundle.js?v=16627:1:51200) at http://10.15.41.10:5601/bundles/commons.bundle.js?v=16627:1:1456497 at Array.forEach () at AggConfigs.VisAggConfigsProvider.AggConfigs.toDsl (http://10.15.41.10:5601/bundles/commons.bundle.js?v=16627:1:1456280) at http://10.15.41.10:5601/bundles/commons.bundle.js?v=16627:1:419792

I have seen the type thing is some kind of bug. Have you seen this?

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 3:00 PM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

How much JVM space did you give to Elasticsearch?

You can already see that your CPU is pegged... 0.7 id. Is this a VM or physical HW?

At 300-400 flows per second, and everything on the same system, you should start out with something like 8 cores and 32GB. Give Elasticsearch 8GB JVM heap. Give Logstash 2GB of JVM heap. You may need to bump up the heap size of both, but you should be OK with this, which will leave the remaining RAM for the OS to use a page cache.

At 10K flows per 30sec you will write approx 12GB per day. You will need to size storage accordingly for the volume of data you want to retain.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389972585, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyUeL9KwQV06po8f32H6DmnvKCka5ks5tzcjQgaJpZM4UABcK.

[dfdalamar]

Rob,

I think I found the issue.

When I look at the visualization Elastiflow: Sources (bytes) - donut, it does not work and is missing a field. Looking at the code, I see that it is looking for field "field": "flow.src_hostname". For some reason that does not exist. For me that field is flow.src_hostname.keyword. I am not sure why it has the .keyword at the end, but that seems to be the case with most of my fields. Any idea where that came from?

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 3:00 PM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

How much JVM space did you give to Elasticsearch?

You can already see that your CPU is pegged... 0.7 id. Is this a VM or physical HW?

At 300-400 flows per second, and everything on the same system, you should start out with something like 8 cores and 32GB. Give Elasticsearch 8GB JVM heap. Give Logstash 2GB of JVM heap. You may need to bump up the heap size of both, but you should be OK with this, which will leave the remaining RAM for the OS to use a page cache.

At 10K flows per 30sec you will write approx 12GB per day. You will need to size storage accordingly for the volume of data you want to retain.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389972585, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyUeL9KwQV06po8f32H6DmnvKCka5ks5tzcjQgaJpZM4UABcK.

[dfdalamar]

Rob,

I have almost everything working. I have a few fields that are not listed in the index. Packets, geo location pin something or other. I do get lat and lon for the geo, just not sure how to take that data and make it the pin. I did have to completely edit the dashboard.json file for the fields I was having issues with. In case someone else runs into this issue, here is my dashboard.

The experimental graphs that have the curvy left to right flows are working beautifully.

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Thursday, May 17, 2018 3:00 PM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

How much JVM space did you give to Elasticsearch?

You can already see that your CPU is pegged... 0.7 id. Is this a VM or physical HW?

At 300-400 flows per second, and everything on the same system, you should start out with something like 8 cores and 32GB. Give Elasticsearch 8GB JVM heap. Give Logstash 2GB of JVM heap. You may need to bump up the heap size of both, but you should be OK with this, which will leave the remaining RAM for the OS to use a page cache.

At 10K flows per 30sec you will write approx 12GB per day. You will need to size storage accordingly for the volume of data you want to retain.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-389972585, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyUeL9KwQV06po8f32H6DmnvKCka5ks5tzcjQgaJpZM4UABcK.

[robcowart]

Your issues with .keyword fields mean that the index template was not applied when the index was created. The most likely reason being that the template wasn't uploaded from Logstash to Elasticsearch. What this means is that you still are having issues when installing and configuring.

[dfdalamar]

Rob,

I did notice the index did not get created. I am using this command to generate the index etc.

curl -X POST http://10.15.41.10:5601/api/saved_objects/index-pattern/elastiflow-* -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @/etc/kibana/elastiflow.index_pattern.json

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Monday, May 21, 2018 11:02 AM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

Your issues with .keyword fields mean that the index template was not applied when the index was created. The most likely reason being that the template wasn't uploaded from Logstash to Elasticsearch. What this means is that you still are having issues when installing and configuring.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-390681397, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TybnI2uQvIwzaKqfXUYYOcAB64sNBks5t0tcRgaJpZM4UABcK.

[dfdalamar]

Rob,

I meant template. I did have to wait quite sometime after starting logstash after adding the index for the template to come in. When did a refresh on the Index Patterns, I notice it removes some and adds the duplicate with the .keyword at the end.

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Monday, May 21, 2018 11:02 AM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

Your issues with .keyword fields mean that the index template was not applied when the index was created. The most likely reason being that the template wasn't uploaded from Logstash to Elasticsearch. What this means is that you still are having issues when installing and configuring.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-390681397, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TybnI2uQvIwzaKqfXUYYOcAB64sNBks5t0tcRgaJpZM4UABcK.

[robcowart]

I don't know what you mean when you say that you had to wait for the Index Template to "come in".

[dfdalamar]

Rob,

After running the command and the index showing up, when I run GET _template the elastiflow one is not there. I have to restart logstash and then wait about 10 minutes and then when I run GET _template it is there.

Thanks,

Jared

---

From: Rob Cowart notifications@github.com Sent: Monday, May 21, 2018 11:31 AM To: robcowart/elastiflow Cc: dfdalamar; Author Subject: Re: [robcowart/elastiflow] No option for netflow.fwd_flow_delta_bytes (#97)

I don't know what you mean when you say that you had to wait for the Index Template to "come in".

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/97#issuecomment-390689819, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ak8TyeReYOYPU6sL88p1322m1HnYoC4xks5t0t21gaJpZM4UABcK.

[njhashmi]

Hi,

I am having same issue on kibana dashboard for elastiflow I get error "The request for this panel failed The aggregations key is missing from the response, check your permissions for this request." I have increased the heap to 8gb but still same issue.