bdorr1105
commented
3 years ago 
Here is a picture of the logs as well
Closed bdorr1105 closed 3 years ago
bdorr1105
commented
3 years ago
Here is a picture of the logs as well
robcowart
commented
3 years ago Please make sure that you let it run for a while to ensure that template records have been received. Some device, send templates very infrequently. For example, some Fortinet devices only send templates every 15-30 minutes.
If sufficient time has been allowed for templates to be received, this can result when the flow record includes fields that the Logstash Netflow codec doesn't support. I would need to see a PCAP to confirm this. If you would like me to take a look you can send a PCAP, which must include templates, to elastiflow@gmail.com.
bdorr1105
commented
3 years ago Ok, I am using pfsense softflow plugin, I will stop and restart the service to see if it will push out a template and if this is still occurring, I will try to capture a pcap for further investigation.
bdorr1105
commented
3 years ago I am on the latest 7.10 elastic stack, there could be an issue of versions I am assuming. I am just starting to really try to get into this, is there a specific yum install elastic specifying the version? Also, not sure how to and if I need to pull a specific elastiflow zip that has the correct version.
Essentially, I am eluding to that I am the moron in this situation trying to learn it, I know it has to be something I did. This is where I pulled the zip...cd
sudo git clone https://github.com/robcowart/elastiflow.git
I used this as a guide for the majority of it
bdorr1105
commented
3 years ago I sent a PCAP to you
robcowart
commented
3 years ago A quick visual inspection doesn’t reveal any issues with the records in the PCAP. I will try to replay it to confirm, but I will need a day or two to get to that.
I really can't speak to the accuracy of any guides out there. I suggest that you follow the instructions in INSTALL.md. You should also be using a release, not the master branch.
bdorr1105
commented
3 years ago I think that is probably the issue, I think I was using version 5 beta of elastiflow, which makes sense. I was able to figure out the docker container with ubuntu. I had a really difficult time trying to integrate xpack in that. I had to scour the world wide web. I am going to close this out because I was able to figure out the issue with Docker. I had netflow v10 yesterday which there is a statement that this is IPFIX and i saw that in the config that was a different port versus v5/9 being to what i set at 9995. All is well currently, I was able to really rough xpack working. I just wanted to have the ability to log in versus it just being welcome to anyone. I am going to lock the device down on the host based firewall and then figure how to put nginx in front of it and enable SSL. Thanks for all the help. I am learning a good bit, and prefer docker to avoid issues with OS updates. I think docker is the way to go. It is just hard to get xpack to work so far with it. Thanks
robcowart
commented
3 years ago @bdorr1105 the v5 beta is an all new collector that completely removes Logstash from the solution. Your questions and logs are for Logstash. You may want to take a look at the new collector and join the ElastiFlow Community Slack if you haven't already. More details are here... https://www.elastiflow.com/get-started
Also, setting up the Elastic Stack with X-Pack Basic for TLS and login support is not difficult, even when using Docker. Some time between now and the new year I will be making a video that walks through everything. If possible you may want to wait for that.
bdorr1105
commented
3 years ago In trying to integrate xpack, I edited the docker-compose.yml and added them to the environment section, and when I would go into each container utilizing docker exec -it elastiflow-nameofwhatever sh -> only on elastiflow did I have root privilege. I can't use OpenSSL on the kibana container to generate a ssl cert. additionally, for enabling a username and password for kibana, I still had to run the elastiflow-setup-password interactive to ensure the password was actually set. When setting the variables, I did not see evidence of these settings being enabled in the yml files of each container. I still had to manually log into to each container to enable xpack security and set the password. I would love to see a video on it for sure especially after everything I went through trying to get it to work. Thanks
I installed this on Cent OS-7, been having all kinds of trouble with docker, I do not have the right permissions to tail the logstash log on docker and I am not sure what the creds are to elevate. I am not receiving any logs via docker, and currently, having the same issue on this set up.
When tailing the log, i see the correct listeners start and correct ports. The tcpdump shows I am receiving the netflow logs, but logstash seems to not understand how to process them. This is what my log is showing
[2020-12-19T13:22:03,786][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 1024 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute. [2020-12-19T13:22:04,177][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 1024 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute. [2020-12-19T13:22:04,262][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 1024 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute. [2020-12-19T13:22:04,493][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 1024 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute. [2020-12-19T13:22:05,297][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 1024 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute. [2020-12-19T13:22:05,531][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 1024 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute. [2020-12-19T13:22:05,642][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 1024 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute. [2020-12-19T13:22:06,311][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 1024 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute. [2020-12-19T13:22:06,486][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 1024 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute. [2020-12-19T13:22:07,565][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 1024 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute.