elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
75 stars 3.5k forks source link

On a fresh install of ELK stack - on a brand new ubuntu 18.04 server with nothing else installed except for Java, #10847

Closed davidshirk80 closed 5 years ago

davidshirk80 commented 5 years ago

root@usmlbnetflw:/usr/share/logstash/bin# /usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow Validating /usr/share/logstash/bin/logstash-codec-sflow Unable to download data from https://rubygems.org - Connection reset by peer (https://api.rubygems.org/latest_specs.4.8.gz) ERROR: Installation aborted, verification failed for /usr/share/logstash/bin/logstash-codec-sflow

On Fresh install of Ubuntu 18.04 with the following steps done first:

apt update apt install apt-transport-https apt install -y openjdk-8-jdk export JAVA_HOME=/path/to/java wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list apt update apt-get -y install elasticsearch /bin/systemctl daemon-reload /bin/systemctl enable elasticsearch.service systemctl start elasticsearch.service apt update apt-get -y install kibana /bin/systemctl daemon-reload /bin/systemctl enable kibana.service systemctl start kibana.service apt update apt-get -y install logstash nano /etc/elasticsearch/elasticsearch.yml change to network.host: x.x.x.x systemctl restart elasticsearch nano /etc/kibana/kibana.yml change to serverhost x.x.x.x systemctl restart kibana ufw allow from x.x.x.x to any port 5601 proto tcp nano /etc/logstash/jvm.options change -Xms1g -Xmx1g to -Xms4g -Xmx4g save and exit /usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow

So that is the detailed steps on how I got there. It is funny because I installed earlier last week with the exact same setup with no issues, so this was unexpected.

For security vulnerabilities please only send reports to security@elastic.co. See https://www.elastic.co/community/security for more information.

Logstash Plugins are located in a different organization: https://github.com/logstash-plugins. For bugs on specific Logstash plugins, for example, if Redis Output has a defect, please open it in the respective Redis Output repository.

For all general issues, please provide the following details for fast resolution:

yaauie commented 5 years ago

The following error indicates that you had a networking issue while installing a plugin:

Validating /usr/share/logstash/bin/logstash-codec-sflow
Unable to download data from https://rubygems.org - Connection reset by peer (https://api.rubygems.org/latest_specs.4.8.gz)
ERROR: Installation aborted, verification failed for /usr/share/logstash/bin/logstash-codec-sflow

The HTTP connection was reset by a peer, causing the installation of the plugin to fail. This could have to do with your netfilter firewall, or could otherwise be related to your network topology.

davidshirk80 commented 5 years ago

I know it was reset by the peer.......nothing wrong with my side of the NW.... root@usmlbnetflw:/usr/share/logstash/bin# ping api.rubygems.org PING rubygems.org (151.101.128.70) 56(84) bytes of data. 64 bytes from 151.101.128.70 (151.101.128.70): icmp_seq=1 ttl=55 time=12.2 ms 64 bytes from 151.101.128.70 (151.101.128.70): icmp_seq=2 ttl=55 time=11.5 ms ^C --- rubygems.org ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 11.532/11.900/12.269/0.384 ms root@usmlbnetflw:/usr/share/logstash/bin# ping google.com PING google.com (172.217.2.78) 56(84) bytes of data. 64 bytes from ord08s13-in-f14.1e100.net (172.217.2.78): icmp_seq=1 ttl=54 time=18.7 ms 64 bytes from ord08s13-in-f14.1e100.net (172.217.2.78): icmp_seq=2 ttl=54 time=18.7 ms ^C --- google.com ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 18.702/18.717/18.733/0.137 ms

I can literally download anything from anywhere else. All plugins downloaded on install except for that one....: logstash-codec-cef logstash-codec-collectd logstash-codec-dots logstash-codec-edn logstash-codec-edn_lines logstash-codec-es_bulk logstash-codec-fluent logstash-codec-graphite logstash-codec-json logstash-codec-json_lines logstash-codec-line logstash-codec-msgpack logstash-codec-multiline logstash-codec-netflow logstash-codec-plain logstash-codec-rubydebug logstash-filter-aggregate logstash-filter-anonymize logstash-filter-cidr logstash-filter-clone logstash-filter-csv logstash-filter-date logstash-filter-de_dot logstash-filter-dissect logstash-filter-dns logstash-filter-drop logstash-filter-elasticsearch logstash-filter-fingerprint logstash-filter-geoip logstash-filter-grok logstash-filter-http logstash-filter-jdbc_static logstash-filter-jdbc_streaming logstash-filter-json logstash-filter-kv logstash-filter-memcached logstash-filter-metrics logstash-filter-mutate logstash-filter-ruby logstash-filter-sleep logstash-filter-split logstash-filter-syslog_pri logstash-filter-throttle logstash-filter-translate logstash-filter-truncate logstash-filter-urldecode logstash-filter-useragent logstash-filter-xml logstash-input-azure_event_hubs logstash-input-beats logstash-input-dead_letter_queue logstash-input-elasticsearch logstash-input-exec logstash-input-file logstash-input-ganglia logstash-input-gelf logstash-input-generator logstash-input-graphite logstash-input-heartbeat logstash-input-http logstash-input-http_poller logstash-input-imap logstash-input-jdbc logstash-input-kafka logstash-input-pipe logstash-input-rabbitmq logstash-input-redis logstash-input-s3 logstash-input-snmp logstash-input-snmptrap logstash-input-sqs logstash-input-stdin logstash-input-syslog logstash-input-tcp logstash-input-twitter logstash-input-udp logstash-input-unix logstash-output-cloudwatch logstash-output-csv logstash-output-elastic_app_search logstash-output-elasticsearch logstash-output-email logstash-output-file logstash-output-graphite logstash-output-http logstash-output-kafka logstash-output-lumberjack logstash-output-nagios logstash-output-null logstash-output-pagerduty logstash-output-pipe logstash-output-rabbitmq logstash-output-redis logstash-output-s3 logstash-output-sns logstash-output-sqs logstash-output-stdout logstash-output-tcp logstash-output-udp logstash-output-webhdfs logstash-patterns-core

davidshirk80 commented 5 years ago

So my question is why that particular codec did not download with the rest - and i have re-wiped and re-installed 3 times now with the same outcome. Also to note - I got these upon initial install: When setting up elasticsearch 6.8: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME

and when installing logstash: Setting up logstash (1:6.8.0-1) ... Using provided startup.options file: /etc/logstash/startup.options /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated

davidshirk80 commented 5 years ago

Is there anything I can do to help? I like digging and am generally good at finding things - I just need to know where to look!

davidshirk80 commented 5 years ago

FYI - Tried again this morning and it failed again - were you able to duplicate?

krasa commented 5 years ago

Worked fine for me.

davidshirk80 commented 5 years ago

Yeah, this can be closed.....With my deepest apologies. Seriously - I had one of the worst brain skips in my life on this one - ssl......You would think after figuring this out when installing Oxidized, that I would have known better here. Again, apologies for any inconvenience caused!

davidshirk80 commented 5 years ago

Sorry - but I fixed the ssl issue - and the result is the same. It should be noted that when I run the same thing on VM, it works just fine. I have tried 2 bare metal servers though with a new install of ubuntu (apt updated all the good stuff), and I get a failure as noted above.

davidshirk80 commented 5 years ago

Again: "Validating logstash-codec-sflow Unable to download data from https://rubygems.org - Connection reset by peer (https://api.rubygems.org/latest_specs.4.8.gz) ERROR: Installation aborted, verification failed for logstash-codec-sflow " only happen on the bare metal installs and not the VM's using the exact same instructions.

yaauie commented 5 years ago

Since this is a debugging question and not a reproducible bug report or a feature request, I am closing this issue.

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

-- Logstash Issue Template

Hopefully the following will help you get on track, but if you need more, please open a thread on the forum.


I would attempt cutting Logstash out of the equation; can you get that resource with curl? If not, does the verbose output give you any clues?

curl --verbose --output latest_specs.4.8.gz https://api.rubygems.org/latest_specs.4.8.gz
davidshirk80 commented 5 years ago

I will go to the forum - thanks! - FYI:

Anyway - thank you for pointing me in the right direction, and sorry for false alarm!

davidshirk80 commented 5 years ago

FYI - Logstash sflow install cannot be ran if your server is using TSL v1.3 - only 1.2 . So while it is a problem I had with my server - it should be noted that is only because my server uses the current version 1.3 as is normal for an up to date ubuntu install. 1.2 should be specified in install directions for logstash - it is not.

davidshirk80 commented 5 years ago

Just not sure where to put this as it only happens when i try to install logstash sflow:

root@xxxxxxxxxxxxx:~# openssl ciphers -v | awk '{print $2}' | sort | uniq SSLv3 TLSv1 TLSv1.2 TLSv1.3

davidshirk80 commented 5 years ago

Aaaaand nm NOT a TLS issue....... And my server is not natted or anything else. So not sure how it is a 'server' issue.....