search

elastic / fleet-server

The Fleet server allows managing a fleet of Elastic Agents.
Other
11 stars 81 forks source link

[Fleet / self managed] Error dialing x509: certificate signed by unknown authority | Fleet server not able to connect with elasticsearch #1945

Open nathanatimu opened 2 years ago

nathanatimu commented 2 years ago

Elastic - Kibana Version: 8.4.2 OS - Ubuntu 22 Installation - Self managed

Steps to Reproduce:

Expected result

for details please check: https://discuss.elastic.co/t/elastic-agent-error-dialing-x509-certificate-signed-by-unknown-authority/315675

ddatsh commented 2 years ago

any progress?

m-katunar commented 1 year ago

Hi, facing the exact same issue here with Elastic 8.6.

It seems as if Fleet Agent is ignoring all the certificate flags during the installation. Even tried the --insecure flag to try and bypass certificate validation, but ended up with the same results.

Any updates on how it was fixed?

Thanks

nathanatimu commented 1 year ago

i didn't fix it, is just decided not to use fleet at that moment.

i will try again in the future to see if i can get this working with 8.6

m-katunar commented 1 year ago

During the bootstrap, the Metricbeat and Filebeat are trying to connect to http://localhost:9200, which is the default value for Output once you first go to the Fleet Settings. That setting is of course changed to real output, let's say https://elastic:9200 and it does appear as a generated flag during the installation procedure, but it is ignored at the begging in favor of http://localhost:9200.

After it times out enough times at http://localhost:9200, you get a prompt that the Fleet Server is up, healthy and ready to be used, but no data streams are created as no data is flowing in.

Then when you check the .json logs you get the "certificate signed by unknown authority" that you've mentioned as the Filebeat and Metricbeat are now trying to connect to real output, https://elastic:9200, but are completely ignoring the CA that has been specified during the bootstrap.

I've tried just connecting to https://elastic:9200 using solely Filebeat and it works just fine with no cert error.

I'd say that the whole array of flags that you get following the procedure for bootstrapping Fleet Server during installation are not used properly, at least in our cases.

emilyastranova commented 1 year ago

Having the same issue

cyberdaddy93 commented 1 year ago

Almost a whole year still no fix.....

jlind23 commented 1 year ago

@yeetdomain93 sorry for the delay here, this is not a problem we faced while testing but we will try to reproduce. @michel-laterman will take a look at it and see where it goes.

michel-laterman commented 1 year ago

I've been unable to recreate with v8.8.0.

Here are the step by step instructions I used:

# Download ES + Kibana
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.8.0-darwin-x86_64.tar.gz
tar -xzf elasticsearch-8.8.0-darwin-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-8.8.0-darwin-x86_64.tar.gz
tar -xzf kibana-8.8.0-darwin-x86_64.tar.gz

# Create cert for fleet-server
cd elasticsearch-8.8.0
./bin/elasticsearch-certutil ca --pem --out stack.zip
unzip stack.zip
# Note $IP is my local IP address
./bin/elasticsearch-certutil cert --name fleet-server --ca-cert ca/ca.crt --ca-key ca/ca.key --pem --ip $IP --out cert.zip

# put certs in /tmp for now
unzip stack.zip -d /tmp
unzip cert.zip -d /tmp

# start ES in another terminal with
cd elasticsearch-8.8.0
./bin/elasticsearch

# start Kibana in another terminal with
cd kibana-8.8.0
./bin/kibana

# Open Kibana on localhost:5601 and past in enrollment token from elasticsearch terminal
# login with elastic + the password from the elasticsearch terminal
# go to fleet click add a fleet-server -> advanced, then follow the instructions (use production as security mode instead of quick-start)

# Install fleet-server
# I use a linux VM and copied my certs
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.8.0-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.8.0-linux-x86_64.tar.gz
cd elastic-agent-8.8.0-linux-x86_64
sudo ./elastic-agent install --url=https://$IP:8220 \
  --fleet-server-es=https://$IP:9200 \
  --fleet-server-service-token=$SERVICE_TOKEN \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-es-ca-trusted-fingerprint=$ES_FINGERPRINT \
  --certificate-authorities=/tmp/ca/ca.crt \
  --fleet-server-cert=/tmp/fleet-server/fleet-server.crt \
  --fleet-server-cert-key=/tmp/fleet-server/fleet-server.key \
  --fleet-server-port=8220

# after fleet-server is detected click add agent, create the policy then install an agent somewhere
# I use another VM here:
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.8.0-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.8.0-linux-x86_64.tar.gz
cd elastic-agent-8.8.0-linux-x86_64
# Install the agent - note that I added the --certificate-authorities flag
sudo ./elastic-agent install --url=https://$IP:8220 --enrollment-token=$TOKEN --certificate-authorities=/tmp/ca/ca.crt

Are you generating a CA + cert for Elasticsearch in your attempts? Did you make sure that the ca fingerprint is specified in the elasticsearch output in the fleet settings of Kibana? Or inline the cert contents? Our docs have a bit more information

Tean commented 1 year ago

I try add es-ca file to /usr/local/share/ca-certificates/ then update-ca-certificates, It's may be work, but I'm not sure this is a better way.

lurk-er commented 2 months ago

also having this issue with elastic stack 8.15