successfully connect to Fleet Server but did not receive data

cloud-chang commented 9 months ago

OS

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04 LTS
Release:        22.04
Codename:       jammy

install Elastic

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
sudo apt-get install apt-transport-https
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

sudo apt-get update && sudo apt-get install elasticsearch

elasticsearch.yml

network.host: 0.0.0.0
xpack.security.authc.api_key.enabled: true

systemctl restart elasticsearch

install Kibana

sudo apt-get update && sudo apt-get install kibana

kibana.yml

server.host: "0.0.0.0"

sudo systemctl start kibana

Fleet Server

install server on this host (124.222.18.X)

sudo ./elastic-agent install \
  --fleet-server-es=https://10.0.12.12:9200 \
  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MDMzMjUzOTMyNjE6b05TMFBjYTVTRTZ3UjZpN1FiN2N2UQ \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-es-ca-trusted-fingerprint=00c7a760b4ed7747ca7b64e122b2f6ffd230118ccb6981c26121edd3337af0eb \
  --fleet-server-port=8220

---> Elastic Agent has been successfully installed.

Install the agent on my own personal computer(Mac)

sudo ./elastic-agent install --url=https://124.222.18.X:8220 --enrollment-token=aXBraWxvd0JXbk5WVXlibi1DQXM6R1dRWTlUazZSclNybGQ2S0RWdF9qUQ== --insecure

--> Elastic Agent has been successfully installed.

Elasticsearch did not receive data：

still not work:

michel-laterman commented 9 months ago

Hi, let's keep the issue to the discuss thread for now

cloud-chang commented 8 months ago

elastic-agent-diagnostics-2023-12-27T06-09-10Z-00.zip

@michel-laterman discuss thread can not upload log files

cloud-chang commented 8 months ago

I installed the agent with root privileges 91703435828_ pic

elastic-agent.yml

###################### Agent Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The elastic-agent.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.

######################################
# Fleet configuration
######################################
outputs:
  default:
    type: elasticsearch
    hosts: [FleetServerIP:9200]
    #api_key: "example-key"
    username: "elastic"
    password: "password"

# Here you can configure your list of inputs. You can either configure all the inputs as a list of arrays
# or create an "inputs.d" directory containing your input configurations.
# See https://www.elastic.co/guide/en/fleet/current/elastic-agent-configuration.html for how to structure the "inputs.d" directory.
inputs:
  # Collecting system metrics
  - type: system/metrics
    # Each input must have a unique ID.
    id: unique-system-metrics-input
    # Namespace name must conform to the naming conventions for Elasticsearch indices, cannot contain dashes (-), and cannot exceed 100 bytes
    # For index naming restrictions, see https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params
    data_stream.namespace: default
    use_output: default
    streams:
      - metricsets:
        - cpu
        # Dataset name must conform to the naming conventions for Elasticsearch indices, cannot contain dashes (-), and cannot exceed 100 bytes
        data_stream.dataset: system.cpu
      - metricsets:
        - memory
        data_stream.dataset: system.memory
      - metricsets:
        - network
        data_stream.dataset: system.network
      - metricsets:
        - filesystem
        data_stream.dataset: system.filesystem

#   # Collecting log files
#   - type: filestream
#     # Input ID allowing Elastic Agent to track the state of this input. Must be unique.
#     id: your-input-id
#     streams:
#       # Stream ID for this data stream allowing Filebeat to track the state of the ingested files. Must be unique.
#       # Each filestream data stream creates a separate instance of the Filebeat filestream input.
#       - id: your-filestream-stream-id
#         data_stream:
#           dataset: generic
#         paths:
#           - /var/log/*.log

# agent.monitoring:
#   # enabled turns on monitoring of running processes
#   enabled: true
#   # enables log monitoring
#   logs: true
#   # enables metrics monitoring
#   metrics: true
#   # exposes /debug/pprof/ endpoints
#   # recommended that these endpoints are only enabled if the monitoring endpoint is set to localhost
#   pprof.enabled: false
#   # The name of the output to use for monitoring data.
#   use_output: monitoring
#   # exposes agent metrics using http, by default sockets and named pipes are used
#   http:
#       # enables http endpoint
#       enabled: false
#       # The HTTP endpoint will bind to this hostname, IP address, unix socket or named pipe.
#       # When using IP addresses, it is recommended to only use localhost.
#       host: localhost
#       # Port on which the HTTP endpoint will bind. Default is 0 meaning feature is disabled.
#       port: 6791
#       # Metrics buffer endpoint
#       buffer.enabled: false
#   # Configuration for the diagnostics action handler
#   diagnostics:
#       # Rate limit for the action handler. Does not affect diagnostics collected through the CLI.
#       limit:
#           # Rate limit interval.
#           interval: 1m
#           # Rate limit burst.
#           burst: 1
#       # Configuration for the file-upload client. Client may retry failed requests with an exponential backoff.
#       uploader:
#           # Max retries allowed when uploading a chunk.
#           max_retries: 10
#           # Initial duration of the backoff.
#           init_dur: 1s
#           # Max duration of the backoff.
#           max_dur: 1m

# # Allow fleet to reload its configuration locally on disk.
# # Notes: Only specific process configuration will be reloaded.
# agent.reload:
#   # enabled configure the Elastic Agent to reload or not the local configuration.
#   #
#   # Default is true
#   enabled: true

#   # period define how frequent we should look for changes in the configuration.
#   period: 10s

# management:
#   # Mode of management, the Elastic Agent support two modes of operation:
#   #
#   # local: The Elastic Agent will expect to find the inputs configuration in the local file.
#   #
#   # Default is local.
#   mode: "local"

# fleet:
#   access_api_key: ""
#   kibana:
#     # kibana minimal configuration
#     hosts: ["localhost:5601"]
#     ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

#     # optional values
#     #protocol: "https"
#     #service_token: "example-token"
#     #path: ""
#     #ssl.verification_mode: full
#     #ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]
#     #ssl.cipher_suites: []
#     #ssl.curve_types: []
#   reporting:
#     # Reporting threshold indicates how many events should be kept in-memory before reporting them to fleet.
#     #reporting_threshold: 10000
#     # Frequency used to check the queue of events to be sent out to fleet.
#     #reporting_check_frequency_sec: 30

# agent.download:
#   # source of the artifacts, requires elastic like structure and naming of the binaries
#   # e.g /windows-x86.zip
#   sourceURI: "https://artifacts.elastic.co/downloads/beats/"
#   # path to the directory containing downloaded packages
#   target_directory: "${path.data}/downloads"
#   # timeout for downloading package
#   timeout: 120s
#   # install_path describes the location of installed packages/programs. It is also used
#   # for reading program specifications.
#   install_path: "${path.data}/install"
#   # retry_sleep_init_duration is the duration to sleep for before the first retry attempt. This
#   # duration will increase for subsequent retry attempts in a randomized exponential backoff manner.
#   retry_sleep_init_duration: 30s

# agent.process:
#   # timeout for creating new processes. when process is not successfully created by this timeout
#   # start operation is considered a failure
#   spawn_timeout: 30s
#   # timeout for stopping processes. when process is not stopped by this timeout then the process.
#   # is force killed
#   stop_timeout: 30s

# agent.grpc:
#   # listen address for the GRPC server that spawned processes connect back to.
#   address: localhost
#   # port for the GRPC server that spawned processes connect back to.
#   port: 6789
#   # max_message_size limits the message size in agent internal communication
#   # default is 100MB
#   max_message_size: 104857600

# agent.retry:
#   # Enabled determines whether retry is possible. Default is false.
#   enabled: true
#   # RetriesCount specifies number of retries. Default is 3.
#   # Retry count of 1 means it will be retried one time after one failure.
#   retriesCount: 3
#   # Delay specifies delay in ms between retries. Default is 30s
#   delay: 30s
#   # MaxDelay specifies maximum delay in ms between retries. Default is 300s
#   maxDelay: 5m
#   # Exponential determines whether delay is treated as exponential.
#   # With 30s delay and 3 retries: 30, 60, 120s
#   # Default is false
#   exponential: false

# Logging

# There are four options for the log output: file, stderr, syslog, eventlog
# The file output is the default.

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#agent.logging.level: info

# Enable debug output for selected components. To enable all selectors use ["*"]
# Other available selectors are "beat", "publish", "service"
# Multiple selectors can be chained.
#agent.logging.selectors: [ ]

# Send all logging output to stderr. The default is false.
agent.logging.to_stderr: true

# Send all logging output to syslog. The default is false.
#agent.logging.to_syslog: false

# Send all logging output to Windows Event Logs. The default is false.
#agent.logging.to_eventlog: false

# If enabled, Elastic-Agent periodically logs its internal metrics that have changed
# in the last period. For each metric that changed, the delta from the value at
# the beginning of the period is logged. Also, the total values for
# all non-zero internal metrics are logged on shutdown. This setting is also passed
# to beats running under the agent. The default is true.
#agent.logging.metrics.enabled: true

# The period after which to log the internal metrics. The default is 30s.
#agent.logging.metrics.period: 30s

# Logging to rotating files. Set logging.to_files to false to disable logging to
# files.
#agent.logging.to_files: true
#agent.logging.files:
  # Configure the path where the logs are written. The default is the logs directory
  # under the home path (the binary location).
  #path: /var/log/elastic-agent

  # The name of the files where the logs are written to.
  #name: elastic-agent

  # Configure log file size limit. If limit is reached, log file will be
  # automatically rotated
  #rotateeverybytes: 20971520 # = 20MB

  # Number of rotated log files to keep. Oldest files will be deleted first.
  #keepfiles: 7

  # The permissions mask to apply when rotating log files. The default value is 0600.
  # Must be a valid Unix-style file permissions mask expressed in octal notation.
  #permissions: 0600

  # Enable log file rotation on time intervals in addition to size-based rotation.
  # Intervals must be at least 1s. Values of 1m, 1h, 24h, 7*24h, 30*24h, and 365*24h
  # are boundary-aligned with minutes, hours, days, weeks, months, and years as
  # reported by the local system clock. All other intervals are calculated from the
  # Unix epoch. Defaults to disabled.
  #interval: 0

  # Rotate existing logs on startup rather than appending to the existing
  # file. Defaults to true.
  # rotateonstartup: true

# Set to true to log messages in JSON format.
#agent.logging.json: false

# Providers

# Providers supply the key/values pairs that are used for variable substitution
# and conditionals. Each provider's keys are automatically prefixed with the name
# of the provider.

#providers:

# Agent provides information about the running agent.
#  agent:
#    enabled: true

# Docker provides inventory information from Docker.
#  docker:
#    enabled: true
#    host: "unix:///var/run/docker.sock"
#    cleanup_timeout: 60

# Env providers information about the running environment.
#  env:
#    enabled: true

# Host provides information about the current host.
#  host:
#    enabled: true

# Local provides custom keys to use as variable.
#  local:
#    enabled: true
#    vars:
#      foo: bar

# Local dynamic allows you to define multiple key/values to generate multiple configurations.
#  local_dynamic:
#    enabled: true
#    items:
#      - vars:
#          my_var: key1
#      - vars:
#          my_var: key2
#      - vars:
#          my_var: key3

cloud-chang commented 8 months ago

If Fleet Server and Agent are installed on the same machine without any issues，it won't work if it's on different machines，I have installed it many times, but each time it is not possible.

michel-laterman commented 8 months ago

This slipped my notice, i'll take a look today.

EDIT: It looks like there is a network restriction/firewall/proxy that is interfering with the agent->Elasticsearch communications. The *beat processes on the elastic-agent instance are unable to connect to ES, there are timeout errors:

{"log.level":"error","@timestamp":"2023-12-27T05:48:16.034Z","message":"Failed to connect to backoff(elasticsearch(https://10.0.12.12:9200)): Get \"https://10.0.12.12:9200\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

It looks like the instance where fleet-server is installed does not have the same restriction (it is able to connect to ES).

dharick04 commented 6 months ago

even I'm also facing this error , how to resolve issues, I'm trying to get log in different network.

![Uploading image.png…]()

dharick04 commented 6 months ago

This slipped my notice, i'll take a look today.

EDIT: It looks like there is a network restriction/firewall/proxy that is interfering with the agent->Elasticsearch communications. The *beat processes on the elastic-agent instance are unable to connect to ES, there are timeout errors:
{"log.level":"error","@timestamp":"2023-12-27T05:48:16.034Z","message":"Failed to connect to backoff(elasticsearch(https://10.0.12.12:9200)): Get \"https://10.0.12.12:9200\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
It looks like the instance where fleet-server is installed does not have the same restriction (it is able to connect to ES).

Hi @michel-laterman sir, I also facing same issue , i have change elastic-agent.yml (outputs: default: type: elasticsearch hosts: [publicIP:9200]) , i want to monitor outside of the network.

still I'm unable to get log sir

agent Log: ({"log.level":"warn","@timestamp":"2024-03-16T07:20:53.946Z","message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"log.origin":{"file.line":331,"file.name":"beater/filebeat.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"} )

elastic / fleet-server