vikassharma12345 commented 5 years ago

Describe the bug A clear and concise description of what the bug is.

APM Server goes down for While. APM agent queue/buffers increased.please suggest,how to free that Queue From all APM Agent. I have 15 different APM Agent to only one APM Server. Using elastic-apm-agent-1.8.0.jar, elasticapm.properties files to tomcat bin and added following to setenv.bat:CATALINA_OPTS="$CATALINA_OPTS -javaagent:/home/tomcat/elastic-apm-agent-1.8.0.jar -Delastic.apm.service_name=P_24_CA -Delastic.apm.server_url=http://IP -Delastic.apm.environment=PR -Delastic.apm.capture_body=all"

eyalkoren commented 5 years ago

The agent will keep these events in queue until connection to the APM server is restored. The advantage of that is not losing data during short connectivity failures.

What is your concern? If it is about the amount of memory occupied by the buffer in such cases, you can reduce their size through the max_queue_size configuration.

What would be the alternative/additional feature/behaviour you wished you had?

vikassharma12345 commented 5 years ago

Hi eyalkoren, my concern is to APM server is working. but in elasticsearch Apm-* index is not created. Apm server logs getting 503 and 202 responses. Please suggest what i do now. I have more then 15 Agent configure on that APM server.

command ./apm-server test output

elasticsearch: http://IP parse url... OK connection... parse host... OK dns lookup... OK addresses: IP dial up... OK TLS... WARN secure connection disabled talk to server... OK version: 7.3.0 elasticsearch: http://IP.. parse url... OK connection... parse host... OK dns lookup... OK addresses: IP dial up... OK TLS... WARN secure connection disabled talk to server... OK version: 7.3.0

eyalkoren commented 5 years ago

@vikassharma12345 this is the java agent repo. If you think there is a bug or you want to make a feature/enhancement request regarding the java agent- this is the proper place. If this is the case, please try to make sure this is indeed a bug and not a setup problem, and if it is, make sure you provide clearer description of the problem.

So far, I didn't understand whether this is a bug or a request and even whether it is a problem with the APM server or agent...

If this is neither a bug report nor an enhancement request, but only a request for assistance, please use our forum. There, as well, we will not be able to assist if you don't provide clear description of the problem, including what setup you are using, and at least logs from APM server/agent.

vikassharma12345 commented 5 years ago

ElasticSearch Index

green open .kibana qcymOwbiR-agbnGCO0ytEg 1 1 4 0 66.3kb 36.4kb green open metricbeat-7.3.0-2019.08.18-000001 J-IdzmtMSnKMXPiLTG6cPw 1 1 125852 0 98.6mb 33.1mb green open .monitoring-es-7-mb-2019.08.18 oLvxrPRlTf-IJF1zuWll5w 1 1 72 0 689.5kb 259.5kb green open .monitoring-beats-7-2019.08.18 whVMQcOXT8C0ZqoD84bCJw 1 1 94 0 612.5kb 307.1kb

APM Server logs

2019-08-18T19:04:26.749+0530 INFO instance/beat.go:614 Beat ID: 73f22b2d-91e7-4472-a4b2-268e415fcf25 2019-08-18T19:04:26.750+0530 INFO [index-management] idxmgmt/std.go:178 Set output.elasticsearch.index to 'apm-server-7.3.0' as ILM is enabled. 2019-08-18T19:04:26.750+0530 INFO elasticsearch/client.go:170 Elasticsearch url: http://IP:9200 2019-08-18T19:04:26.750+0530 INFO elasticsearch/client.go:170 Elasticsearch url: http://IP:9201 2019-08-18T19:04:26.754+0530 INFO elasticsearch/client.go:743 Attempting to connect to Elasticsearch version 7.3.0 2019-08-18T19:04:26.798+0530 INFO elasticsearch/client.go:743 Attempting to connect to Elasticsearch version 7.3.0

![Uploading image.png…]()

################### APM Server Configuration #########################

############################# APM Server ######################################

apm-server:

Defines the host and port the server is listening on. use "unix:/path/to.sock" to listen on a unix domain socket.

host: "0.0.0.0:8200"

Maximum permitted size in bytes of a request's header accepted by the server to be processed.

max_header_size: 1048576

Maximum idle time until the server closes a connection.

idle_timeout: 45s

Maximum permitted duration for reading an entire request.

read_timeout: 30s

Maximum permitted duration for writing a response.

write_timeout: 30s

Maximum duration in seconds before releasing resources when shutting down the server.

shutdown_timeout: 5s

Maximum allowed size in bytes of a single event

max_event_size: 307200

--

Maximum number of new connections to accept simultaneously (0 means unlimited)

max_connections: 0

Authorization token to be checked. If a token is set here the agents must

send their token in the following format: Authorization: Bearer .

It is recommended to use an authorization token in combination with SSL enabled,

and save the token in the apm-server keystore.

secret_token:

Enable secure communication between APM agents and the server. By default ssl is disabled.

ssl:

#enabled: false

# Configure a list of root certificate authorities for verifying client certificates
#certificate_authorities: []

# Path to file containing the certificate for server authentication
# Needs to be configured when ssl is enabled
#certificate: ''

# Path to file containing server certificate key
# Needs to be configured when ssl is enabled
#key: ''

# Optional configuration options for ssl communication

# Passphrase for decrypting the Certificate Key.
# It is recommended to use the provided keystore instead of entering the passphrase in plain text.
#key_passphrase: ''

# List of supported/valid protocol versions. By default TLS versions 1.1 up to 1.2 are enabled.
#supported_protocols: [TLSv1.1, TLSv1.2]

# Configure cipher suites to be used for SSL connections
#cipher_suites: []

# Configure curve types for ECDHE based cipher suites
#curve_types: []

# Configure which type of client authentication is supported.
# Options are `none`, `optional`, and `required`. Default is `optional`.
#client_authentication: "optional"

rum:

To enable real user monitoring (RUM) support set this to true.

enabled: false

#event_rate:

  # Defines the maximum amount of events allowed to be sent to the APM Server RUM
  # endpoint per IP per second. Defaults to 300.
  #limit: 300

  # An LRU cache is used to keep a rate limit per IP for the most recently seen IPs.
  # This setting defines the number of unique IPs that can be tracked in the cache.
  # Sites with many concurrent clients should consider increasing this limit. Defaults to 1000.
  #lru_size: 1000

#-- General RUM settings

# Comma separated list of permitted origins for real user monitoring.
# User-agents will send an origin header that will be validated against this list.
# An origin is made of a protocol scheme, host and port, without the url path.
# Allowed origins in this setting can have * to match anything (eg.: http://*.example.com)
# If an item in the list is a single '*', everything will be allowed
#allow_origins : ['*']

# Regexp to be matched against a stacktrace frame's `file_name` and `abs_path` attributes.
# If the regexp matches, the stacktrace frame is considered to be a library frame.
#library_pattern: "node_modules|bower_components|~"

# Regexp to be matched against a stacktrace frame's `file_name`.
# If the regexp matches, the stacktrace frame is not used for calculating error groups.
# The default pattern excludes stacktrace frames that have a filename starting with '/webpack'
#exclude_from_grouping: "^/webpack"

# If a source map has previously been uploaded, source mapping is automatically applied
# to all error and transaction documents sent to the RUM endpoint.
#source_mapping:

  # Source maps are always fetched from Elasticsearch, by default using the output.elasticsearch configuration.
  # A different instance must be configured when using any other output.
  # This setting only affects sourcemap reads - the output determines where sourcemaps are written.
  #elasticsearch:
    # Array of hosts to connect to.
    # Scheme and port can be left out and will be set to the default (http and 9200)
    # In case you specify and additional path, the scheme is required: http://localhost:9200/path
    # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
    # hosts: ["localhost:9200"]

    # Optional protocol and basic auth credentials.
    #protocol: "https"
    #username: "elastic"
    #password: "changeme"

  # The `cache.expiration` determines how long a source map should be cached before fetching it again from Elasticsearch.
  # Note that values configured without a time unit will be interpreted as seconds.
  #cache:
    #expiration: 5m

  # Source maps are stored in a separate index.
  # If the default index pattern for source maps at 'outputs.elasticsearch.indices'
  # is changed, a matching index pattern needs to be specified here.
  #index_pattern: "apm-*-sourcemap*"

If set to true, APM Server augments data received by the agent with the original IP of the backend server,

or the IP and User Agent of the real user (RUM requests). It defaults to true.

capture_personal_data: true

golang expvar support - https://golang.org/pkg/expvar/

expvar:

# Set to true to expose expvar
#enabled: false

# Url to expose expvar
#url: "/debug/vars"

Instrumentation support for the server's HTTP endpoints and event publisher.

instrumentation:

# Set to true to enable instrumentation of the APM Server itself.
#enabled: false
# Environment in which the APM Server is running on (eg: staging, production, etc.)
#environment: ""
# Remote hosts to report instrumentation results to.
#hosts:
#  - http://remote-apm-server:8200
# Remote apm-servers' secret_token
#secret_token:

A pipeline is a definition of processors applied to documents when writing them to Elasticsearch.

Using pipelines involves two steps:

(1) registering a pipeline

(2) applying a pipeline during data ingestion (see `output.elasticsearch.pipelines`)

#

You can manually register pipelines, or use this configuration option to ensure

pipelines are loaded and registered at the configured Elasticsearch instances.

Automatic pipeline registration requires

* `output.elasticsearch` to be enabled and configured.

* having the required Elasticsearch Processor Plugins installed.

APM Server default pipelines require you to have the `Ingest User Agent Plugin` installed.

Find the default pipeline configuration at `ingest/pipeline/definition.json`.

#

register.ingest.pipeline:

# Registers pipeline definitions in Elasticsearch on APM Server startup. Defaults to false.
#enabled: false

# Overwrites existing pipeline definitions in Elasticsearch. Defaults to true.
#overwrite: false

When ilm is set to true the APM Server will take care of ILM setup.

Configurations in output.elasticsearch.index and output.elasticsearch.indices sections will be ignored

ilm can only be enabled for output.elasticsearch

ilm:

#enabled: false

================================ General ======================================

Internal queue configuration for buffering events to be published.

queue:

Queue type by name (default 'mem')

The memory queue will present all available events (up to the outputs

bulk_max_size) to the output, the moment the output is ready to server

another batch of events.

mem:

# Max number of events the queue can buffer.
events: 5120

# Hints the minimum number of events stored in the queue,
# before providing a batch of events to the outputs.
# The default value is set to 2048.
# A value of 0 ensures events are immediately available
# to be sent to the outputs.
flush.min_events: 2048

# Maximum duration after which events are available to the outputs,
# if the number of events stored in the queue is < min_flush_events.
flush.timeout: 1s

Sets the maximum number of CPUs that can be executing simultaneously. The

default is the number of logical CPUs available in the system.

max_procs:

============================== Template =====================================

A template is used to set the mapping in Elasticsearch

By default template loading is enabled and the template is loaded.

These settings can be adjusted to load your own template or overwrite existing ones.

Set to false to disable template loading.

setup.template.enabled: true

Template name. By default the template name is "apm-%{[observer.version]}"

The template name and pattern has to be set in case the elasticsearch index pattern is modified.

setup.template.name: "apm-%{[observer.version]}"

Template pattern. By default the template pattern is "apm-%{[observer.version]}-*" to apply to the default index settings.

The first part is the version of apm-server and then -* is used to match all daily indices.

The template name and pattern has to be set in case the elasticsearch index pattern is modified.

setup.template.pattern: "apm-%{[observer.version]}-*"

Path to fields.yml file to generate the template

setup.template.fields: "${path.config}/fields.yml"

Overwrite existing template

setup.template.overwrite: false

Elasticsearch template settings

setup.template.settings:

A dictionary of settings to place into the settings.index dictionary

of the Elasticsearch template. For more details, please check

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

index: number_of_shards: 4 codec: best_compression number_of_routing_shards: 30 mapping.total_fields.limit: 2000

============================= Elastic Cloud ==================================

These settings simplify using APM Server with the Elastic Cloud (https://cloud.elastic.co/).

The cloud.id setting overwrites the `output.elasticsearch.hosts` option.

You can find the `cloud.id` in the Elastic Cloud web UI.

cloud.id:

The cloud.auth setting overwrites the `output.elasticsearch.username` and

`output.elasticsearch.password` settings. The format is `<user>:<pass>`.

cloud.auth:

================================ Outputs =====================================

Configure what output to use when sending the data collected by apm-server.

-------------------------- Elasticsearch output ------------------------------

output.elasticsearch:

Array of hosts to connect to.

Scheme and port can be left out and will be set to the default (http and 9200)

In case you specify and additional path, the scheme is required: http://localhost:9200/path

IPv6 addresses should always be defined as: https://[2001:db8::1]:9200

hosts: ["10.247.74.21:9200","10.247.74.21:9201"]

Boolean flag to enable or disable the output module.

enabled: true

Set gzip compression level.

compression_level: 0

Optional protocol and basic auth credentials.

protocol: "https"

username: "elastic"

password: "changeme"

Dictionary of HTTP parameters to pass within the url with index operations.

parameters:

#param1: value1
#param2: value2

Number of workers per Elasticsearch host.

worker: 1

By using the configuration below, apm documents are stored to separate indices,

depending on their `processor.event`:

- error

- transaction

- span

- sourcemap

#

The indices are all prefixed with `apm-%{[observer.version]}`.

To allow managing indices based on their age, all indices (except for sourcemaps)

end with the information of the day they got indexed.

e.g. "apm-6.3.0-transaction-2018.03.20"

#

Be aware that you can only specify one Elasticsearch template.

In case you modify the index patterns you must also update those configurations accordingly,

as they need to be aligned:

* `setup.template.name`

* `setup.template.pattern`

index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"

indices:

- index: "apm-%{[observer.version]}-sourcemap"

when.contains:

processor.event: "sourcemap"

#

- index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"

when.contains:

processor.event: "error"

#

- index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"

when.contains:

processor.event: "transaction"

#

- index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"

when.contains:

processor.event: "span"

#

- index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"

when.contains:

processor.event: "metric"

#

- index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"

when.contains:

processor.event: "onboarding"

A pipeline is a definition of processors applied to documents when writing them to Elasticsearch.

APM Server comes with a default pipeline definition, located at `ingest/pipeline/definition.json`.

Pipelines are disabled by default. To make use of them you have to:

(1) ensure pipelines are registered in Elasticsearch, see `apm-server.register.ingest.pipeline`

(2) enable the following:

pipelines:

- pipeline: "apm"

Optional HTTP Path

path: "/elasticsearch"

Custom HTTP headers to add to each request

headers:

X-My-Header: Contents of the header

Proxy server url

proxy_url: http://proxy:3128

The number of times a particular Elasticsearch index operation is attempted. If

the indexing operation doesn't succeed after this many retries, the events are

dropped. The default is 3.

max_retries: 3

The maximum number of events to bulk in a single Elasticsearch bulk API index request.

The default is 50.

bulk_max_size: 5120

The number of seconds to wait before trying to reconnect to Elasticsearch

after a network error. After waiting backoff.init seconds, apm-server

tries to reconnect. If the attempt fails, the backoff timer is increased

exponentially up to backoff.max. After a successful connection, the backoff

timer is reset. The default is 1s.

backoff.init: 1s

The maximum number of seconds to wait before attempting to connect to

Elasticsearch after a network error. The default is 60s.

backoff.max: 60s

Configure http request timeout before failing an request to Elasticsearch.

timeout: 90

Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication.

ssl.enabled: true

Optional SSL configuration options. SSL is off by default, change the `protocol` option if you want to enable `https`.

Configure SSL verification mode. If `none` is configured, all server hosts

and certificates will be accepted. In this mode, SSL based connections are

susceptible to man-in-the-middle attacks. Use only for testing. Default is

`full`.

ssl.verification_mode: full

List of supported/valid TLS versions. By default all TLS versions 1.0 up to

1.2 are enabled.

ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]

List of root certificates for HTTPS server verifications

ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

ssl.key: "/etc/pki/client/cert.key"

Optional passphrase for decrypting the Certificate Key.

It is recommended to use the provided keystore instead of entering the passphrase in plain text.

ssl.key_passphrase: ''

Configure cipher suites to be used for SSL connections

ssl.cipher_suites: []

Configure curve types for ECDHE based cipher suites

ssl.curve_types: []

Configure what types of renegotiation are supported. Valid options are

never, once, and freely. Default is never.

ssl.renegotiation: never

----------------------------- Console output ---------------------------------

output.console:

Boolean flag to enable or disable the output module.

enabled: false

Configure JSON encoding

codec.json:

# Pretty-print JSON event
#pretty: false

# Configure escaping HTML symbols in strings.
#escape_html: false

----------------------------- Logstash output ---------------------------------

output.logstash:

Boolean flag to enable or disable the output module.

enabled: false

The Logstash hosts

hosts: ["localhost:5044"]

Number of workers per Logstash host.

worker: 1

Set gzip compression level.

compression_level: 3

Configure escaping html symbols in strings.

escape_html: true

Optional maximum time to live for a connection to Logstash, after which the

connection will be re-established. A value of `0s` (the default) will

disable this feature.

#

Not yet supported for async connections (i.e. with the "pipelining" option set)

ttl: 30s

Optional load balance the events between the Logstash hosts. Default is false.

loadbalance: false

Number of batches to be sent asynchronously to Logstash while processing

new batches.

pipelining: 2

If enabled only a subset of events in a batch of events is transferred per

group. The number of events to be sent increases up to `bulk_max_size`

if no error is encountered.

slow_start: false

The number of seconds to wait before trying to reconnect to Logstash

after a network error. After waiting backoff.init seconds, apm-server

tries to reconnect. If the attempt fails, the backoff timer is increased

exponentially up to backoff.max. After a successful connection, the backoff

timer is reset. The default is 1s.

backoff.init: 1s

The maximum number of seconds to wait before attempting to connect to

Logstash after a network error. The default is 60s.

backoff.max: 60s

Optional index name. The default index name is set to apm

in all lowercase.

index: 'apm'

SOCKS5 proxy server URL

proxy_url: socks5://user:password@socks5-server:2233

Resolve names locally when using a proxy server. Defaults to false.

proxy_use_local_resolver: false

Enable SSL support. SSL is automatically enabled if any SSL setting is set.

ssl.enabled: false

Optional SSL configuration options. SSL is off by default.

Configure SSL verification mode. If `none` is configured, all server hosts

and certificates will be accepted. In this mode, SSL based connections are

susceptible to man-in-the-middle attacks. Use only for testing. Default is

`full`.

ssl.verification_mode: full

List of supported/valid TLS versions. By default all TLS versions 1.0 up to

1.2 are enabled.

ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]

List of root certificates for HTTPS server verifications

ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

ssl.key: "/etc/pki/client/cert.key"

Optional passphrase for decrypting the Certificate Key.

It is recommended to use the provided keystore instead of entering the passphrase in plain text.

ssl.key_passphrase: ''

Configure cipher suites to be used for SSL connections

ssl.cipher_suites: []

Configure curve types for ECDHE based cipher suites

ssl.curve_types: []

Configure what types of renegotiation are supported. Valid options are

never, once, and freely. Default is never.

ssl.renegotiation: never

------------------------------- Kafka output ----------------------------------

output.kafka:

Boolean flag to enable or disable the output module.

enabled: false

The list of Kafka broker addresses from where to fetch the cluster metadata.

The cluster metadata contain the actual Kafka brokers events are published

to.

hosts: ["localhost:9092"]

The Kafka topic used for produced events. The setting can be a format string

using any event field. To set the topic from document type use `%{[type]}`.

topic: beats

The Kafka event key setting. Use format string to create unique event key.

By default no event key will be generated.

key: ''

The Kafka event partitioning strategy. Default hashing strategy is `hash`

using the `output.kafka.key` setting or randomly distributes events if

`output.kafka.key` is not configured.

partition.hash:

# If enabled, events will only be published to partitions with reachable
# leaders. Default is false.
#reachable_only: false

# Configure alternative event field names used to compute the hash value.
# If empty `output.kafka.key` setting will be used.
# Default value is empty list.
#hash: []

Authentication details. Password is required if username is set.

username: ''

password: ''

Kafka version libbeat is assumed to run against. Defaults to the "1.0.0".

version: '1.0.0'

Configure JSON encoding

codec.json:

# Pretty print json event
#pretty: false

# Configure escaping html symbols in strings.
#escape_html: true

Metadata update configuration. Metadata do contain leader information

deciding which broker to use when publishing.

metadata:

# Max metadata request retry attempts when cluster is in middle of leader
# election. Defaults to 3 retries.
#retry.max: 3

# Waiting time between retries during leader elections. Default is 250ms.
#retry.backoff: 250ms

# Refresh metadata interval. Defaults to every 10 minutes.
#refresh_frequency: 10m

The number of concurrent load-balanced Kafka output workers.

worker: 1

The number of times to retry publishing an event after a publishing failure.

After the specified number of retries, the events are typically dropped.

Set max_retries to a value less than 0 to retry

until all events are published. The default is 3.

max_retries: 3

The maximum number of events to bulk in a single Kafka request. The default

is 2048.

bulk_max_size: 2048

The number of seconds to wait for responses from the Kafka brokers before

timing out. The default is 30s.

timeout: 30s

The maximum duration a broker will wait for number of required ACKs. The

default is 10s.

broker_timeout: 10s

The number of messages buffered for each Kafka broker. The default is 256.

channel_buffer_size: 256

The keep-alive period for an active network connection. If 0s, keep-alives

are disabled. The default is 0 seconds.

keep_alive: 0

Sets the output compression codec. Must be one of none, snappy and gzip. The

default is gzip.

compression: gzip

Set the compression level. Currently only gzip provides a compression level

between 0 and 9. The default value is chosen by the compression algorithm.

compression_level: 4

The maximum permitted size of JSON-encoded messages. Bigger messages will be

dropped. The default value is 1000000 (bytes). This value should be equal to

or less than the broker's message.max.bytes.

max_message_bytes: 1000000

The ACK reliability level required from broker. 0=no response, 1=wait for

local commit, -1=wait for all replicas to commit. The default is 1. Note:

If set to 0, no ACKs are returned by Kafka. Messages might be lost silently

on error.

required_acks: 1

The configurable ClientID used for logging, debugging, and auditing

purposes. The default is "beats".

client_id: beats

Enable SSL support. SSL is automatically enabled if any SSL setting is set.

ssl.enabled: false

Optional SSL configuration options. SSL is off by default.

Configure SSL verification mode. If `none` is configured, all server hosts

and certificates will be accepted. In this mode, SSL based connections are

susceptible to man-in-the-middle attacks. Use only for testing. Default is

`full`.

ssl.verification_mode: full

List of supported/valid TLS versions. By default all TLS versions 1.0 up to

1.2 are enabled.

ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]

List of root certificates for HTTPS server verifications

ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

ssl.key: "/etc/pki/client/cert.key"

Optional passphrase for decrypting the Certificate Key.

It is recommended to use the provided keystore instead of entering the passphrase in plain text.

ssl.key_passphrase: ''

Configure cipher suites to be used for SSL connections

ssl.cipher_suites: []

Configure curve types for ECDHE based cipher suites

ssl.curve_types: []

Configure what types of renegotiation are supported. Valid options are

never, once, and freely. Default is never.

ssl.renegotiation: never

================================= Paths ======================================

The home path for the apm-server installation. This is the default base path

for all other path settings and for miscellaneous files that come with the

distribution.

If not set by a CLI flag or in the configuration file, the default for the

home path is the location of the binary.

path.home:

The configuration path for the apm-server installation. This is the default

base path for configuration files, including the main YAML configuration file

and the Elasticsearch template file. If not set by a CLI flag or in the

configuration file, the default for the configuration path is the home path.

path.config: ${path.home}

The data path for the apm-server installation. This is the default base path

for all the files in which apm-server needs to store its data. If not set by a

CLI flag or in the configuration file, the default for the data path is a data

subdirectory inside the home path.

path.data: ${path.home}/data

The logs path for an apm-server installation. If not set by a CLI flag or in the

configuration file, the default is a logs subdirectory inside the home path.

path.logs: ${path.home}/logs

================================ Logging ======================================

#

There are three options for the log output: syslog, file, stderr.

Under Windows systems, the log files are per default sent to the file output,

under all other system per default to syslog.

Sets log level. The default log level is info.

Available log levels are: error, warning, info, debug

logging.level: info

Enable debug output for selected components. To enable all selectors use ["*"]

Other available selectors are "beat", "publish", "service"

Multiple selectors can be chained.

logging.selectors: [ ]

Send all logging output to syslog. The default is false.

logging.to_syslog: true

If enabled, apm-server periodically logs its internal metrics that have changed

in the last period. For each metric that changed, the delta from the value at

the beginning of the period is logged. Also, the total values for

all non-zero internal metrics are logged on shutdown. The default is false.

logging.metrics.enabled: false

The period after which to log the internal metrics. The default is 30s.

logging.metrics.period: 30s

Logging to rotating files. Set logging.to_files to false to disable logging to

files.

logging.to_files: true

logging.files:

Configure the path where the logs are written. The default is the logs directory

under the home path (the binary location).

path: /var/log/apm-server

The name of the files where the logs are written to.

name: apm-server

Configure log file size limit. If limit is reached, log file will be

automatically rotated

rotateeverybytes: 10485760 # = 10MB

Number of rotated log files to keep. Oldest files will be deleted first.

keepfiles: 7

The permissions mask to apply when rotating log files. The default value is 0600.

Must be a valid Unix-style file permissions mask expressed in octal notation.

permissions: 0600

Enable log file rotation on time intervals in addition to size-based rotation.

Intervals must be at least 1s. Values of 1m, 1h, 24h, 724h, 3024h, and 365*24h

are boundary-aligned with minutes, hours, days, weeks, months, and years as

reported by the local system clock. All other intervals are calculated from the

Unix epoch. Defaults to disabled.

interval: 0

Set to true to log messages in json format.

logging.json: false

================================ HTTP Endpoint ======================================

#

apm-server can expose internal metrics through a HTTP endpoint. For security

reasons the endpoint is disabled by default. This feature is currently experimental.

Stats can be access through http://localhost:5066/stats . For pretty JSON output

append ?pretty to the URL.

Defines if the HTTP endpoint is enabled.

http.enabled: true

The HTTP endpoint will bind to this hostname or IP address. It is recommended to use only localhost.

http.host: 0.0.0.0

Port on which the HTTP endpoint will bind. Default is 5066.

http.port: 5066

============================== X-pack Monitoring ===============================

APM server can export internal metrics to a central Elasticsearch monitoring

cluster. This requires x-pack monitoring to be enabled in Elasticsearch. The

reporting is disabled by default.

Set to true to enable the monitoring reporter.

monitoring.enabled: true

Most settings from the Elasticsearch output are accepted here as well.

Note that these settings should be configured to point to your Elasticsearch monitoring cluster.

Any setting that is not set is automatically inherited from the Elasticsearch

output configuration. This means that if you have the Elasticsearch output configured,

you can simply uncomment the following line.

monitoring.elasticsearch:

Optional protocol and basic auth credentials.

protocol: "https"

username: "apm_system"

password: ""

Array of hosts to connect to.

Scheme and port can be left out and will be set to the default (http and 9200)

In case you specify and additional path, the scheme is required: http://localhost:9200/path

IPv6 addresses should always be defined as: https://[2001:db8::1]:9200

hosts: ["10.247.74.21:9200","10.247.74.21:9201"]

Set gzip compression level.

compression_level: 0

Dictionary of HTTP parameters to pass within the URL with index operations.

parameters:

#param1: value1
#param2: value2

Custom HTTP headers to add to each request

headers:

X-My-Header: Contents of the header

Proxy server url

proxy_url: http://proxy:3128

The number of times a particular Elasticsearch index operation is attempted. If

the indexing operation doesn't succeed after this many retries, the events are

dropped. The default is 3.

max_retries: 3

The maximum number of events to bulk in a single Elasticsearch bulk API index request.

The default is 50.

bulk_max_size: 50

The number of seconds to wait before trying to reconnect to Elasticsearch

after a network error. After waiting backoff.init seconds, apm-server

tries to reconnect. If the attempt fails, the backoff timer is increased

exponentially up to backoff.max. After a successful connection, the backoff

timer is reset. The default is 1s.

backoff.init: 1s

The maximum number of seconds to wait before attempting to connect to

Elasticsearch after a network error. The default is 60s.

backoff.max: 60s

Configure HTTP request timeout before failing an request to Elasticsearch.

timeout: 90

Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication.

ssl.enabled: true

Optional SSL configuration options. SSL is off by default, change the `protocol` option if you want to enable `https`.

Configure SSL verification mode. If `none` is configured, all server hosts

and certificates will be accepted. In this mode, SSL based connections are

susceptible to man-in-the-middle attacks. Use only for testing. Default is

`full`.

ssl.verification_mode: full

List of supported/valid TLS versions. By default all TLS versions 1.0 up to

1.2 are enabled.

ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]

List of root certificates for HTTPS server verifications

ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

ssl.key: "/etc/pki/client/cert.key"

Optional passphrase for decrypting the Certificate Key.

It is recommended to use the provided keystore instead of entering the passphrase in plain text.

ssl.key_passphrase: ''

Configure cipher suites to be used for SSL connections

ssl.cipher_suites: []

Configure curve types for ECDHE based cipher suites

ssl.curve_types: []

Configure what types of renegotiation are supported. Valid options are

never, once, and freely. Default is never.

ssl.renegotiation: never

metrics.period: 10s

state.period: 1m

eyalkoren commented 5 years ago

Sorry, I am not able to assist. Please describe a problem and add only errors from APM server and agent logs in the forum. Closing this issue.

vikassharma12345 commented 5 years ago

Hi eyalkoren,

Concern

Apm server is creating index in ElasticSearch.
Apm server logs getting 503 and 202 responses with error Queue is Full from APM agent.
Without ElasticSearch index not get APM in Kibana.

vikassharma12345 commented 5 years ago

Apm server is creating index in ElasticSearch.
Apm server logs getting 503 and 202 responses with error Queue is Full from APM agent.
Without ElasticSearch index not get APM in Kibana.

Thanks & Regards,

Vikas Sharma | Software Engineer | Managed Services

https://kmp.digispice.com/images/dsmail.png

https://www.digispice.com www.digispice.com

https://www.digispice.com Spice Mobility Limited

D-91, Industrial Area, Phase 7, Mohali-160055, India

Mobile: +91 96466 09361 Tel: +91 Email: Vikas.Sharma@digispice.com

From: eyalkoren [mailto:notifications@github.com] Sent: Sunday, August 18, 2019 8:08 PM To: elastic/apm-agent-java Cc: vikassharma12345; Mention Subject: Re: [elastic/apm-agent-java] APM agent queue/buffers increased (#798)

Sorry, I am not able to assist. Please describe a problem and add only errors from APM server and agent logs in the forum. Closing this issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/elastic/apm-agent-java/issues/798?email_source=notifications&email_token=AM5XHO73AMQYLI3HV2PXY6DQFFNFHA5CNFSM4IMSTUK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4RBLGY#issuecomment-522327451 , or mute the thread https://github.com/notifications/unsubscribe-auth/AM5XHO4IMENBJI23BHKUVO3QFFNFHANCNFSM4IMSTUKQ . https://github.com/notifications/beacon/AM5XHO4T6FIUHAKJK5IVHADQFFNFHA5CNFSM4IMSTUK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4RBLGY.gif

elastic / apm-agent-java

APM agent queue/buffers increased #798

Defines the host and port the server is listening on. use "unix:/path/to.sock" to listen on a unix domain socket.

Maximum permitted size in bytes of a request's header accepted by the server to be processed.

max_header_size: 1048576

Maximum idle time until the server closes a connection.

idle_timeout: 45s

Maximum permitted duration for reading an entire request.

read_timeout: 30s

Maximum permitted duration for writing a response.

write_timeout: 30s

Maximum duration in seconds before releasing resources when shutting down the server.

shutdown_timeout: 5s

Maximum allowed size in bytes of a single event

max_event_size: 307200

--

Maximum number of new connections to accept simultaneously (0 means unlimited)

max_connections: 0

Authorization token to be checked. If a token is set here the agents must

send their token in the following format: Authorization: Bearer .

It is recommended to use an authorization token in combination with SSL enabled,

and save the token in the apm-server keystore.

secret_token:

Enable secure communication between APM agents and the server. By default ssl is disabled.

ssl:

To enable real user monitoring (RUM) support set this to true.

If set to true, APM Server augments data received by the agent with the original IP of the backend server,

or the IP and User Agent of the real user (RUM requests). It defaults to true.

capture_personal_data: true

golang expvar support - https://golang.org/pkg/expvar/

expvar:

Instrumentation support for the server's HTTP endpoints and event publisher.

instrumentation:

A pipeline is a definition of processors applied to documents when writing them to Elasticsearch.

Using pipelines involves two steps:

(1) registering a pipeline

(2) applying a pipeline during data ingestion (see output.elasticsearch.pipelines)

You can manually register pipelines, or use this configuration option to ensure

pipelines are loaded and registered at the configured Elasticsearch instances.

Automatic pipeline registration requires

* output.elasticsearch to be enabled and configured.

* having the required Elasticsearch Processor Plugins installed.

APM Server default pipelines require you to have the Ingest User Agent Plugin installed.

Find the default pipeline configuration at ingest/pipeline/definition.json.

register.ingest.pipeline:

When ilm is set to true the APM Server will take care of ILM setup.

Configurations in output.elasticsearch.index and output.elasticsearch.indices sections will be ignored

ilm can only be enabled for output.elasticsearch

ilm:

================================ General ======================================

Internal queue configuration for buffering events to be published.

queue:

Queue type by name (default 'mem')

The memory queue will present all available events (up to the outputs

bulk_max_size) to the output, the moment the output is ready to server

another batch of events.

mem:

Sets the maximum number of CPUs that can be executing simultaneously. The

default is the number of logical CPUs available in the system.

max_procs:

============================== Template =====================================

A template is used to set the mapping in Elasticsearch

By default template loading is enabled and the template is loaded.

These settings can be adjusted to load your own template or overwrite existing ones.

Set to false to disable template loading.

Template name. By default the template name is "apm-%{[observer.version]}"

The template name and pattern has to be set in case the elasticsearch index pattern is modified.

setup.template.name: "apm-%{[observer.version]}"

Template pattern. By default the template pattern is "apm-%{[observer.version]}-*" to apply to the default index settings.

The first part is the version of apm-server and then -* is used to match all daily indices.

The template name and pattern has to be set in case the elasticsearch index pattern is modified.

setup.template.pattern: "apm-%{[observer.version]}-*"

Path to fields.yml file to generate the template

setup.template.fields: "${path.config}/fields.yml"

Overwrite existing template

Elasticsearch template settings

A dictionary of settings to place into the settings.index dictionary

of the Elasticsearch template. For more details, please check

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

============================= Elastic Cloud ==================================

(2) applying a pipeline during data ingestion (see `output.elasticsearch.pipelines`)

* `output.elasticsearch` to be enabled and configured.

APM Server default pipelines require you to have the `Ingest User Agent Plugin` installed.

Find the default pipeline configuration at `ingest/pipeline/definition.json`.

The cloud.id setting overwrites the `output.elasticsearch.hosts` option.

You can find the `cloud.id` in the Elastic Cloud web UI.

The cloud.auth setting overwrites the `output.elasticsearch.username` and

`output.elasticsearch.password` settings. The format is `<user>:<pass>`.

depending on their `processor.event`:

The indices are all prefixed with `apm-%{[observer.version]}`.

* `setup.template.name`

* `setup.template.pattern`

APM Server comes with a default pipeline definition, located at `ingest/pipeline/definition.json`.

(1) ensure pipelines are registered in Elasticsearch, see `apm-server.register.ingest.pipeline`