search

logdna / logdna-agent-v2

The blazingly fast, resource efficient log collection client
https://logdna.com
MIT License
65 stars 46 forks source link
container-logging daemonset k8s kubernetes kubernetes-monitoring linux log-management logger logging pod rust

readme

LogDNA Agent

agent version made-with-rust

Mezmo, formerly LogDNA, enables enterprises to ingest all of their log data to a single platform, normalize it, and route it to the appropriate teams so that they can take meaningful action in real time. Join us at the Mezmo community forum.

The LogDNA agent is a resource-efficient log collection client that forwards logs to LogDNA. This version of the agent is written in Rust to ensure maximum performance, and when coupled with LogDNA's web application, provides a powerful log management tool for distributed systems, including Kubernetes clusters. Supported platforms include Linux, Windows and Mac.

:information_source: Information
    As part of the new company name change, we will be changing the name of our agent to mezmo-agent. This will happen over a series of releases. In 3.6 we will be introducing environment variables prefixed with MZ_ in addition to the existing prefix of LOGDNA_. In 3.7 we will release both the agent binaries and yaml files with mezmo in the names. Here again we will do this in tandem with the logdna name. In our 4.0 version, we will fully deprecate all previous references to LOGDNA_ and logdna.

Table of Contents

Managing Deployments

The agent is supported for Kubernetes 1.9+ and Red Hat OpenShift 4.5+ environments.

Installing on Kubernetes

You can use the manifest YAML files in this repository or use Helm package manager to deploy the agent in your Kubernetes cluster.

Using manifest files

NOTE: The Kubernetes manifest YAML files in this repository (and referenced in this documentation) describe the version of the LogDNA Agent in the current commit tree and no other version of the LogDNA agent. If you apply the Kubernetes manifest found in the current tree then your cluster will be running the version described by the current commit which may not be ready for general use.

Follow the full instructions in the documentation to deploy on Kubernetes using resource files.

Using Helm

Visit the documentation to install the agent on Kubernetes using Helm.

Installing on OpenShift

Follow the instructions for deploying the agent on Red Hat OpenShift.

Installing on Linux

Refer to the documentation for deploying the agent on Linux.

Installing on Windows

Refer to the documentation for installing the agent on Windows.

Running as Non-Root

By default the agent will run as root. Below are environment-specific instructions for running the agent as a non-root user.

If you configure the LogDNA Agent to run as non-root, review the documentation about enabling "statefulness" for the agent.

Additional Installation Options

More information about managing your deployments is documented for Kubernetes or OpenShift. This includes topics such as

Installing on Windows

Refer to the documentation for installing the agent on Windows.

Building the LogDNA Agent

Obtain the source code to build the agent from our GitHub repository. You can either download the source files as a .zip file, or clone the repository.

git clone https://github.com/logdna/logdna-agent-v2.git
cd logdna-agent-v2

Next, select how you want to build the agent: as a docker image or as a Linux binary.

Building Docker image

To build a Docker image of the agent, ensure the Docker command is installed properly, verify that the Docker engine is running, and then run the following command:

make build-image

The resulting image can be found by listing the images:

foo@bar:~$ docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
logdna-agent-v2         dcd54a0             e471b3d8a409        22 seconds ago      135MB

You can also obtain the image and review our tagging scheme on DockerHub.

Building Agent Binary on Linux

The agent requires rustc cargo-make to build. If rustc and cargo are installed, simply run the following command to build the agent:

cargo build --release

The compiled binary will be built to ./target/release/logdna-agent.

Helper scripts

Some make helper scripts are located under ./scripts folder:

* mk                    make "build" target - build Agent from rust sources
* mk.lint               make "lint" target - run linting
* mk.test               make "test" target - run unit tests
* mk.integration_test   make "integration_test" target - run integration tests
* mk.image              make "image" target - create Agent container image and publish it in local docker

Note:

Configuration

Options

The agent accepts configuration from three different sources: environment variables, command line arguments and/or a configuration YAML file. The default configuration yaml file is located at /etc/logdna/config.yaml. Unless noted, when an option says List of, this would mean a comma separated list for Environment variable and a YAML array list for YAML.

Some variables do not have a corresponding Yaml path because they are only applicable to k8s deployments and therefor would use environment variables.

For backward compatibility agent v1 configuration file format is still supported. For example - upgrade to Agent v2 on Windows can re-use v1 config file from previous installation as-is. The format of these v1 configuration parameter names are the same as the environment variables without LOGDNA_ and in lower case.

Environment Variable Name Yaml Path Name Description Default
LOGDNA_INGESTION_KEY
Deprecated: LOGDNA_AGENT_KEY		 http.ingestion_key Required: The ingestion key associated with your LogDNA account
LOGDNA_CONFIG_FILE
Deprecated: DEFAULT_CONF_FILE		 Path to the configuration yaml /etc/logdna/config.yaml
LOGDNA_HOST
Deprecated: LDLOGHOST		 http.host The host to forward logs to logs.logdna.com
LOGDNA_ENDPOINT
Deprecated: LDLOGPATH		 http.endpoint The endpoint to forward logs to /logs/agent
LOGDNA_USE_SSL
Deprecated: LDLOGSSL		 http.use_ssl Whether to use a SSL for sending logs true
LOGDNA_USE_COMPRESSION
Deprecated: COMPRESS		 http.use_compression Whether to compress logs before sending true
LOGDNA_GZIP_LEVEL
Deprecated: GZIP_COMPRESS_LEVEL		 http.gzip_level If compression is enabled, this is the gzip compression level to use 2
LOGDNA_HOSTNAME http.params.hostname The hostname metadata to attach to lines forwarded from this agent
LOGDNA_IP http.params.ip The IP metadata to attach to lines forwarded from this agent
LOGDNA_TAGS http.params.tags Comma separated list of tags metadata to attach to lines forwarded from this agent
LOGDNA_MAC http.params.mac The MAC metadata to attach to lines forwarded from this agent
LOGDNA_LOG_DIRS
Deprecated: LOG_DIRS		 log.dirs[] List of folders to recursively monitor for log events /var/log/
LOGDNA_EXCLUSION_RULES
Deprecated: LOGDNA_EXCLUDE		 log.exclude.glob[] List of glob patterns to exclude files from monitoring 1 /var/log/wtmp,/var/log/btmp,/var/log/utmp,
/var/log/wtmpx,/var/log/btmpx,/var/log/utmpx,
/var/log/asl/**,/var/log/sa/**,/var/log/sar*,
/var/log/tallylog,/var/log/fluentd-buffers/**/*,
/var/log/pods/**/*
LOGDNA_EXCLUSION_REGEX_RULES
Deprecated: LOGDNA_EXCLUDE_REGEX		 log.exclude.regex[] List of regex patterns to exclude files from monitoring
LOGDNA_INCLUSION_RULES
Deprecated: LOGDNA_INCLUDE		 log.include.glob[] List of glob patterns to includes files for monitoring 1 *.log
LOGDNA_INCLUSION_REGEX_RULES
Deprecated: LOGDNA_INCLUDE_REGEX		 log.include.regex[] List of regex patterns to include files from monitoring
LOGDNA_LINE_EXCLUSION_REGEX log.line_exclusion_regex[] List of regex patterns to exclude log lines. When set, the Agent will NOT send log lines that match any of these patterns.
LOGDNA_LINE_INCLUSION_REGEX log.line_inclusion_regex[] List of regex patterns to include log lines. When set, the Agent will send ONLY log lines that match any of these patterns.
LOGDNA_REDACT_REGEX log.line_redact_regex List of regex patterns used to mask matching sensitive information (such as PII) before sending it in the log line.
LOGDNA_JOURNALD_PATHS journald.paths[] List of paths (directories or files) of journald paths to monitor
MZ_SYSTEMD_JOURNAL_TAILER True/False toggles journald on the agent 'true'
LOGDNA_LOOKBACK log.lookback The lookback strategy on startup none
LOGDNA_K8S_STARTUP_LEASE Determines whether or not to use K8 leases on startup never
LOGDNA_USE_K8S_LOG_ENRICHMENT Determines whether the agent should query the K8s API to enrich log lines from other pods. always
LOGDNA_LOG_K8S_EVENTS Determines whether the agent should log Kubernetes resource events. This setting only affects tracking and logging Kubernetes resource changes via watches. When disabled, the agent may still query k8s metadata to enrich log lines from other pods depending on the value of LOGDNA_USE_K8S_LOG_ENRICHMENT setting value. never
LOGDNA_LOG_METRIC_SERVER_STATS Determines whether or not metrics usage statistics is enabled. never
LOGDNA_DB_PATH log.db_path The directory in which the agent will store its state database. Note that the agent must have write access to the directory and be a persistent volume. /var/lib/logdna
LOGDNA_METRICS_PORT log.metrics_port The port number to expose a Prometheus endpoint target with the agent internal metrics.
LOGDNA_INGEST_TIMEOUT http.timeout The timeout of the API calls to the ingest API in milliseconds 10000
LOGDNA_INGEST_BUFFER_SIZE http.body_size The size, in bytes, of the ingest data buffer used to batch log data with. 2097152
LOGDNA_RETRY_DIR http.retry_dir The directory used by the agent to store data temporarily while retrying calls to the ingestion API. /tmp/logdna
LOGDNA_RETRY_DISK_LIMIT http.retry_disk_limit The maximum amount of disk space the agent will use to store retry data. The value can be the total number of bytes or a human representation of space using suffixes "KB", "MB", "GB" or "TB", e.g. 10 MB If left unset, the agent will not limit disk usage. If set to 0, no retry data will be stored on disk.
LOGDNA_META_APP Overrides/omits APP field in log line metadata. Examples
LOGDNA_META_HOST Overrides/omits HOST field in log line metadata.
LOGDNA_META_ENV Overrides/omits ENV field in log line metadata.
LOGDNA_META_FILE Overrides/omits FILE field in log line metadata.
LOGDNA_META_K8S_FILE Overrides/omits FILE field in k8s log line metadata. Follows LOGDNA_META_FILE.
LOGDNA_META_JSON Overrides/omits META filed in log line metadata.
LOGDNA_META_ANNOTATIONS Overrides specific kay-value-pairs inside ANNOTATIONS field in log line metadata.
LOGDNA_META_LABELS Overrides specific kay-value-pairs inside LABELS field in log line metadata.
MZ_METADATA_RETRY_DELAY The number of seconds to wait for Pod label availability before sending log lines. 0
MZ_FLUSH_DURATION http.flush_duration The number of milliseconds to wait before flushing captured logs to the ingestion API. 5000

All regular expressions use Perl-style syntax with case sensitivity by default. If you don't want to differentiate between capital and lower-case letters, use non-capturing groups with a flag: (?flags:exp), for example:

(?i:my_case_insensitive_regex)
  1. We support this flavor of globber syntax.

As listed above, by default the agent will only ingest files with a .log extention. Globing patterns are checked against full file path. Some other common globing patterns to include in the LOGDNA_INCLUSION_RULES could be:

  1. Pattern to ingest files with both a .log extention AND extention-less files:
    • *.log,!(*.*)
  2. Applied to specific log dir:
    • /var/log/containers/mypod*.log
    • */containers/mypod*.log
  3. Applied to all log dirs:
    • */app?.log

Configuring the Environment

To configure the DaemonSet, modify the envs section of the DaemonSet [spec.template.spec.containers.0.env]. For example, to change the hostname add the following environment variable to the env list:

env:
  - name: LOGDNA_HOSTNAME
    value: my-hostname

Check out Kubernetes documentation for more information about injecting environment variables into applications!

Configuring Lookback

The lookback strategy determines how the agent handles existing files on agent startup. This strategy is determined by the LOGDNA_LOOKBACK variable.

By default, the agent provides a "stateful", or persistent, collection of files that can be referenced whenever the agent is restarted, in order to return (or look back) to see any files that were ingested during the time that the agent was not running. The state directory location is defined using the LOGDNA_DB_PATH environment variable in the YAML file (the default path is /var/lib/logdna).

The valid values for this option are:

Notes:

Configuring Lease Startup

The lease startup configuration uses Kubernetes Leases to limit the number of agents that can start at one time on a cluster. When enabled, the agent will "claim" a lease before starting. Once started, the agent will then release the lease. If no leases are available, the agent will wait for one to become available. This feature would only be needed if running the agent on a cluster large enough that you'd risk crashing etcd if all the the agents tried to connect at once.

The valid values for this option are:

Note: For more information on how to configure leases on a your Kubernetes cluster, see the "Installation Steps" in the KUBERNETES.md file in docs directory of this repository.

Configuring Journald

You can monitor journald log files or directories that the agent pods have access to by adding them to the LOGDNA_JOURNALD_PATHS and setting MZ_SYSTEMD_JOURNAL_TAILER to true. Alternatively you leave MZ_SYSTEMD_JOURNAL_TAILER blank as it defaults to true. Common values include for LOGDNA_JOURNALD_PATHS - /var/log/journal and /run/systemd/journal. To specify both, use a comma separated list: /var/log/journal,/run/systemd/journal.

Take a look at enabling journald monitoring for Kubernetes or OpenShift.

Configuring regex for redaction and exclusion or inclusion

You can define rules, using regex (regular expressions), to control what log data is collected by the agent and forwarded to LogDNA.

For example, you can identify certain types of logs that are noisy and not needed at all, and then write a regex pattern to match those lines and preclude them from collection. Conversely, you can use regex expressions to match the log lines that you do want to include, and collect only those log lines.

Additionally, you can use the environment variable LOGDNA_REDACT_REGEX to remove certain parts of a log line. Any redacted data is replaced with [REDACTED]. Redacting information is recommended when logs might contain PII (Personally Identifiable Information).

In general, using Agent-level environment variables to define what log data does not ever leave the data center or environment allows a higher level of control over sensitive data, and can also reduce any potential data egress fees.

You can use regex pattern matching via environment variables to:

To access our library of common regex patterns refer to our regex library documentation.

Notes:

Resource Limits

The agent is deployed as a Kubernetes DaemonSet, creating one pod per node selected. The agent collects logs of all the pods in the node. The resource requirements of the agent are in direct relation to the amount of pods per node, and the amount of logs producer per pod.

The agent requires at least 128Mib of memory and twenty millicpu (20m).

Different features can also increase resource utilization. When line exclusion/inclusion or redaction rules are specified, you can expect to additional CPU consumption per line and per regex rule defined. When Kubernetes event logging is enabled (disabled by default), additional CPU usage will occur on the oldest agent pod.

We do not recommend placing traffic shaping or CPU limits on the agent to ensure data can be sent to our log ingestion service.

Exposing Agent Metrics

The LogDNA agent records internal metrics that can be relevant for monitoring and alerting, such as number log files currently tracked or number of bytes parsed, along with process status information. Check out the documentation for internal metrics for more information.