Getting Started • Getting Involved • Migrating from Smart Agent
Architecture • Components • Monitoring • Security • Sizing • Troubleshooting
Splunk OpenTelemetry Collector is a distribution of the OpenTelemetry Collector. It provides a unified way to receive, process, and export metric, trace, and log data for Splunk Observability Cloud:
sapm
exporter.
The otlphttp
exporter
can be used with a custom
configuration.
More information available
here.signalfx
exporter.
More information available
here.While it is recommended to use Splunk
Forwarders
to send data to Splunk
Cloud or Splunk
Enterprise,
Splunk OpenTelemetry Collector can be configured to send data to them via the
splunk_hec
exporter.
The following resources are available:
All you need to get started is:
This distribution is supported on and packaged for a variety of platforms including:
You can consult additional use cases in the examples directory.
A variety of default configuration files are provided:
full_config_linux.yaml
for a commented configuration with links to full
documentation. The logs_config_linux.yaml
is a good starting point for using
the collector for collecting application logs on Linux environments.
agent_config.yaml
is the recommended starting configuration for most environments.*.conf
files as well as the conf.d
directory. Common sources including filelog,
journald, and Windows event viewer are included.In addition, the following components can be configured:
The Splunk OpenTelemetry Collector provides a sensitive value-redacting, local config server listening at
http://localhost:55554/debug/configz/effective
that is helpful in troubleshooting. To enable this feature please
set the SPLUNK_DEBUG_CONFIG_SERVER
environment variable to true
. To set the desired port to
listen to configure the SPLUNK_DEBUG_CONFIG_SERVER_PORT
environment variable.
You can use the environment variable SPLUNK_LISTEN_INTERFACE
and associated installer option to configure the network
interface on which the collector's receivers and telemetry endpoints will listen.
The default value of SPLUNK_LISTEN_INTERFACE
is set to 127.0.0.1
for the default agent configuration and 0.0.0.0
otherwise.
The following changes need to be done to configuration files for Splunk OTel Collector for specific version upgrades. We provide automated scripts included in the bundle that cover backward compatibility on the fly, but configuration files will not be overridden, so you need to update them manually before the backward compatibility is dropped. For every configuration update use the default agent config as a reference.
memory_ballast
is no longer effective. The garbage collection is now controlled by the soft memory limit set to 90%
of total memory (SPLUNK_MEMORY_TOTAL_MIB
env var) by default.
If you haven't customized the memory_ballast
, just remove it from the configuration.
If you have customized it via SPLUNK_BALLAST_SIZE_MIB
(or extensions::memory_ballast::size_mib
config), you should
remove the memory_ballast
extension and use the GOMEMLIMIT
environment variable to set a custom soft memory limit:
GOMEMLIMIT
to a higher value than the default 90% of total memory.GOMEMLIMIT
to a lower value than the default 90% of total memory.gke
and gce
resource detectors in resourcedetection
processor are replaced with gcp
resource detector.
If you have gke
and gce
detectors configured in the resourcedetection
processor, please update your
configuration accordingly. More details: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/10347 $
symbol with $$
to prevent unwanted environment variable expansion. The issue was fixed in
0.42.0 version. Any occurrences of $$
in your configuration should be replaced with $
.Configuration parameter "exporters
-> otlp
-> insecure
" is moved to
"exporters
-> otlp
-> tls
-> insecure
".
More details: https://github.com/open-telemetry/opentelemetry-collector/pull/4063/.
Configuration part for otlp
exporter should look like this:
exporters:
otlp:
endpoint: "${SPLUNK_GATEWAY_URL}:4317"
tls:
insecure: true
ballast_size_mib
parameter moved from memory_limiter
processor to memory_ballast
extension
as size_mib
.
More details: https://github.com/signalfx/splunk-otel-collector/pull/567.
Remove ballast_size_mib
parameter from memory_limiter
and make sure that it's added to
memory_ballast
extension as size_mib
parameter instead:
extensions:
memory_ballast:
size_mib: ${SPLUNK_BALLAST_SIZE_MIB}
It is possible to use the upstream OpenTelemetry Collector instead of this distribution. The following features are not available upstream at this time:
:warning: Splunk only provides best-effort support for upstream OpenTelemetry
In order to use the upstream OpenTelemetry Collector:
An example configuration for upstream, that ensures infrastructure correlation is properly configured, is available here.
Apache Software License version 2.0.
ℹ️ SignalFx was acquired by Splunk in October 2019. See Splunk SignalFx for more information.