Different processors and namespaces to different data_streams for apm-server

[b2ronn]

i have a test simple configuration.

---
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-apm-sample
  namespace: elastic
spec:
  data_stream:
    namespace: test2
  config:
    apm-server.data_streams.namespace: testname
  count: 1
  elasticsearchRef:
    name: es-apm-sample
  http:
    service:
      metadata: {}
      spec: {}
    tls:
      certificate: {}
  kibanaRef:
    name: kb-apm-sample
  podTemplate:
    metadata:
      creationTimestamp: null
    spec:
      containers: null
  version: 8.0.1

---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: es-apm-sample
  namespace: elastic
spec:
  auth: {}
  http:
    service:
      metadata: {}
      spec: {}
    tls:
      certificate: {}
  monitoring:
    logs: {}
    metrics: {}
  nodeSets:
    - config:
        node.store.allow_mmap: false
      count: 1
      name: default
      podTemplate:
        metadata:
          creationTimestamp: null
        spec:
          containers: null
  transport:
    service:
      metadata: {}
      spec: {}
    tls:
      certificate: {}
  updateStrategy:
    changeBudget: {}
  version: 8.0.1

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kb-apm-sample
  namespace: elastic
spec:
  config:
    xpack.fleet.packages:
      - name: apm
        version: latest
  count: 1
  elasticsearchRef:
    name: es-apm-sample
  http:
    service:
      metadata: {}
      spec: {}
    tls:
      certificate: {}
  monitoring:
    logs: {}
    metrics: {}
  podTemplate:
    metadata:
      creationTimestamp: null
    spec:
      containers: null
  version: 8.0.1

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: petclinic
  namespace: elastic
  labels:
    app.kubernetes.io/component: test-application
    app.kubernetes.io/name: petclinic
    app.kubernetes.io/part-of: elastic-stack
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: test-application
      app.kubernetes.io/name: petclinic
      app.kubernetes.io/part-of: elastic-stack
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: test-application
        app.kubernetes.io/name: petclinic
        app.kubernetes.io/part-of: elastic-stack
    spec:
      volumes:
        - name: elastic-apm-agent
          emptyDir: {}
      initContainers:
        - name: elastic-java-agent
          image: 'docker.elastic.co/observability/apm-agent-java:1.29.0'
          command:
            - cp
            - '-v'
            - /usr/agent/elastic-apm-agent.jar
            - /elastic/apm/agent
          resources: {}
          volumeMounts:
            - name: elastic-apm-agent
              mountPath: /elastic/apm/agent
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      containers:
        - resources: {}
          terminationMessagePath: /dev/termination-log
          name: petclinic
          env:
            - name: ELASTIC_APM_SERVER_URL
              value: 'https://apm-apm-sample-apm-http:8200'
            - name: ELASTIC_APM_SERVICE_NAME
              value: petclinic
            - name: ELASTIC_APM_APPLICATION_PACKAGES
              value: org.springframework.samples.petclinic
            - name: ELASTIC_APM_ENVIRONMENT
              value: dev
            - name: JAVA_TOOL_OPTIONS
              value: '-javaagent:/elastic/apm/agent/elastic-apm-agent.jar'
            - name: ELASTIC_APM_GLOBAL_LABELS
              value: namespace=test333
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: ELASTIC_APM_VERIFY_SERVER_CERT
              value: 'false'
          ports:
            - name: petclinic-http
              containerPort: 8080
              protocol: TCP
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: elastic-apm-agent
              mountPath: /elastic/apm/agent
          terminationMessagePolicy: File
          image: 'arey/springboot-petclinic:latest'
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

traces and metrics are written. but there are a few questions (maybe I didn't find it in the documentation)

1. why don't all transactions have kubernetes.namespace?
2. how to change data_streams.namespace by default?
3. how to make the data written to different datastreams according to the template < type>-- (traces-apm.app.petclinic-elastic, metrics-apm.app.petclinic-elastic) ? in apm-server 7.15 it was, but in 8.0.0 it was deprecated

   output.elasticsearch.index: "apm-%{[observer.version]}-%{[processor.event]}-%{[kubernetes.namespace]}-%{+yyyy.MM.dd}"

   ps:

   • OpenShift version 4.9.10
   • Elasticsearch (ECK) Operator 2.0.0 provided by Elastic

[thbkrkr]

First, the spec.data_stream field in your manifest doesn't exist (API doc).

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-apm-sample
  namespace: elastic
spec:
  data_stream:
    namespace: test2

Your manifest should be rejected:

> kubectl apply -f eck-5441-apm.yml 
error: error validating " eck-5441-apm.yml ": error validating data: ValidationError(ApmServer.spec): 
  unknown field "data_stream" in co.elastic.k8s.apm.v1.ApmServer.spec; 
  if you choose to ignore these errors, turn validation off with --validate=false

Second, the standalone APM Server is now considered as legacy and will be soon deprecated because the Elastic APM integration became generally available starting 7.16. This method of installing APM Server will be removed in a future release. Please consider getting started with the Elastic APM integration instead.

Third, the following questions are not ECK related but directly related to using APM, so I will transfer the issue to get help from the APM team who will be in a better position to answer it.

why don't all transactions have kubernetes.namespace?

how to change data_streams.namespace by default?

how to make the data written to different datastreams according to the template < type>-- (traces-apm.app.petclinic-elastic, metrics-apm.app.petclinic-elastic) ? in apm-server 7.15 it was, but in 8.0.0 it was deprecated

[b2ronn]

spec.data_stream.namespace: test2 - Yes. it was just a test and didn't delete it.

data_stream.namespace by default can be changed through integrations, but I have an elastic-agent with an apm-server in one namespace, and java-agents will write there from other namespaces. through the fleet and the elastic-agent it is also not possible to change the data_stream.namespace for apm and java-agent. data_stream.namespace

---
kind: Namespace
apiVersion: v1
metadata:
  name: elastic
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 8.0.1
  count: 1
  elasticsearchRef:
    name: elasticsearch
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.elastic.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.elastic.svc:8220"]
    xpack.fleet.packages:
    - name: apm
      version: latest
    xpack.fleet.agentPolicies:
    - name: Default Fleet Server on ECK policy
      is_default_fleet_server: true
      package_policies:
      - package:
          name: fleet_server
        name: fleet_server-1
    - name: Default Elastic Agent on ECK policy
      is_default: true
      unenroll_timeout: 900
      package_policies:
      - package:
          name: apm
        name: apm-1
        inputs:
        - type: apm
          enabled: true
          vars:
          - name: host
            value: 0.0.0.0:8200
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 8.0.1
  nodeSets:
  - name: elastic
    count: 1
    config:
      node.store.allow_mmap: false
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
spec:
  image: docker.elastic.co/beats/elastic-agent:8.0.1
  version: 8.0.1
  kibanaRef:
    name: kibana
  elasticsearchRefs:
  - name: elasticsearch
  mode: fleet
  fleetServerEnabled: true
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: fleet-server
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata: 
  name: elastic-agent
spec:
  image: docker.elastic.co/beats/elastic-agent:8.0.1
  version: 8.0.1
  kibanaRef:
    name: kibana
  fleetServerRef: 
    name: fleet-server
  mode: fleet
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: fleet-server
        securityContext:
          runAsUser: 0
---
apiVersion: v1
kind: Service
metadata:
  name: apm
spec:
  selector:
    agent.k8s.elastic.co/name: elastic-agent
  ports:
  - protocol: TCP
    port: 8200
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fleet-server
rules:
- apiGroups: [""]
  resources:
  - pods
  - namespaces
  - nodes
  verbs:
  - get
  - watch
  - list
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs:
  - get
  - create
  - update
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fleet-server
  namespace: elastic
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fleet-server
subjects:
- kind: ServiceAccount
  name: fleet-server
  namespace: elastic
roleRef:
  kind: ClusterRole
  name: fleet-server
  apiGroup: rbac.authorization.k8s.io

[SanjuTechie87]

@b2ronn . Hi, were you able to solve the issue? I am also not able to change the namespace of APM integration through UI, it is always creating index and data stream in default space. I am using eck 8.2.3 . Have created a issue https://github.com/elastic/fleet-server/issues/1628 for the same. Any help much appreciated.

[marclop]

@SanjuTechie87 perhaps unrelated, but we recently fixed a bug in 8.3.0 where changing the APM Server data stream didn't work: https://www.elastic.co/guide/en/apm/guide/current/release-notes-8.3.html#_bug_fixes