[AWS] CloudWatch logs integration fails with custom namespace and dataset

[kaiyan-sheng]

When I create a cloudwatch log integration with a custom namespace and dataset name, I got an error:

Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2021, time.August, 23, 22, 37, 42, 0, time.UTC), Meta:{"_id":"36344823736062113326896489163324524257160844400983539712","raw_index":"logs-generic-development"}, Fields:{"agent":{"ephemeral_id":"00856a00-c823-4cae-b466-1ce1a4ee569c","id":"82e3ed3b-3cc9-4dcc-82eb-ba22a838b3f5","name":"KaiyanMacBookPro","type":"filebeat","version":"8.1.1"},"awscloudwatch":{"ingestion_time":"2021-08-23T22:39:02.000Z","log_group":"vpc-flow-logs","log_stream":"eni-0c6c1b04df2cef779-all"},"cloud":{"provider":"aws","region":"us-east-1"},"data_stream":{"dataset":"generic","namespace":"development","type":"logs"},"ecs":{"version":"8.0.0"},"elastic_agent":{"id":"82e3ed3b-3cc9-4dcc-82eb-ba22a838b3f5","snapshot":false,"version":"8.1.1"},"event":{"dataset":"generic","id":"36344823736062113326896489163324524257160844400983539712","ingested":"2022-04-15T19:08:57.617Z"},"input":{"type":"aws-cloudwatch"},"log.file.path":"vpc-flow-logs/eni-0c6c1b04df2cef779-all","message":"2 428152502467 eni-0c6c1b04df2cef779 - - - - - - - 1629758262 1629758275 - NODATA","tags":["forwarded","aws-cloudwatch-logs"]}, Private:(*aws.EventACKTracker)(0xc0008b4b70), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [RcqOLoABmYpVVVo44aQW] of user [elastic/fleet-server] on indices [logs-generic-development], this action is granted by the index privileges [auto_configure,create_index,manage,all]"}, dropping event!

Here is the agent policy:

id: 955058d0-bceb-11ec-aeba-5dc0c943dba9
revision: 6
outputs:
  default:
    type: elasticsearch
    hosts:
      - >-
        https://60c02689d0634326981fc8b55bdb50ad.eastus2.azure.elastic-cloud.com:443
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices:
        - names:
            - logs-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
    _elastic_agent_checks:
      cluster:
        - monitor
    aws-1:
      indices:
        - names:
            - logs-aws.cloudwatch_logs-development
          privileges:
            - auto_configure
            - create_doc
agent:
  monitoring:
    enabled: true
    use_output: default
    namespace: default
    logs: true
    metrics: true
inputs:
  - id: aws-cloudwatch-cloudwatch-168581d5-4824-4e98-96c6-e0040dc11cda
    name: aws-1
    revision: 4
    type: aws-cloudwatch
    use_output: default
    meta:
      package:
        name: aws
        version: 1.14.0
    data_stream:
      namespace: development
    streams:
      - id: >-
          aws-cloudwatch-aws.cloudwatch_logs-168581d5-4824-4e98-96c6-e0040dc11cda
        dataset: newdataset
        data_stream: null
        log_group_arn: 'arn:aws:logs:us-east-1:1234:log-group:vpc-flow-logs:*'
        start_position: beginning
        access_key_id: xxx
        api_sleep: 200ms
        endpoint: amazonaws.com
        secret_access_key: yyy
        scan_frequency: 1m
        tags:
          - forwarded
          - aws-cloudwatch-logs
        publisher_pipeline.disable_host: true
fleet:
  hosts:
    - >-
      https://615303b6bcd341d484a3d995f2a4aaaf.fleet.eastus2.azure.elastic-cloud.com:443

We can see from the agent policy, logs-aws.cloudwatch_logs-development is the index with the correct privileges, not logs-generic-development.

[matschaffer]

Looks like this is the same issue showing up in this discuss thread: https://discuss.elastic.co/t/aws-fleet-integration-fails-with-api-key-error-on-ingest/304846/3

[aspacca]

the same seems to be when setting a cloudwatch logs integration without custom namespace and dataset:

- data_stream:
    namespace: default
[...]
  streams:
  - dataset: generic
[...]

the default is logs-generic-default

the permission are on logs-aws.cloudwatch_logs-default

      indices:
      - names:
        - logs-aws.cloudwatch_logs-default
        privileges:
        - auto_configure
        - create_doc

[aspacca]

fixed by #3844