Bug: MISP elastic-agent integration don't get any logs in Kibana discover view

[Nicolas-Pellletier]

Hello,

I use docker-elk stack. I use the following configuration for the elsatic-agent: elastic-agent.yml : Notice that I didn't add any Elastic-agent/System monitoring to just have an index linked to MISP, but i tried with it as well and i well receive logs in Kibana:

id: 13b45530-cbe8-11ed-9937-055ed48fa1ae
revision: 2
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://localhost:9200'
    username: 'elastic'
    password: 'changeme'
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices: []
    _elastic_agent_checks:
      cluster:
        - monitor
    9612d037-1eb3-4e88-8654-003fafae8a1d:
      indices:
        - names:
            - logs-ti_misp.threat-default
          privileges:
            - auto_configure
            - create_doc
agent:
  download:
    sourceURI: 'https://artifacts.elastic.co/downloads/'
  monitoring:
    enabled: false
    logs: false
    metrics: false
inputs:
  - id: httpjson-ti_misp-9612d037-1eb3-4e88-8654-003fafae8a1d
    name: MISP
    revision: 1
    type: httpjson
    use_output: default
    meta:
      package:
        name: ti_misp
        version: 1.10.1
    data_stream:
      namespace: default
    package_policy_id: 9612d037-1eb3-4e88-8654-003fafae8a1d
    streams:
      - id: httpjson-ti_misp.threat-9612d037-1eb3-4e88-8654-003fafae8a1d
        data_stream:
          dataset: ti_misp.threat
          type: logs
        config_version: '2'
        interval: 10m
        request.method: POST
        request.url: 'https://localhost/events/restSearch'
        request.ssl:
          verification_mode: none
        request.timeout: 30s
        request.body: null
        request.transforms:
          - set:
              target: header.Authorization
              value: BEpdSXuPb2lRyhVjNy9nHiA7EApYdD9ajMRafBZQ
          - set:
              target: body.page
              value: 1
          - set:
              target: body.limit
              value: 4
          - set:
              target: body.returnFormat
              value: json
          - set:
              target: body.timestamp
              value: '[[.cursor.timestamp]]'
              default: '[[ formatDate (now (parseDuration "-600h")) "UnixDate" ]]'
        response.split:
          target: body.response
          split:
            target: body.Event.Attribute
            ignore_empty_value: true
            keep_parent: true
            split:
              target: body.Event.Object
              keep_parent: true
              split:
                target: body.Event.Object.Attribute
                keep_parent: true
        response.request_body_on_pagination: true
        response.pagination:
          - set:
              target: body.page
              value: >-
                [[if (ne (len .last_response.body.response) 0)]][[add
                .last_response.page 1]][[end]]
              fail_on_template_error: true
        cursor:
          timestamp:
            value: '[[.last_event.Event.timestamp]]'
        tags:
          - forwarded
          - misp-threat
        publisher_pipeline.disable_host: true

When i launch the agent with sudo ./elastic-agent install:

1°) An index is created in elasticsearch view (/_cat/indices): yellow open .ds-logs-ti_misp.threat-default-2023.03.26-000001 WtqVxyThQdKN2yq8uD1jKA 1 1 968156 0 444mb 444mb

2°) I well see the MISP information (about attributes) as documents stored/hits: /.ds-logs-ti_misp.threat-default-2023.03.26-000001/_search?size=100

But got nothing in Kibana logs view (Analytics - Discover part - Logs/Metrics view).

Then i erase all docker components and restart all from scratch. This means that the integration part (agent policy, standalone agent, Kibana and elasticsearch assets for MISP) is gone as well...

But i still used the old elastic-agent.yml file from the previous run, and guess what, i receive logs in Kibana logs view (from MISP). (see the picture below)

Then i add integration policies and install Kibana and Elastic assets for MISP and then back again got nothing in the logs view

In an early test as well i've noticed that when i had in MISP instance, MISP events without any attributes attach to them i well receive them in Kibana logs (but only those ones whose was for test purpose. Misp events without attributes doesn't exist in real). That's very strange because i've read a pull request linked to that: https://github.com/elastic/integrations/pull/5390. (feature drop empty event sets). For me the empty event test is the only one that i got.

Here are some of dicuss elastic.co forum exchange that i had (order by revelance):

https://discuss.elastic.co/t/elastic-agent-process-another-repeated-request-in-loop-indefinitely/328251. https://discuss.elastic.co/t/elastic-agent-with-misp-integration-policy-no-data-received-while-no-errors-comes-up/328183

If you can help me resolve this issue that would be very kind of you. Thanks

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[RiBro]

👍 I have the same problem. Create a dataview which has the ingested field as timestamp helps on this side. However the dashboards are still empty

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)