elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
28 stars 444 forks source link

Fleet Package Policy API: Error Adding Winlog Integration - Stream Template Not Found #9769

Open zedtran opened 6 months ago

zedtran commented 6 months ago

Full disclosure: I detail this further in the official Elastic support portal under case #01610754.


Noticing an issue on a fresh stack version 8.12.1 install where the create fleet package policy API for the winlog integration fails.

Data stream backing index template "logs-winlog.winlog", an ingest pipeline, and component templates "logs-winlog.winlog@package" and "logs-winlog.winlog@custom" are not loaded with an apparent Kibana log message error which reads:

[2024-04-30T20:34:20.445+00:00][ERROR][plugins.fleet] Error: Stream template not found, unable to find dataset winlog.winlog
    at _compilePackageStream (/usr/share/kibana/node_modules/@kbn/fleet-plugin/server/services/package_policy.js:1442:11)
    at /usr/share/kibana/node_modules/@kbn/fleet-plugin/server/services/package_policy.js:1391:55
    at Array.map (<anonymous>)
    at _compilePackageStreams (/usr/share/kibana/node_modules/@kbn/fleet-plugin/server/services/package_policy.js:1391:41)
    at /usr/share/kibana/node_modules/@kbn/fleet-plugin/server/services/package_policy.js:1356:35
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Promise.all (index 0)

Did a manual upload of winlog version 2.1.1 where you can see the .kibana_ingest document referenced by _id: epm-packages:winlog appears to be missing objects/refs typically listed in _source['epm-packages]['installed_es'].

GET .kibana_ingest/_search?q=epm-packages.name:winlog
{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1.3862942,
    "hits": [
      {
        "_index": ".kibana_ingest_8.12.1_001",
        "_id": "epm-packages:winlog",
        "_score": 1.3862942,
        "_source": {
          "epm-packages": {
            "installed_kibana": [],
            "installed_kibana_space_id": "default",
            "installed_es": [],
            "package_assets": [
              {
                "id": "293a22b2-8e4e-5c7d-8249-6b32a38a651b",
                "type": "epm-packages-assets"
              },
              {
                "id": "2367f0f5-ce06-5e20-b9dc-75f5807da180",
                "type": "epm-packages-assets"
              },
              {
                "id": "cc49d170-a6c9-56b3-a40f-fa662f700661",
                "type": "epm-packages-assets"
              },
              {
                "id": "c9576881-c05a-5c54-99aa-40f2c0fefd7d",
                "type": "epm-packages-assets"
              },
              {
                "id": "b3de468f-d2e5-5ab6-9e95-ade91d0371e4",
                "type": "epm-packages-assets"
              },
              {
                "id": "d8fb4d51-b5ff-5446-8a75-d9da1ecde649",
                "type": "epm-packages-assets"
              },
              {
                "id": "fec7a0e7-5d0a-5ad0-bad3-e6bddc286ccd",
                "type": "epm-packages-assets"
              },
              {
                "id": "db6a867f-8dcc-5bac-8289-c20ce3b8bb9f",
                "type": "epm-packages-assets"
              },
              {
                "id": "eae37891-e71b-5bad-b369-c717ba725f92",
                "type": "epm-packages-assets"
              },
              {
                "id": "ee1dd87b-e640-56d7-a149-7503868a51b3",
                "type": "epm-packages-assets"
              },
              {
                "id": "f51e6b17-6813-576d-bbc0-5aa4d35defd7",
                "type": "epm-packages-assets"
              }
            ],
            "es_index_patterns": {},
            "name": "winlog",
            "version": "2.1.1",
            "install_version": "2.1.1",
            "install_status": "installed",
            "install_started_at": "2024-04-30T19:26:54.301Z",
            "install_source": "upload",
            "install_format_schema_version": "1.1.0",
            "verification_status": "verified",
            "verification_key_id": "d27d666cd88e42b4",
            "latest_install_failed_attempts": []
          },
          "type": "epm-packages",
          "references": [],
          "managed": false,
          "coreMigrationVersion": "8.8.0",
          "typeMigrationVersion": "10.1.0",
          "updated_at": "2024-04-30T19:26:54.903Z",
          "created_at": "2024-04-29T23:46:55.955Z"
        }
      }
    ]
  }
}

Can anyone confirm whether or not they can replicate this issue regarding the PR in subject?

Originally posted by @zedtran in https://github.com/elastic/integrations/issues/8010#issuecomment-2087691580

elasticmachine commented 2 months ago

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)