[Windows Integration]: Error installing windows 1.47.0

riahc3 commented 3 months ago

Integration Name

Windows [windows]

Dataset Name

No response

Integration Version

1.47.0

Agent Version

8.13.4

Agent Output Type

elasticsearch

Elasticsearch Version

8.13.4

OS Version and Architecture

Windows 10, Windows 11 and Debian 12

Software/API Version

No response

Error Message

Error installing windows 1.47.0: mapper_parsing_exception Caused by: illegal_argument_exception: analyzer [powershell_script_analyzer] has not been configured in mappings Root causes: mapper_parsing_exception: Failed to parse mapping: analyzer [powershell_script_analyzer] has not been configured in mappings

Event Original

No response

What did you do?

I tried to add Windows Integration

What did you see?

Error installing windows 1.47.0: mapper_parsing_exception Caused by: illegal_argument_exception: analyzer [powershell_script_analyzer] has not been configured in mappings Root causes: mapper_parsing_exception: Failed to parse mapping: analyzer [powershell_script_analyzer] has not been configured in mappings

What did you expect to see?

The integration properly added

Anything else?

No response

elasticmachine commented 3 months ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

riahc3 commented 3 months ago

Thank you @andrewkroh

elasticmachine commented 3 months ago

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

andrewkroh commented 3 months ago

I was not able to reproduce this on 8.13.4. I spun up a new cluster for this test. Did you do anything different 🤔?

https://github.com/user-attachments/assets/cf64e4f7-5954-473d-b940-c294c022861e

riahc3 commented 3 months ago

I was not able to reproduce this on 8.13.4. I spun up a new cluster for this test. Did you do anything different 🤔?

elastic-8.13.4-windows-1.47.0-install-assets.mov

Nope:

https://github.com/user-attachments/assets/0ef0e521-ef4c-4d81-8583-cb8a393d79c4

Nothing different. Doesnt work and if I reboot the machine, I get the screen to install the integration again.

Elastic, Kibana and Logstash all on the same machine.

marc-gr commented 3 months ago

Did you have any previous version of the integration installed before? Or is this a fresh install?

riahc3 commented 3 months ago

Did you have any previous version of the integration installed before? Or is this a fresh install?

Fresh install. Not a upgraded build from anything previous.

Fortigate integration is installed, configured, and setup and firewalls are sending logs to here but besides that.....nothing else I could think up

marc-gr commented 3 months ago

can you check if you have any logs-windows.powershell* indices? If so, can you delete them, and retry?

jameswiggins commented 3 months ago

I am having the same issue. I am trying to upgrade the Windows integration from 1.2.2 to 1.47.0. I do not have any logs-windows.powershell* indices. My cluster is on 8.14.2.

jameswiggins commented 3 months ago

Deleting Windows related data streams is what allowed me to update the integration without error.

I didn't see any indices originally because they were hidden indices. Had to toggle on "include hidden indices" and discovered I actually did have these:

.ds-logs-windows.powershell-default-2022.10.21-000023
.ds-logs-windows.powershell_operational-default-2022.10.21-000023
.ds-metrics-windows.perfmon-default-2022.10.21-000023
.ds-metrics-windows.service-default-2022.10.21-000023

I deleted the associated data streams:

metrics-windows.service-default
metrics-windows.perfmon-default
logs-windows.powershell-default
logs-windows.powershell_operational-default

I also added the below to these component templates logs-windows.forwarded@custom, logs-windows.powershell@custom, logs-windows.powershell_operational@custom but I don't think that helped.

,
    "analysis": {
      "analyzer": {
        "powershell_script_analyzer": {
          "pattern": "[\\W&&[^-]]+",
          "type": "pattern"
        }
      }
    }

marc-gr commented 3 months ago

Thanks for confirming that @jameswiggins , in your case it seems it was the upgrade process that did not let you add the analyzer with the open datastreams.

@riahc3 can you confirm you do not have such datastreams/indices, also?

cc @elastic/fleet maybe something to take a look into here

riahc3 commented 3 months ago

logs-windows.powershell

I dont see anything?

riahc3 commented 3 months ago

Thanks for confirming that @jameswiggins , in your case it seems it was the upgrade process that did not let you add the analyzer with the open datastreams.

@riahc3 can you confirm you do not have such datastreams/indices, also?

cc @elastic/fleet maybe something to take a look into here

CONFIRMED. In data stream, I searched logs-windows.powershell , deleted everything and the integration installed perfectly.

Seems to be a bug.

belimawr commented 3 months ago

I've been looking into that and ES documetation states:

The analyzer setting can not be updated on existing fields using the update mapping API.

Doing some testing, I believe I managed to set the analyser using the following PUT request:

PUT logs-windows.powershell-default/_settings?reopen=true
{
  "analysis": {
    "analyzer": {
      "powershell_script_analyzer":{
        "type": "pattern",
        "pattern":"[\\W&&[^-]]+"
      }
    }
  }
}

Looking at the git history, this analyser got added in 1.4.0. I believe any upgrade from before 1.4.0 to 1.4.0 or newer will fail with this error.

I'll look if there is a way to get the integration to update without deleting the datastreams.

belimawr commented 3 months ago

To upgrade the integration I only needed to delete:

logs-windows.powershell-default
logs-windows.powershell_operational-default

elastic / integrations