search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.61k stars 8.22k forks source link

[Security Solution]Endpoint Security installation fails on windows server machines #86766

Closed ghost closed 3 years ago

ghost commented 3 years ago

Describe the bug Endpoint Security installation fails on windows server machines

Build Details:

Platform : Stagging
Version : 8.0.0-SNAPSHOT
Commit : a76c666a6153b062d2f9d97b5c7e87620339a181
Build:39157
Platform: Production
version: 7.10.1
Commit:608c5e5dd32659e8afadd520d0cdc58766ba505b
Build: 36063

8.0.0-Artifact Page 7.10.1-Artifact PAge

Browser Details All

OS

Preconditions

  1. Elastic Cloud environment should be available having "8.0.0-SNAPSHOT" ES version running.
  2. window server machine [ eg. windows 2012R2 and 2019 ] should be available.

Steps to Reproduce

  1. Navigate to Fleet / Policies page and add "endpoint-security" integration under default policy
  2. Fetch the Kibana URL and Token.
  3. RDP to the server machine [ eg. windows 2012R2 and 2019 ]
  4. Installation the Agent + Endpoint Security on the server machine
  5. Observed that Agents installed successfully ,but endpoint-security fails to install on windows server machines.

`Note: issue is also occurring on 7.10.1'

Test data N/A

Impacted Test case(s) N/A

Actual Result Endpoint Security installation fails on windows server machines

Expected Result Endpoint Security should successfully install on windows server machines

What's Working

What's not Working

Screenshots

Logs

Logs: Error: Application: endpoint-security--7.10.1[2ae6a386-df1c-4c96-932c-9065eda984da]: State changed to FAILED: operation 'Exec' failed: No Logs found of endpoint under "C:\Program Files\Elastic" directory image

ghost commented 3 years ago

@manishgupta-qasource Please review

manishgupta-qasource commented 3 years ago

Reviewed & Assigned to @EricDavisX

EricDavisX commented 3 years ago

@ferullo is this one for your group to research or you want the Agent side to poke it first? @ph I tried to get a quick second opinion on our 8.0 based Demo / Test env but it has a deploy problem so could not tell us anything - but even just days ago it was working fine to deploy to Win 2019 daily (I think last time I checked was before the 7.11 BC1 build)

ferullo commented 3 years ago

Is test signing enabled or disabled on this machine?

ghost commented 3 years ago

Hi @ferullo

Test-Signing was enabled on both the server machines

thanks !!

ferullo commented 3 years ago

Hi @karanbirsingh-qasource could you please try installing Endpoint without Agent so we can see log messages from when Endpoint tries to install. This is not a supported way to run Endpoint so please don't try to do further testing on Endpoint other than install (and if it happens to work, uninstall).

  1. Make sure Agent is not installed (to help minimize variables for this test)
  2. From the 8.0.0 Artifact Page you linked to download endpoint-security-8.0.0-SNAPSHOT-windows-x86_64.zip.
  3. Extract it. (unzip endpoint-security-8.0.0-SNAPSHOT-windows-x86_64.zip)
  4. Enter the created subdirectory (cd endpoint-security-8.0.0-SNAPSHOT-windows-x86_64)
  5. Install Endpoint and share the output back to us (endpoint-security.exe install --resources endpoint-security-resources.zip --log stdout --log-level trace)
  6. If Endpoint installed, uninstall it (endpoint-security.exe uninstall --log stdout --log-level trace)
ghost commented 3 years ago

Thanks @ferullo for providing the detailed steps for only endpoint security installation .

we have followed the shared steps on the freshly created machines [ 2019 & 2012R2 Server Machine] .

Please find below complete details.

Build Details;

Platform: Staging
Version: 8.0.0-SNAPSHOT
Commit:a5cfc7fb4a7ce0363b73a118299e228e925814df
Build:39211

endpoint-security-8.0.0-SNAPSHOT-windows-x86_64.zip

2019 Server Machine

4. Enter the created subdirectory (`cd endpoint-security-8.0.0-SNAPSHOT-windows-x86_64`)

image

  1. Install Endpoint and share the output back to us (endpoint-security.exe install --resources endpoint-security-resources.zip --log stdout --log-level trace) CMD logs:

    C:\Program Files\endpoint-security-8.0.0-SNAPSHOT-windows-x86_64>endpoint-security.exe install --resources endpoint-security-resources.zip --log stdout --log-level trace
2020-12-29 07:58:11: info: Main.cpp:228 Executing install (protected)
2020-12-29 07:58:11: info: InstallLib.cpp:181 Installing from endpoint-security-resources.zip
2020-12-29 07:58:11: info: Internal.cpp:188 Extracting installation artifacts
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\elastic-endpoint.yaml
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Windows\System32\Drivers\elastic-endpoint-driver.sys
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Windows\System32\Drivers\eelam.sys
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\SecurityProductInformation.ini
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-model
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-exceptionlist
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-blocklist
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\diagnostic-ransomware-v1-windows
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\global-exceptionlist-windows
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\global-trustlist-windows-v1
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\diagnostic-configuration-v1
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.json
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.sig
2020-12-29 07:58:12: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\resources\elastic-endpoint-security.png
2020-12-29 07:58:12: debug: Util.cpp:721 Creating service to start "C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe" run
2020-12-29 07:58:12: info: Util.cpp:317 Endpoint restart settings [ElasticEndpoint] count=15 delay=15 reset=600
2020-12-29 07:58:12: debug: Service.cpp:803 PPL is supported
2020-12-29 07:58:12: info: Util.cpp:596 PPL is enabled
2020-12-29 07:58:12: debug: Util.cpp:246 Setting up minifilter registry keys successful.

  2. If Endpoint installed, uninstall it (endpoint-security.exe uninstall --log stdout --log-level trace) No Endpoint entry reflect on Security/Administration/Endpoints page image

image

Endpoint Logs (location : C:\Users\zeus\Desktop\Endpoint\state\log) endpoint-000000-2019-Machine.zip

2012 R2 Server Machine

4. Enter the created subdirectory (`cd endpoint-security-8.0.0-SNAPSHOT-windows-x86_64`)

image

  1. Install Endpoint and share the output back to us (endpoint-security.exe install --resources endpoint-security-resources.zip --log stdout --log-level trace) CMD logs: 
    
C:\Windows\system32>cd C:\Program Files\endpoint-security-8.0.0-SNAPSHOT-windows
-x86_64

C:\Program Files\endpoint-security-8.0.0-SNAPSHOT-windows-x86_64>endpoint-securi ty.exe install --resources endpoint-security-resources.zip --log stdout --log-le vel trace 2020-12-29 07:03:26: info: Main.cpp:228 Executing install (protected) 2020-12-29 07:03:26: info: InstallLib.cpp:181 Installing from endpoint-security- resources.zip 2020-12-29 07:03:26: info: Internal.cpp:188 Extracting installation artifacts 2020-12-29 07:03:26: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\elastic-endpoint.yaml 2020-12-29 07:03:26: info: Internal.cpp:353 Writing installation file C:\Windows \System32\Drivers\elastic-endpoint-driver.sys 2020-12-29 07:03:26: info: Internal.cpp:353 Writing installation file C:\Windows \System32\Drivers\eelam.sys 2020-12-29 07:03:26: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\SecurityProductInformation.ini 2020-12-29 07:03:26: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe 2020-12-29 07:03:27: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-model 2020-12-29 07:03:27: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-exception list 2020-12-29 07:03:27: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-blocklist

2020-12-29 07:03:27: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\diagnostic-ransomware-v 1-windows 2020-12-29 07:03:27: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\global-exceptionlist-wi ndows 2020-12-29 07:03:27: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\global-trustlist-window s-v1 2020-12-29 07:03:27: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\diagnostic-configuratio n-v1 2020-12-29 07:03:27: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.json 2020-12-29 07:03:27: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.sig 2020-12-29 07:03:27: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\resources\elastic-endpoint-security.png 2020-12-29 07:03:27: debug: Util.cpp:721 Creating service to start "C:\Program F iles\Elastic\Endpoint\elastic-endpoint.exe" run 2020-12-29 07:03:27: info: Util.cpp:317 Endpoint restart settings [ElasticEndpoi nt] count=15 delay=15 reset=600 2020-12-29 07:03:27: debug: Service.cpp:803 PPL is supported 2020-12-29 07:03:27: info: Util.cpp:596 PPL is enabled 2020-12-29 07:03:27: debug: Util.cpp:246 Setting up minifilter registry keys suc cessful.


>     6. If Endpoint installed, uninstall it (`endpoint-security.exe uninstall --log stdout --log-level trace`)
No Endpoint entry reflect on `Security/Administration/Endpoints` page
![image](https://user-images.githubusercontent.com/59917825/103267682-10e2e180-49d8-11eb-97f2-d361bacf3077.png)

![image](https://user-images.githubusercontent.com/59917825/103268403-623fa080-49d9-11eb-9745-59e5d90bc18a.png)

>Endpoint Logs (location : C:\Users\zeus\Desktop\Endpoint\state\log) [endpoint-000000-2012-R2-Machine.zip](https://github.com/elastic/kibana/files/5749947/endpoint-000000-2012-R2-Machine.zip)

**Additional Observation:**
1. User is able to read endpoint files through file explorer on window server 2012 R2 . However "access denied" is thrown on reading endpoint files on window server 2019
![image](https://user-images.githubusercontent.com/59917825/103268129-fb21ec00-49d8-11eb-9649-a0d7c408a9df.png)
![image](https://user-images.githubusercontent.com/59917825/103268123-f6f5ce80-49d8-11eb-8c39-c442ea3fb3c4.png)

**Query:**
1. do we have to edit any yml ( eg : provide cloud id to specify the kibana location) in above shared installation method ( i.e only endpoint-security installation)

Please let us known if any other details required for this issue.

thanks !!
ferullo commented 3 years ago

Thanks @karanbirsingh-qasource. Both Endpoint's installed properly. You need to run the uninstall command to uninstall them now.

I should have been clearer. You didn't see them reflected in Kibana because there is no Agent. Lots of things won't work, like Endpoint appearing in Kibana, without Agent. I asked you to install Endpoint manually without Agent to see what the install logs printed to the console were (thanks for posting them!) to see why it might be failing when Agent runs that command instead of you. The Endpoint install command worked though, so I'm not sure why it fails when Agent runs it.

Can you try installing Endpoint again via Agent following the correct, normal workflow and see what happens. If it still fails to install we'll need help from the Agent team to debug further.

ghost commented 3 years ago

Hi @ferullo

Thanks for providing update on our query.

Could you also please provide the update on below query if expected or not that is User is able to read endpoint files through file explorer on window server 2012 R2 . However "access denied" is thrown on reading endpoint files on window server 2019

image

image

Normal installation for Endpoint that is Endpoint via Agent fleet mode is Blocked 🟣 due to Policy Token inaccessibility as Agent Tab is stuck under loading , will share the observation once this https://github.com/elastic/kibana/issues/86864 issue got fixed.

image

Please find below observation for uninstall command on windows server machine .

2019 Server machine

  1. endpoint-security.exe install --resources endpoint-security-resources.zip --log stdout --log-level trace
C:\Program Files\endpoint-security-8.0.0-SNAPSHOT-windows-x86_64>endpoint-security.exe install --resources endpoint-security-resources.zip --log stdout --log-level trace
2020-12-30 06:37:16: info: Main.cpp:228 Executing install (protected)
2020-12-30 06:37:16: info: InstallLib.cpp:181 Installing from endpoint-security-resources.zip
2020-12-30 06:37:16: info: Internal.cpp:188 Extracting installation artifacts
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\elastic-endpoint.yaml
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Windows\System32\Drivers\elastic-endpoint-driver.sys
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Windows\System32\Drivers\eelam.sys
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\SecurityProductInformation.ini
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-model
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-exceptionlist
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-blocklist
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\diagnostic-ransomware-v1-windows
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\global-exceptionlist-windows
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\global-trustlist-windows-v1
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\diagnostic-configuration-v1
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.json
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.sig
2020-12-30 06:37:16: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\resources\elastic-endpoint-security.png
2020-12-30 06:37:16: debug: Util.cpp:721 Creating service to start "C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe" run
2020-12-30 06:37:16: info: Util.cpp:317 Endpoint restart settings [ElasticEndpoint] count=15 delay=15 reset=600
2020-12-30 06:37:17: debug: Service.cpp:803 PPL is supported
2020-12-30 06:37:17: info: Util.cpp:596 PPL is enabled
2020-12-30 06:37:17: debug: Util.cpp:246 Setting up minifilter registry keys successful.

  1. endpoint-security.exe uninstall --log stdout --log-level trace

    C:\Program Files\endpoint-security-8.0.0-SNAPSHOT-windows-x86_64>endpoint-security.exe uninstall --log stdout --log-level trace
2020-12-30 06:44:40: info: Main.cpp:241 Executing uninstall
2020-12-30 06:44:40: debug: Service.cpp:803 PPL is supported
2020-12-30 06:44:40: info: Util.cpp:491 Sending sevice command to facilitate uninstall
2020-12-30 06:44:40: info: Util.cpp:518 Service command to facilitate uninstall succeeded
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Windows\ELAMBKUP\eelam.sys]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\elastic-endpoint.yaml]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Windows\System32\Drivers\elastic-endpoint-driver.sys]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Windows\System32\Drivers\eelam.sys]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\SecurityProductInformation.ini]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-model]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-exceptionlist]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-blocklist]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\diagnostic-ransomware-v1-windows]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\global-exceptionlist-windows]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\global-trustlist-windows-v1]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\diagnostic-configuration-v1]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.json]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.sig]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\resources\elastic-endpoint-security.png]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\state\\documents-0.cfg]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\state\\documents-1.cfg]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\state\\documents-2020-12-30T013841.log]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\state\\documents-2020-12-30T014217.log]
2020-12-30 06:44:42: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\state\\log\endpoint-000000.log]
2020-12-30 06:44:42: trace: File.cpp:343 Function returned error status (Failure in an external software component)
2020-12-30 06:44:42: trace: File.cpp:343 Function returned error status (Failure in an external software component)
2020-12-30 06:44:42: info: InstallLib.cpp:247 Uninstall completed

    2012 R2 Server machine

  2. endpoint-security.exe install --resources endpoint-security-resources.zip --log stdout --log-level trace

    
C:\Program Files\endpoint-security-8.0.0-SNAPSHOT-windows-x86_64>endpoint-security.exe install --resources endpoint-security-resources.zip --log stdout --l
og-level trace
2020-12-30 05:57:48: info: Main.cpp:228 Executing install (protected)
2020-12-30 05:57:48: info: InstallLib.cpp:181 Installing from endpoint-security-resources.zip
2020-12-30 05:57:48: info: Internal.cpp:188 Extracting installation artifacts
2020-12-30 05:57:48: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\elastic-endpoint.yaml
2020-12-30 05:57:48: info: Internal.cpp:353 Writing installation file C:\Windows\System32\Drivers\elastic-endpoint-driver.sys
2020-12-30 05:57:48: info: Internal.cpp:353 Writing installation file C:\Windows\System32\Drivers\eelam.sys
2020-12-30 05:57:48: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\SecurityProductInformation.ini
2020-12-30 05:57:48: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe
2020-12-30 05:57:49: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-mode
l
2020-12-30 05:57:49: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-exce
ptionlist
2020-12-30 05:57:49: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-bloc
klist
2020-12-30 05:57:49: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\diagnostic-ransomw
are-v1-windows
2020-12-30 05:57:49: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\global-exceptionli
st-windows
2020-12-30 05:57:49: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\global-trustlist-w
indows-v1
2020-12-30 05:57:49: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\diagnostic-configu
ration-v1
2020-12-30 05:57:49: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.json
2020-12-30 05:57:49: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.sig
2020-12-30 05:57:49: info: Internal.cpp:353 Writing installation file C:\Program Files\Elastic\Endpoint\cache\resources\elastic-endpoint-security.png
2020-12-30 05:57:49: debug: Util.cpp:721 Creating service to start "C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe" run
2020-12-30 05:57:49: info: Util.cpp:317 Endpoint restart settings [ElasticEndpoint] count=15 delay=15 reset=600
2020-12-30 05:57:49: debug: Service.cpp:803 PPL is supported
2020-12-30 05:57:49: info: Util.cpp:596 PPL is enabled
2020-12-30 05:57:49: debug: Util.cpp:246 Setting up minifilter registry keys successful.

C:\Program Files\endpoint-security-8.0.0-SNAPSHOT-windows-x86_64>


2. endpoint-security.exe uninstall --log stdout --log-level trace

C:\Program Files\endpoint-security-8.0.0-SNAPSHOT-windows-x86_64>endpoint-security.exe uninstall --log stdout --log-level trace 2020-12-30 06:06:36: info: Main.cpp:241 Executing uninstall 2020-12-30 06:06:36: debug: Service.cpp:803 PPL is supported 2020-12-30 06:06:36: info: Util.cpp:491 Sending sevice command to facilitate uninstall 2020-12-30 06:06:36: error: Util.cpp:513 Failure sending DisablePPL message to allow our service to stop. Send status: Success. Command Status: Failure in an external software component 2020-12-30 06:06:36: trace: Util.cpp:515 Function returned error status (Failure in an external software component) 2020-12-30 06:06:36: warning: Util.cpp:869 Error encountered while unprotecting service for unintall 2020-12-30 06:06:38: debug: File.cpp:471 Removing [C:\Windows\ELAMBKUP\eelam.sys] 2020-12-30 06:06:38: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\elastic-endpoint.yaml] 2020-12-30 06:06:38: debug: File.cpp:471 Removing [C:\Windows\System32\Drivers\elastic-endpoint-driver.sys] 2020-12-30 06:06:38: debug: File.cpp:471 Removing [C:\Windows\System32\Drivers\eelam.sys] 2020-12-30 06:06:38: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\SecurityProductInformation.ini] 2020-12-30 06:06:38: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe] 2020-12-30 06:06:38: trace: File.cpp:478 Function returned error status (I/O error) because of system status (5/Access is denied.) 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-model] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-exceptionlist] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\endpointpe-v4-blocklist] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\diagnostic-ransomware-v1-windows] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\global-exceptionlist-windows] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\global-trustlist-windows-v1] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\diagnostic-configuration-v1] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.json] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.sig] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\resources\elastic-endpoint-security.png] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\cache\artifacts\global-artifacts\manifest.etag] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\state\documents-0.cfg] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\state\documents-1.cfg] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\state\documents-2020-12-30T010139.log] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\state\documents-2020-12-30T010528.log] 2020-12-30 06:06:39: debug: File.cpp:471 Removing [C:\Program Files\Elastic\Endpoint\state\log\endpoint-000000.log] 2020-12-30 06:06:40: trace: File.cpp:343 Function returned error status (Failure in an external software component) 2020-12-30 06:06:40: trace: File.cpp:343 Function returned error status (Failure in an external software component) 2020-12-30 06:06:40: trace: File.cpp:343 Function returned error status (Failure in an external software component) 2020-12-30 06:06:40: info: InstallLib.cpp:247 Uninstall completed

C:\Program Files\endpoint-security-8.0.0-SNAPSHOT-windows-x86_64>



**Logs:**
1. 2019 machine:[endpoint-000000-2019-machine.log](https://github.com/elastic/kibana/files/5753182/endpoint-000000-2019-machine.log)
2. 2012R2 machine:[endpoint-000000-2012-R2-machine.log](https://github.com/elastic/kibana/files/5753183/endpoint-000000-2012-R2-machine.log)

Please let us known if any other details required .

thanks !!
ferullo commented 3 years ago

Receiving "Access is Denied" on 2019 is expected. This is Endpoint self-protection stopping File Explorer from being able to read it's files. You can still access Endpoint files using an Administrator cmd.exe window.

It's surprising you could browse into Endpoint's directory on 2021 R2. @bjmcnic do you have a way to try to replicate this? It would be good to understand and document if needed.

bjmcnic commented 3 years ago

I've reproduced the failure on a Win 8.1 VM (8.1 ˜= 2012 R2).

{"@timestamp":"2020-12-30T14:36:18.2084793Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"error","origin":{"file":{"line":301,"name":"File.cpp"}}},"message":"File.cpp:301 SetNamedSecurityInfoW() failed: 5","process":{"pid":4540,"thread":{"id":4552}}}

{"@timestamp":"2020-12-30T14:36:18.2084793Z","agent":{"id":"","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"error","origin":{"file":{"line":1034,"name":"Util.cpp"}}},"message":"Util.cpp:1034 Apply process trust label for: C:\\Program Files\\Elastic\\Endpoint\\ failed with: Failure in an external software component","process":{"pid":4540,"thread":{"id":4552}}}

Looks like we're getting ERROR_ACCESS_DENIED applying the process trust label SACL. We are running as a protected process though, so I'm not yet sure why we failed. Digging into this.

ferullo commented 3 years ago

@karanbirsingh-qasource did you try installing via Agent again to see what happens? It seems like both Server 2012R2 and Server 2019 will work, with the exception of part of self protection not turning on for Server 2012R2.

ghost commented 3 years ago

@karanbirsingh-qasource did you try installing via Agent again to see what happens? It seems like both Server 2012R2 and Server 2019 will work, with the exception of part of self protection not turning on for Server 2012R2.

Normal installation for Endpoint that is Endpoint via Agent fleet mode is Blocked 🟣 due to Policy Token inaccessibility as Agent Tab is stuck under loading , will share the observation once this #86864 issue got fixed.

image

ghost commented 3 years ago

Hi @ferullo

Please find addition observation for the Agent and Endpoint Security installation status on 7.11.0 & 8.0.0 SNAPSHOT and 7.11.0 BC1.

Agent and Endpoint Security installation Observation

Platform Version Commit Build Test-Signing ON Test-Signing ON Test-Signing OFF Test-Signing OFF Remark
        Agent Only installation Agent + Endpoint installation Agent Only installation Agent + Endpoint installation  
Staging 8.0.0-SNAPSHOT a5cfc7fb4a7ce0363b73a118299e228e925814df  39211  Blocked 🟣 Blocked  🟣 N/A N/A  https://github.com/elastic/kibana/issues/86864
Staging 7.11.0-SNAPSHOT 876e859238cdc8ec487ca2608aacb1aa7faaa958 37480 Unhealthy Agent🔴 No Endpoint Reflect under admin tab🔴 N/A N/A https://github.com/elastic/kibana/issues/86097
Production 7.11.0 BC1  f3abc08ac648f8b302733c5c22a39048314a027c 37399  Unhealthy Agent ( keeps in unhealthy through out )🔴  Endpoint not installed🔴  Unhealthy Agent ( keeps in unhealthy through out )🔴  Endpoint not installed🔴 https://github.com/elastic/beats/issues/23332

Note: Additional Observation for 7.11.0 BC1

while looking into agent/endpoint installation on 7.11.0 BC1 we have found a way by which we get Healthy Agent + Endpoint Installed under Admin Tab. Please find below complete details for the same.

Steps:

  1. Create a policy having only system integration or use the fresh default policy.
  2. Install the agent on windows vm 64 bit
  3. Agent installed with Unhealthy State. Logs : 
    [elastic_agent][error] failed to load markeropen C:\Program Files\Elastic\Agent\data\.update-marker: The system cannot find the file specified.
[elastic_agent][error] Elastic Agent status changed to: 'error'
  4. Edit the Policy created in step 1 or Default policy and add Endpoint Security Integration.
  5. After a minute or 2 agent becomes healthy and endpoint entry also start reflecting under admin tab . image image image image

Condition for above steps is : Endpoint Security should not be added after agent installation . If we create a policy with both integration (security + endpoint security ) first and then install agent then agent stuck in unhealthy state and no endpoint reflect under admin tab . image

Success steps video: https://user-images.githubusercontent.com/59917825/103409255-f3ef0f80-4b8b-11eb-83b0-258268f63df0.mp4

Failure steps video: https://user-images.githubusercontent.com/59917825/103409259-f8b3c380-4b8b-11eb-9bd6-2dcc21ee216e.mp4

Snap-Shoots:

image image

Logs:

14:46:55.473
elastic_agent
[elastic_agent][error] failed to dispatch actions, error: operator: failed to execute step sc-run, error: operation 'Exec' failed: : operation 'Exec' failed: 
14:47:02.502
elastic_agent
[elastic_agent][error] failed to dispatch actions, error: operator: failed to execute step sc-run, error: operation 'Exec' failed: : operation 'Exec' failed: 
4:08:44.963
elastic_agent
[elastic_agent][error] failed to load markeropen C:\Program Files\Elastic\Agent\data\.update-marker: The system cannot find the file specified.
14:15:57.363
elastic_agent
[elastic_agent][error] Could not communicate with Checking API will retry, error: Status code: 503, Kibana returned an error: Service Unavailable, message: socket hang up
14:27:37.705
elastic_agent
[elastic_agent][error] Could not communicate with Checking API will retry, error: Status code: 503, Kibana returned an error: Service Unavailable, message: socket hang up

[elastic_agent][error] failed to load markeropen C:\Program Files\Elastic\Agent\data\.update-marker: The system cannot find the file specified.

[elastic_agent][error] failed to dispatch actions, error: operator: failed to execute step sc-run, error: operation 'Exec' failed: : operation 'Exec' failed: 
[elastic_agent][error] Could not communicate with Checking API will retry, error: Status code: 503, Kibana returned an error: Service Unavailable, message: socket hang up
[elastic_agent][error] 2020-12-31T05:52:00-05:00: type: 'ERROR': sub_type: 'FAILED' message: Application: endpoint-security--7.11.0[1e08dca0-4b56-11eb-8e3c-b37ff8759758]: State changed to FAILED: operation 'Exec' failed:

C.C @EricDavisX @kevinlog

ferullo commented 3 years ago

@ph @EricDavisX based on @karanbirsingh-qasource 's last comment, it sounds like this is the same unhealthy Agent issue that the Agent team is already aware of. Do you agree?

ghost commented 3 years ago

HI @EricDavisX @ferullo

We have re-tested this issue on latest commit of 8.0.0-SNAPSHOT build and found that now this issue has been fixed. Now agents are successfully installing on server machines.

Build Details:

Platform: Staging
Version: 8.0.0
Commit : 52f6c7cf5c1eb4d8d68c726af6ac91583cabe453
Build : 39299

Snapshots:

image

Hence we are closing this issue.

thanks !!