search

Ylianst / MeshCentral

A complete web-based remote monitoring and management web site. Once setup you can install agents and perform remote desktop session to devices on the local network or over the Internet.
https://meshcentral.com
Apache License 2.0
4.23k stars 568 forks source link

Meshagent crashes on OS: Windows 10 only during an active connection to the Meshserver #3675

Open btcsupport opened 2 years ago

btcsupport commented 2 years ago

Meshcentral2 Ver: 0.9.79 Meshagent Ver: 0.2.1.3

Case 1: OS: Windows 10 20H2 Meshagent regularly crashes on computers running the W10 operating system while connected to the server. No actions are performed from the Meshcetral console during this time.

There is an error in the event log:

Faulting application name: MeshAgent.exe, version: 0.2.1.3, time stamp: 0x61d893cc
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x03eefb48
Faulting process id: 0x350c
Faulting application start time: 0x01d82344e92d7552
Faulting application path: C:\Program Files (x86)\Mesh Agent\MeshAgent.exe
Faulting module path: unknown
Report Id: e3fe6993-981e-404b-bebd-b82a6933cc82
Faulting package full name: 
Faulting package-relative application ID:

Frequency: 7-10 times / day

The bug also occurs when Microsoft Defender is disabled. The error does not occur when Mesh Agent is not connected to Meshserver.

Tip: 1.There are always entries in the event log at the same time: Updated Windows Defender status successfully to SECURITY_PRODUCT_STATE_ON.

Case 2: OS: Windows Server 2012R2 and Windows Server 2019 Mesh Agent Crash: None Tip: 1.The systems do not have Microsoft Defender?

Case 3: OS: Windows 10

I prepared memory dumps from windows 10 computers: https://btcszczecin-my.sharepoint.com/:u:/g/personal/tbaut_btc_com_pl/EV7TEP2qx05Chr0CDDuej6wBbCMKPeE6pT0COPNbhqTlsw?e=tWxMKv

Videos present case 3: https://btcszczecin-my.sharepoint.com/:u:/g/personal/tbaut_btc_com_pl/EQpSEDhbw2BBhAXUvDoUAK4B43qKVhgHsXcKI-ileDKRbA?e=XZFRbU

krayon007 commented 2 years ago

Is there a .log or .dmp file in the agent installed folder?

krayon007 commented 2 years ago

If you open up Windows Security, and go to "App & Browser Control", and then "Reputation based protection", then under "Potentially unwanted app blocking", click on "protection history". Is anything listed there?

btcsupport commented 2 years ago

In the agent folder there is a file with the log. [2022-02-09 04:04:35 PM] [19F470D41D70288E] ..\microstack\ILibParsers.c:10805 (0,0) Info: No certificate was found in db

In the protection control tab there is only an entry about blocking the meshagent installer.

Detected:Program:Win32/Uwamson.A!ml
Status:Active
Date:2/16/2022 2:48 PM
Details:This program has potentially unwanted behavior.
Affected items: file: C:\Users\***\Desktop\logitot\meshagent32-BTC(2).exe

"logitot" folder contains only .exe with meshagent installer

screens: https://btcszczecin-my.sharepoint.com/:f:/g/personal/tbaut_btc_com_pl/ElaR9aqZH11FrUU0xJIiPRAByIOdLyqREKOn9YG7df0O_Q?e=bWfOxA

krayon007 commented 2 years ago

For case #3, what build number of Windows 10 are you running? I'm trying to recreate the issue, but can't reproduce it. I'm using Windows 10 Pro, 21H1/19043. I left it running all night, with Windows Defender running, and haven't been able to reproduce what you are seeing.

btcsupport commented 2 years ago

In case 3, I am using Windows 10 Enterprise. Version: 20H2 OS Build: 19042,1466 Experience: Windows Feature Experience Pack 120.2212.3920.0

Disk is encrypted with bitlocker. Windows Defender has an exception on the folder with the mesh agent.

https://btcszczecin-my.sharepoint.com/:i:/g/personal/tbaut_btc_com_pl/ESH7mzs2e6hBiHy0r8NYwVoBzsFzyXcm1bzgV7au0pwrlw?e=n3X9XJ

Computer info

`

WindowsBuildLabEx : 19041.1.amd64fre.vb_release.191206-1406 WindowsCurrentVersion : 6.3 WindowsEditionId : Enterprise WindowsInstallationType : Client WindowsInstallDateFromRegistry : 10/6/2021 12:19:18 PM WindowsProductId : 00329-10438-14520-AA067 WindowsProductName : Windows 10 Enterprise WindowsRegisteredOrganization : WindowsRegisteredOwner : admin WindowsSystemRoot : C:\Windows WindowsVersion : 2009 BiosCharacteristics : {7, 9, 11, 12...} BiosBIOSVersion : {HUAWEI - 2, 1.13, XXXXXX - 10013} BiosBuildNumber : BiosCaption : 1.13 BiosCodeSet : BiosCurrentLanguage : en|US|iso8859-1,0 BiosDescription : 1.13 BiosEmbeddedControllerMajorVersion : 1 BiosEmbeddedControllerMinorVersion : 13 BiosFirmwareType : Uefi BiosIdentificationCode : BiosInstallableLanguages : 2 BiosInstallDate : BiosLanguageEdition : BiosListOfLanguages : {en|US|iso8859-1,0, zh|CN|unicode,0} BiosManufacturer : HUAWEI BiosName : 1.13 BiosOtherTargetOS : BiosPrimaryBIOS : True BiosReleaseDate : 8/6/2020 2:00:00 AM BiosSeralNumber : JWEPM20926000582 BiosSMBIOSBIOSVersion : 1.13 BiosSMBIOSMajorVersion : 3 BiosSMBIOSMinorVersion : 2 BiosSMBIOSPresent : True BiosSoftwareElementState : Running BiosStatus : OK BiosSystemBiosMajorVersion : 1 BiosSystemBiosMinorVersion : 13 BiosTargetOperatingSystem : 0 BiosVersion : HUAWEI - 2 CsAdminPasswordStatus : Unknown CsAutomaticManagedPagefile : True CsAutomaticResetBootOption : True CsAutomaticResetCapability : True CsBootOptionOnLimit : CsBootOptionOnWatchDog : CsBootROMSupported : True CsBootStatus : CsBootupState : Normal boot CsCaption : DESKTOP-A85809L CsChassisBootupState : Safe CsChassisSKUNumber : NobelB-WAH9A CsCurrentTimeZone : 60 CsDaylightInEffect : False CsDescription : AT/AT COMPATIBLE CsDNSHostName : DESKTOP-A85809L CsDomain : *.local CsDomainRole : MemberWorkstation CsEnableDaylightSavingsTime : True CsFrontPanelResetStatus : Unknown CsHypervisorPresent : True CsInfraredSupported : False CsInitialLoadInfo : CsInstallDate : CsKeyboardPasswordStatus : Unknown CsLastLoadInfo : CsManufacturer : HUAWEI CsModel : NBLB-WAX9N CsName : DESKTOP-A85809L CsNetworkAdapters : {Wi-Fi, Ethernet, Połączenie sieciowe Bluetooth, Ethernet 2...} CsNetworkServerModeEnabled : True CsNumberOfLogicalProcessors : 8 CsNumberOfProcessors : 1 CsProcessors : {Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz} CsOEMStringArray : {$HUA001PL11000, OemString2, OemString3} CsPartOfDomain : True CsPauseAfterReset : -1 CsPCSystemType : Mobile CsPCSystemTypeEx : Mobile CsPowerManagementCapabilities : CsPowerManagementSupported : CsPowerOnPasswordStatus : Unknown CsPowerState : Unknown CsPowerSupplyState : Safe CsPrimaryOwnerContact : CsPrimaryOwnerName : admin CsResetCapability : Other CsResetCount : -1 CsResetLimit : -1 CsRoles : {LM_Workstation, LM_Server, SQLServer, NT} CsStatus : OK CsSupportContactDescription : CsSystemFamily : MagicBook CsSystemSKUNumber : C178 CsSystemType : x64-based PC CsThermalState : Safe CsTotalPhysicalMemory : 8415129600 CsPhyicallyInstalledMemory : 8388608 CsUserName : *** CsWakeUpType : PowerSwitch CsWorkgroup : OsName : Microsoft Windows 10 Enterprise OsType : WINNT OsOperatingSystemSKU : EnterpriseEdition OsVersion : 10.0.19042 OsCSDVersion : OsBuildNumber : 19042 OsHotFixes : {KB5008876, KB5009467, KB4562830, KB4570334...} OsBootDevice : \Device\HarddiskVolume1 OsSystemDevice : \Device\HarddiskVolume3 OsSystemDirectory : C:\Windows\system32 OsSystemDrive : C: OsWindowsDirectory : C:\Windows OsCountryCode : 48 OsCurrentTimeZone : 60 OsLocaleID : 0415 OsLocale : pl-PL OsLocalDateTime : 2/21/2022 8:22:13 AM OsLastBootUpTime : 2/16/2022 3:04:57 PM OsUptime : 4.17:17:15.9007756 OsBuildType : Multiprocessor Free OsCodeSet : 1250 OsDataExecutionPreventionAvailable : True OsDataExecutionPrevention32BitApplications : True OsDataExecutionPreventionDrivers : True OsDataExecutionPreventionSupportPolicy : OptIn OsDebug : False OsDistributed : False OsEncryptionLevel : 256 OsForegroundApplicationBoost : Maximum OsTotalVisibleMemorySize : 8217900 OsFreePhysicalMemory : 1435100 OsTotalVirtualMemorySize : 17655084 OsFreeVirtualMemory : 7187068 OsInUseVirtualMemory : 10468016 OsTotalSwapSpaceSize : OsSizeStoredInPagingFiles : 9437184 OsFreeSpaceInPagingFiles : 8547776 OsPagingFiles : {C:\pagefile.sys} OsHardwareAbstractionLayer : 10.0.19041.1151 OsInstallDate : 10/6/2021 2:19:18 PM OsManufacturer : Microsoft Corporation OsMaxNumberOfProcesses : 4294967295 OsMaxProcessMemorySize : 137438953344 OsMuiLanguages : {pl-PL, en-US} OsNumberOfLicensedUsers : 0 OsNumberOfProcesses : 269 OsNumberOfUsers : 9 OsOrganization : OsArchitecture : 64-bit OsLanguage : pl-PL OsProductSuites : {TerminalServicesSingleSession} OsOtherTypeDescription : OsPAEEnabled : OsPortableOperatingSystem : False OsPrimary : True OsProductType : WorkStation OsRegisteredUser : admin OsSerialNumber : 00329-10438-14520-AA067 OsServicePackMajorVersion : 0 OsServicePackMinorVersion : 0 OsStatus : OK OsSuites : {TerminalServices, TerminalServicesSingleSession} OsServerLevel : KeyboardLayout : en-US TimeZone : (UTC+01:00) Sarajevo, Skopje, Warsaw, Zagreb LogonServer : \** PowerPlatformRole : Mobile HyperVisorPresent : True HyperVRequirementDataExecutionPreventionAvailable : HyperVRequirementSecondLevelAddressTranslation : HyperVRequirementVirtualizationFirmwareEnabled : HyperVRequirementVMMonitorModeExtensions : DeviceGuardSmartStatus : Off DeviceGuardRequiredSecurityProperties : {0} DeviceGuardAvailableSecurityProperties : {BaseVirtualizationSupport, DMAProtection, SecureMemoryOverwrite, 5...} DeviceGuardSecurityServicesConfigured : {0} DeviceGuardSecurityServicesRunning : {0} DeviceGuardCodeIntegrityPolicyEnforcementStatus : DeviceGuardUserModeCodeIntegrityPolicyEnforcementStatus :

`

Defender

`

AMEngineVersion : 1.1.18900.3 AMProductVersion : 4.18.2201.10 AMRunningMode : Normal AMServiceEnabled : True AMServiceVersion : 4.18.2201.10 AntispywareEnabled : True AntispywareSignatureAge : 0 AntispywareSignatureLastUpdated : 2/20/2022 10:02:36 PM AntispywareSignatureVersion : 1.359.592.0 AntivirusEnabled : True AntivirusSignatureAge : 0 AntivirusSignatureLastUpdated : 2/20/2022 10:02:36 PM AntivirusSignatureVersion : 1.359.592.0 BehaviorMonitorEnabled : True ComputerID : E7D3A3E6-E311-48D8-9A16-2EA897B6CBCA ComputerState : 0 DeviceControlDefaultEnforcement : N/A DeviceControlPoliciesLastUpdated : 2/18/2022 10:40:06 AM DeviceControlState : N/A FullScanAge : 4294967295 FullScanEndTime : FullScanStartTime : IoavProtectionEnabled : True IsTamperProtected : False IsVirtualMachine : False LastFullScanSource : 0 LastQuickScanSource : 2 NISEnabled : True NISEngineVersion : 1.1.18900.3 NISSignatureAge : 0 NISSignatureLastUpdated : 2/20/2022 10:02:36 PM NISSignatureVersion : 1.359.592.0 OnAccessProtectionEnabled : True QuickScanAge : 4 QuickScanEndTime : 2/17/2022 9:08:21 AM QuickScanStartTime : 2/17/2022 7:59:18 AM RealTimeProtectionEnabled : True RealTimeScanDirection : 0 TamperProtectionSource : UI TDTMode : cm TDTStatus : Enabled TDTTelemetry : Disabled PSComputerName :

`

history of threats detected

CategoryID : 27 DidThreatExecute : False IsActive : True Resources : {file:_C:\Users*****\Desktop\logitot\meshagent32-BTC(2).exe} RollupStatus : 1 SchemaVersion : 1.0.0.0 SeverityID : 1 ThreatID : 250070 ThreatName : Program:Win32/Uwamson.A!ml TypeID : 0 PSComputerName :

CategoryID : 8 DidThreatExecute : False IsActive : False Resources : {file:_C:\Users**\Desktop\meshagent32-BTC.exe, file:_C:\Users**\Desktop\logitot\meshagent32-BTC.exe} RollupStatus : 1 SchemaVersion : 1.0.0.0 SeverityID : 5 ThreatID : 2147780193 ThreatName : Trojan:Win32/Sabsik.TE.A!ml TypeID : 0 PSComputerName :

btcsupport commented 2 years ago

I was able to reproduce the problem on Windows 10 evaluation. You can download windows 10 evaluation from microsoft website.

https://www.microsoft.com/en-us/evalcenter/evaluate-windows-10-enterprise

https://btcszczecin-my.sharepoint.com/:i:/g/personal/tbaut_btc_com_pl/ETk845KvTm9GkCWIIkPKMCoBE8TjBjELJ0Jglfs5k1jhPg?e=h1IW9L

btcsupport commented 2 years ago

Maybe the problem is related to the mesh server configuration?

config.json `{ "settings": { "cert": "172.16.115.145", "_minify": true, "noagentupdate": true, "fakeupdate": false, "forceupdate": false, "webrtc": true, "BrowserPing": 60, "BrowserPong": 60, "AgentPing": 60, "AgentPong": 60 }, "domains": {

} }`

btcsupport commented 2 years ago

A similar bug was reported in #551

krayon007 commented 2 years ago

I'm downloading your VM now. That should help a lot to try to track this down, as I haven't been able to reproduce this locally.

krayon007 commented 2 years ago

I downloaded the VM, but it doesn't boot. Is that a Hyper-V hard disc image?

btcsupport commented 2 years ago

Yes these are hyper-v images Try creating a new machine using my image. New > Virtual Machine > Next > Next > Generation 1 > Next (Startup memory: 4096 or more) > Next (select connector e.g. Default switch) > Use an existing virtual disk > Finish

krayon007 commented 2 years ago

Oh ok, I'll try it again. I had selected gen 2.

btcsupport commented 2 years ago

OS: Windows 10 Enterprise (20H2) Meshagent Ver: 0.2.1.3

Case 4 I signed up for the service available at meshcentral.com. I downloaded and installed the agent for windows. While trying to connect, an error appeared in the event log.

Nazwa aplikacji powodującej błąd: MeshAgent.exe, wersja: 0.2.1.3, sygnatura czasowa: 0x612d4568 Nazwa modułu powodującego błąd: unknown, wersja: 0.0.0.0, sygnatura czasowa: 0x00000000 Kod wyjątku: 0xc0000005 Przesunięcie błędu: 0x0143f910 Identyfikator procesu powodującego błąd: 0x2930 Godzina uruchomienia aplikacji powodującej błąd: 0x01d828b403e66799 Ścieżka aplikacji powodującej błąd: C:\Program Files (x86)\Mesh Agent\MeshAgent.exe Ścieżka modułu powodującego błąd: unknown Identyfikator raportu: d3703e77-62fb-4846-bc1d-b91c5087fa74 Pełna nazwa pakietu powodującego błąd: Identyfikator aplikacji względem pakietu powodującego błąd:

https://btcszczecin-my.sharepoint.com/:i:/g/personal/tbaut_btc_com_pl/Ef01Ms2QN7hDukkvP5827V8BRTGz7hoJOAStT-RSyqoBOw?e=v9UYnV

btcsupport commented 2 years ago

I repeated case 3. The only change is to use your server available at meshcentral.com OS: Windows 10 Pro (20H2) Meshagent Ver: 0.2.1.3

The connection was stable for the first 15-20 minutes then dropped. Each subsequent connection was disconnected at random intervals. 

Nazwa` aplikacji powodującej błąd: MeshAgent.exe, wersja: 0.2.1.3, sygnatura czasowa: 0x612d4568
Nazwa modułu powodującego błąd: unknown, wersja: 0.0.0.0, sygnatura czasowa: 0x00000000
Kod wyjątku: 0xc0000005
Przesunięcie błędu: 0x00b3fabc
Identyfikator procesu powodującego błąd: 0x2fc0
Godzina uruchomienia aplikacji powodującej błąd: 0x01d82950fe975523
Ścieżka aplikacji powodującej błąd: C:\Program Files (x86)\Mesh Agent\MeshAgent.exe
Ścieżka modułu powodującego błąd: unknown
Identyfikator raportu: af218014-f474-4caa-b357-44edb7ccfdff
Pełna nazwa pakietu powodującego błąd: 
Identyfikator aplikacji względem pakietu powodującego błąd

Details

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Application Error" /> 
  <EventID Qualifiers="0">1000</EventID> 
  <Version>0</Version> 
  <Level>2</Level> 
  <Task>100</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2022-02-24T07:40:59.1929367Z" /> 
  <EventRecordID>30354</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="0" ThreadID="0" /> 
  <Channel>Application</Channel> 
  <Computer>DESKTOP-JVOBAH2</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>MeshAgent.exe</Data> 
  <Data>0.2.1.3</Data> 
  <Data>612d4568</Data> 
  <Data>unknown</Data> 
  <Data>0.0.0.0</Data> 
  <Data>00000000</Data> 
  <Data>c0000005</Data> 
  <Data>00b3fabc</Data> 
  <Data>2fc0</Data> 
  <Data>01d82950fe975523</Data> 
  <Data>C:\Program Files (x86)\Mesh Agent\MeshAgent.exe</Data> 
  <Data>unknown</Data> 
  <Data>af218014-f474-4caa-b357-44edb7ccfdff</Data> 
  <Data /> 
  <Data /> 
  </EventData>
  </Event>

Screen

https://btcszczecin-my.sharepoint.com/:i:/g/personal/tbaut_btc_com_pl/EV3Y9xjQ1f1BsZbOcYP1g0QBG8-pwJcK2QL6zDHx_KlDiA?e=f5fI9Y

https://btcszczecin-my.sharepoint.com/:i:/g/personal/tbaut_btc_com_pl/EaO2HS4IrshHl5UwkQCSoroBSOC5Qd-o9v7ikUQBx03cvg?e=f5lNJ4

Dumps

https://btcszczecin-my.sharepoint.com/:u:/g/personal/tbaut_btc_com_pl/ES8L6-YpbOxClmRW6GfbIRYBJyubyt-I689xEJ3zEvYiXA?e=52jmk0

JSuenram commented 2 years ago

Same here with latest Agent.... except it crashed directly on start.... rolled back to older Version before new Color-Options where added...

krayon007 commented 2 years ago

I got the Hyper-V image running, but I still couldn't reproduce any issues with the agent... I'm going to try an older agent, and test it with the server at meshcentral.com

krayon007 commented 2 years ago

Same here with latest Agent.... except it crashed directly on start.... rolled back to older Version before new Color-Options where added...

Do you have any crash logs or dumps from this crash?

JSuenram commented 2 years ago

Same here with latest Agent.... except it crashed directly on start.... rolled back to older Version before new Color-Options where added...

Do you have any crash logs or dumps from this crash?

Eventlog gives: (sorry in German) Fehlerbucket 1905273242361910513, Typ 5 Ereignisname: BEX64 Antwort: Nicht verfügbar CAB-Datei-ID: 0

Problemsignatur: P1: itNGO-Agent64-#QuickSupport.exe P2: 0.2.1.3 P3: 62148877 P4: StackHash_e15f P5: 0.0.0.0 P6: 00000000 P7: PCH_19_FROM_ntdll+0x000000000009D8C4 P8: c0000005 P9: 0000000000000008 P10:

Angefügte Dateien: \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER2B12.tmp.dmp \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER2B42.tmp.WERInternalMetadata.xml \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER2B62.tmp.xml \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER2B66.tmp.csv \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER2B87.tmp.txt

Diese Dateien befinden sich möglicherweise hier: \?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_itNGO-Agent64-#Q_82fce6cb28b1ea5ef8b65c419b5297195b8cb35_4a2ea0b3_ef1e0f2e-b07e-4979-91a1-1d5e0651b841

Analysesymbol: Es wird erneut nach einer Lösung gesucht: 0 Berichts-ID: 90ca59b6-31d1-41ef-b292-6f2896f88a79 Berichtstatus: 268435456 Bucket mit Hash: 31eaf66120ca1a8cda70e3eb5b09d4f1 CAB-Datei-Guid: 0

AND

Name der fehlerhaften Anwendung: itNGO-Agent64-#QuickSupport.exe, Version: 0.2.1.3, Zeitstempel: 0x62148877 Name des fehlerhaften Moduls: unknown, Version: 0.0.0.0, Zeitstempel: 0x00000000 Ausnahmecode: 0xc0000005 Fehleroffset: 0x0000000140004380 ID des fehlerhaften Prozesses: 0xc18 Startzeit der fehlerhaften Anwendung: 0x01d8297987baa016 Pfad der fehlerhaften Anwendung: C:\Users\sysop\Downloads\itNGO-Agent64-#QuickSupport.exe Pfad des fehlerhaften Moduls: unknown Berichtskennung: 90ca59b6-31d1-41ef-b292-6f2896f88a79 Vollständiger Name des fehlerhaften Pakets: Anwendungs-ID, die relativ zum fehlerhaften Paket ist:

btcsupport commented 2 years ago

I got the Hyper-V image running, but I still couldn't reproduce any issues with the agent... I'm going to try an older agent, and test it with the server at meshcentral.com

To reproduce the error, the agent must connect to the server. Was this condition met during the test?

mdshoaibumer commented 2 years ago

The issue is coming i have the same meshagent version and its crashing again and again Dear @Ylianst @krayon007 please help

btcsupport commented 2 years ago

@mdshoaibumer on which operating systems are you having problems ? The problem is probably caused by the Microsoft Defender trickery from Windows 10.

mdshoaibumer commented 2 years ago

The problem is here MicrosoftTeams-image (1)

mdshoaibumer commented 2 years ago

Dear @Ylianst , @krayon007 , Please update on this .

dinger1986 commented 12 months ago

@si458 maybe caused by AV, needs exclusions maybe added to docs but can close