d365collaborative / d365fo.tools

Tools used for Dynamics 365 Finance and Operations
MIT License
246 stars 101 forks source link

D365FO Developer VM Blocking Import/Install of D365fo.tools #693

Closed rdakyle closed 1 year ago

rdakyle commented 1 year ago

This just started happening in the last 24 hours. The last successful run of using the tools I had was at 5am EST. Nov 19th 2022. I'm experiencing this on two development D365 FO machines in LCS:

  1. When installing d365fo.tools as administrator, PowerShell quickly processes the request. No messages.
  2. Execute simple command such as Start-D365Environment, returns back error that the term is not recognized.

image

If I open PowerShell ISE as admin and select Modules: d365fo.tools, go to any command and click Show Details, I get a more detailed error that antivirus (windows defender likely) is blocking importing of the module.

image

Has anyone had this issue specifically on a Microsoft hosted D365 development machine before? I'm assuming Defender took an update in the last 24 hours and it's blocking the tools now. If anyone else is able to check their dev machine, please let me know.

Thank you!

RichS-OSG commented 1 year ago

I'm having the same issue and found your post when looking for a solution.

--Rich

Splaxi commented 1 year ago

This is new to us. It could be that we don't sign the module in our release process and that the built-in defender now views this as a risk. I need some help from the community to dive into this - as I don't have enough insights at the moment.

A quick search shows that people have seen issues previously with the defender definition needing to be updated and it resolves any issues after that: https://evotec.xyz/import-module-this-script-contains-malicious-content-and-has-been-blocked-by-your-antivirus-software/

gertvdh commented 1 year ago

same problems here, even after updating the virus definitions to latest

onikolaiev commented 1 year ago

The same issue. I have tested at Win2016/2019/2022, Win 10/11. For a now I have only one workaround - disabling real-time check while d365fo.tools importing and enabling back after importing is done. Exception the module folder from Defender doesnot take any results.

Set-MpPreference -DisableRealtimeMonitoring $true; Import-Module d365fo.tools -Force; Set-MpPreference -DisableRealtimeMonitoring $false;

Splaxi commented 1 year ago

This could be the real issue: https://www.microsoft.com/en-us/wdsi/threats/malware-encyclopedia-description?name=Trojan:PowerShell/MpTamperPShell.HA

MS might have raised the bar significantly - and it is fairly new...

ITRasmus commented 1 year ago

Hi After removing , function Add-D365WindowsDefenderRules in the command.ps1 file, then the module can load again

Splaxi commented 1 year ago

Makes sense!

Splaxi commented 1 year ago

So - we have a work around, for the module at the current state.

But - it leaves us with a design decision: Should we keep the Add-D365WindowsDefenderRules - and take upon us all the work needed to make sure that whenever Microsoft changes anything in their defender definition, we need to catch up?

Or should we "generate" the commands - that the user then can execute by hand? Something like a script, that creates scripts and let the user run it in another session?

Or should we remove it entirely - and simply document it as a gist file, from the wiki page - and then people can copy & paste the gist file and run it locally on their machine?

gertvdh commented 1 year ago

ideally, option 1, but when it happens, it takes time and maybe it's not possible in the end anymore. So option 2 to generate a script maybe the best option in my opinion

RichS-OSG commented 1 year ago

The same issue. I have tested at Win2016/2019/2022, Win 10/11. For a now I have only one workaround - disabling real-time check while d365fo.tools importing and enabling back after importing is done. Exception the module folder from Defender doesnot take any results.

Set-MpPreference -DisableRealtimeMonitoring $true; Import-Module d365fo.tools -Force; Set-MpPreference -DisableRealtimeMonitoring $false;

This also worked for me. Thanks

rdakyle commented 1 year ago

Thanks for the workaround fellas. In my opinion, keeping the tools working without requiring changes to the customers (us) would be best. I say this in selfishness as my organization uses these tools quite a bit in a variety of Azure pipelines, etc... We do not use Add-D365WindowsDefenderRules

Splaxi commented 1 year ago

I'm also leaning towards that the tools should require close to zero knowledge about how to install and run them.

Previously we have considered splitting the module into smaller chunks - but the investment doesn't seem to be worth the effort.

rdakyle commented 1 year ago

That would make sense. My thoughts as well. I'm actually new to the tools and implementation of them myself. My organization had a Microsoft Partner who configured our setup on our behalf and they used these tools. Therefore I've been learning about them myself recently, but not the one who implemented them from scratch.

FH-Inway commented 1 year ago

Just for documentation purposes, if the module import/install cmdlet is executed with the -Verbose parameter, a similar error message to the one from PowerShell ISE above is shown:

C:\windows\system32> import-module d365fo.tools -verbose
VERBOSE: Loading module from path 'C:\Program Files\WindowsPowerShell\Modules\d365fo.tools\0.6.76\d365fo.tools.psd1'.
VERBOSE: Loading 'FormatsToProcess' from path 'C:\Program
Files\WindowsPowerShell\Modules\d365fo.tools\0.6.76\xml\d365fo.tools.Format.ps1xml'.
VERBOSE: Populating RepositorySourceLocation property for module d365fo.tools.
VERBOSE: Loading module from path 'C:\Program Files\WindowsPowerShell\Modules\d365fo.tools\0.6.76\d365fo.tools.psm1'.
Exception calling "InvokeScript" with "4" argument(s): "At line:1 char:1
+
This script contains malicious content and has been blocked by your antivirus software."
At C:\Program Files\WindowsPowerShell\Modules\d365fo.tools\0.6.76\d365fo.tools.psm1:49 char:9
+     else { $ExecutionContext.InvokeCommand.InvokeScript($false, ([scr ...
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : ParseException

Exception calling "InvokeScript" with "4" argument(s): "The term 'Update-ModuleVariables' is not recognized as the
name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was
included, verify that the path is correct and try again."
At C:\Program Files\WindowsPowerShell\Modules\d365fo.tools\0.6.76\d365fo.tools.psm1:49 char:9
+     else { $ExecutionContext.InvokeCommand.InvokeScript($false, ([scr ...
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : CommandNotFoundException

VERBOSE: Exporting function 'Import-ModuleFile'.
FH-Inway commented 1 year ago

Regarding the design decision: Do we have an idea what option 1 (keeping the Add-D365WindowsDefenderRules cmdlet) would entail in this specific case? Is it even possible to exclude a cmdlet that changes the defender rules from the defender check?

I'm guessing that at the end of the day, we will have to separate out at least the parts of Add-D365WindowsDefenderRules that trigger the defender check. Could be a separate file in the repo that does not get included in the module. Instead, when executing the cmdlet, it downloads the file and runs it. This way, the cmdlet would still work as before for the users.

FH-Inway commented 1 year ago

Also note that Azure DevOps pipelines running on Microsoft Hosted Agents can still install d365fo.tools. Seems they have not gotten the Windows Defender update (yet?).

Splaxi commented 1 year ago

There are several tricks we can take into consideration, one is to store the entire list of excludes as a base64 (@ITRasmus suggestion), download the latest and greatest version (saving it locally, showing the command to run it) and several other.

But do we want to support it? If we keep it - we (the entire community), has to be onboard in terms of things might break down the road, and that we end up in the same mess.

Regarding the agents, the agent version is somewhat locked - so nothing new gets included from day to day, there will be an update on the agents, and then they will start to break. When this is going to happen is just a matter of time.

FH-Inway commented 1 year ago

@PaulHeisterkamp @gertvdh , you did some work on the Add-D365WindowsDefenderRules cmdlet, so I assume you (and your colleagues) also use it regularly. Do you have thoughts on if and how the d365fo.tools should support it in the future?

navdeep197 commented 1 year ago

Hi All,

Even on my DevVM I have issues when I click the Show details button on the Commands

image

Is there any workaround to get the tools working? On my VM I cannot add Exclusions to Windows defender as well because of Organization policies.

Regards, Navdeep

loko1275 commented 1 year ago

Hi All,

Even on my DevVM I have issues when I click the Show details button on the Commands

image

Is there any workaround to get the tools working? On my VM I cannot add Exclusions to Windows defender as well because of Organization policies.

Regards, Navdeep

It has been answered here :

Hi After removing , function Add-D365WindowsDefenderRules in the command.ps1 file, then the module can load again

I can confirm it worked for me as well

flauest commented 1 year ago

Hi. Here can I find the command.ps1, so I can edit it? I normally just run the import-module d365fo.tools. Is a new version coming ?

PaulHeisterkamp commented 1 year ago

@PaulHeisterkamp @gertvdh , you did some work on the Add-D365WindowsDefenderRules cmdlet, so I assume you (and your colleagues) also use it regularly. Do you have thoughts on if and how the d365fo.tools should support it in the future?

We use the whole stuff to speed up development VMs. If d365fo.tools does not support this in future, I think we need to reimplement this in a different way.

loko1275 commented 1 year ago

Hi. Here can I find the command.ps1, so I can edit it? I normally just run the import-module d365fo.tools. Is a new version coming ?

It depends on the scope on which you installed it,

CurrentUser scope it's going to be in C:/Users/youruser/Windows Powershell/Modules All users it's going to be in Program Files/Windows Powershell/Modules Else you can look into C:\Windows\System32\WindowsPowerShell

FH-Inway commented 1 year ago

I played around a bit with the contents of commands.ps1. Turns out, there is only one exclusion that triggers Windows Defender, which is this line:

https://github.com/d365collaborative/d365fo.tools/blob/7817028148e8042962f606e4ba03b7daf61d9d8e/d365fo.tools/functions/add-d365windowsdefenderrules.ps1#L99

This is part of the original version by @DarkSmile92 , maybe they can weigh in?

I would consider this a variant of the "keep it" option: Keep the cmdlet in general and just remove the part that triggers Windows Defender.

DarkSmile92 commented 1 year ago

I played around a bit with the contents of commands.ps1. Turns out, there is only one exclusion that triggers Windows Defender, which is this line:

https://github.com/d365collaborative/d365fo.tools/blob/7817028148e8042962f606e4ba03b7daf61d9d8e/d365fo.tools/functions/add-d365windowsdefenderrules.ps1#L99

This is part of the original version by @DarkSmile92 , maybe they can weigh in?

I would consider this a variant of the "keep it" option: Keep the cmdlet in general and just remove the part that triggers Windows Defender.

I don't encounter any issues, but we also did not set up any new env since the mentioned date. So if this line really triggers it, remove it and add the office folder as exception manually - optionally.

Splaxi commented 1 year ago

Ha! I just came to the same conclusion and just wanted to share it with you guys.

I suggest that we remove the line in question and the tools back on track. Then we can find a way to support it.

I'll make the change and include the latest changes from the last few weeks.

FH-Inway commented 1 year ago

Actually, a simple wrapping of the folder in a variable also does not trigger the check:

$folder = "C:\ProgramData\Microsoft\VisualStudio\Packages"
Add-MpPreference -ExclusionPath $folder
Splaxi commented 1 year ago

Epic find! I was planning on base64 encoding it and do some crazy script block support. Might still go that path - to keep us out of the radar of the defender team for a longer period 😉

FH-Inway commented 1 year ago

Was also playing around with the base64 encoding, this works too:

Add-MpPreference -ExclusionPath [System.Text.Encoding]::UTF8.GetString([Convert]::FromBase64String("QzpcUHJvZ3JhbURhdGFcTWljcm9zb2Z0XFZpc3VhbFN0dWRpb1xQYWNrYWdlcw==")) # "C:\ProgramData\Microsoft\VisualStudio\Packages"
FH-Inway commented 1 year ago

A more viable option might be to generate a file with exclusion paths and processes at e.g. C:\Temp\d365fo.tools\WindowsDefenderExclusions.txt and than read and process that file with Add-D365WindowsDefenderRules. This way, power users could also define their own exclusions.

Splaxi commented 1 year ago

We have all the options in the world - but right now I just want to uptake all the changes, get the module back to being easy to work with.

Then we can plan on what to do going forward.

When I played around, I learned that even commenting the single line wasn't enough. So the defender was scanning for the line alone, regardless of being an active line of code or not

FH-Inway commented 1 year ago

Yeah, it's weird that they even scan comments, but then allow such a simple workaround. I guess they just have a regex that includes Add-MpPreference and ProgramData 😄

Splaxi commented 1 year ago

0.6.77 should solve the issue

FH-Inway commented 1 year ago

It does indeed :) image