Modifying TrustedHosts

[janegilring]

Scenario: An Azure Function connected to a VNet have access to on-premises Windows servers it needs to remote into.

To accomplish this, we would need to add the names/IP addresses of the target machines to the TrustedHosts property in the WSMan configuration.

I tried this from an Azure Function: Set-Item WSMan:\localhost\Client\TrustedHosts -Value *

Which resulted in this error: ERROR: Set-Item : Service WinRM was not found on computer '.'. At D:\home\site\wwwroot\HttpTrigger2\run.ps1:16 char:1

Is WinRM-based remoting not supported from Azure Functions? What is the recommended approach in this scenario? Leverage Azure Automation instead?

[JustinGrote]

Remoting doesn't work at all in functions due to the sandboxing of the app service runtime it runs on. Since you can publish fu cations into a container now, it might be possible that way however, but it won't work on the consumption plan in any way I've tried.

On Wed, Oct 30, 2019, 10:42 PM Jan Egil Ring notifications@github.com wrote:

Scenario: An Azure Function connected to a VNet have access to on-premises Windows servers it needs to remote into.

To accomplish this, we would need to add the names/IP addresses of the target machines to the TrustedHosts property in the WSMan configuration.

I tried this from an Azure Function: Set-Item WSMan:\localhost\Client\TrustedHosts -Value *

Which resulted in this error: ERROR: Set-Item : Service WinRM was not found on computer '.'. At D:\home\site\wwwroot\HttpTrigger2\run.ps1:16 char:1

Is WinRM-based remoting not supported from Azure Functions? What is the recommended approach in this scenario? Leverage Azure Automation instead?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-functions-powershell-worker/issues/359?email_source=notifications&email_token=ADUNKUUBR27EYGJPLZGFWA3QRJV45A5CNFSM4JHEVFZKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HVVA5LQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADUNKUX5MKK5CMCR3SF7RILQRJV45ANCNFSM4JHEVFZA .

[mhoeger]

@alexkarcher-msft - mind weighing in?

[anirudhgarg]

I can confirm that this will not work for Consumption plan. Have you tried using the Premium plan. Have a look at this doc article: https://docs.microsoft.com/en-us/azure/azure-functions/functions-hybrid-powershell and this video: https://www.youtube.com/watch?v=jwVyFgUeRaA

[alexkarcher-msft]

I believe if Win-RM is not allowed in the Consumption plan it will not be allowed in any plans due to the sandbox restrictions. There is a pretty decent list of what is and is not supported in the sandbox here: https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox

If WinRM remoting is not supported it would be worth adding to that doc.

[JustinGrote]

@alexkarcher-msft I think what he means is not using Premium directly, but rather having the premium Azure App Service host in turn reach out to an Azure VM or onprem system via the hybrid powershell connector and do the WinRM work there. It's ugly but it's a way.

Alternatively you could just reach out to an Azure VM directly from the function via other means, but doing remoting operations directly in the app service host won't work because of sandboxing mentioned.

However, since it is Powershell Core, you can use Powershell Remoting over SSH, but you have to install Win32_OpenSSH on the Windows Servers in question. In that scenario you could do remoting within the consumption plan (and we have tested it successfully).

@janegilring can you accomplish what you need with the CIM powershell cmdlets instead of powershell remoting? Those work fine.

[eamonoreilly]

I'll investigate this using vNets and update the thread here. For now, you can also use hybrid connections to do remoting to VMs on premises (or in Azure or other clouds) from the function app. https://docs.microsoft.com/en-us/azure/azure-functions/functions-hybrid-powershell Thanks, Eamon

[janegilring]

You understood the issue correctly.

having the premium Azure App Service host in turn reach out to an Azure VM or onprem system via the hybrid powershell connector and do the WinRM work there. It's ugly but it's a way.

I agree, it works but it is cumbersome.

However, since it is Powershell Core, you can use Powershell Remoting over SSH, but you have to install Win32_OpenSSH on the Windows Servers in question. In that scenario you could do remoting within the consumption plan (and we have tested it successfully).

That is also an alternative. Although, having to use a jumpbox via an SSH connection also feels cumbersome.

The scenario I am aiming for is to promote the idea of start using Azure Functions for hybrid automation scenarios, then it should work like people are used to in Azure Automation where regular PS Remoting just works.

This will be especially important when PowerShell 7.0 is released (Feb 2020), where people will be advised to start using that in new automation scenarios. Then they can not use Azure Automation, as it only supports Windows PowerShell. However, when using Azure Functions, WinRM-based PowerShell Remoting does not work as they are used to.

[eamonoreilly]

You can use VNets in an Azure Function premium (or app service plan) and do remoting to an on-premises or Azure VM connected to the vNet. If you follow the description on https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-vnet to set up a VM and a function app connected with the VM subnet. You can follow the steps on https://docs.microsoft.com/en-us/azure/azure-functions/functions-hybrid-powershell to set up remoting on the VM (or on-premises server) and connect into the machine without having to install the hybrid connection agent since traffic will go over the vNet. This will use the https transport since the function app is not connected to the domain.

You can just run the script on https://docs.microsoft.com/en-us/azure/azure-functions/functions-hybrid-powershell#configure-an-on-premises-server-for-powershell-remoting to enable remoting and then follow the steps on https://docs.microsoft.com/en-us/azure/azure-functions/functions-hybrid-powershell#create-an-app-setting-for-the-password-of-an-administrator-account to remote to this machine from the Function app. Thanks, Eamon

[janegilring]

Thanks Eamon, I tried this out and it did work as intended.

Setting up HTTPS listeners on all target machines would be cumbersome and even difficult in some environments. I have a customer with ~1200 servers which is target by some scheduled runbooks, ranging from Windows Server 2008 R2 to Windows Server 2019. Even though it is theoretically possible to enable HTTPS on all those servers it will provide some additional maintenance overhead I suppose since they probably don`t want to use self-signed certificates and hence need to make sure certificates are renewed and possibly listeners updated if the thumbprint changes (I have not tested that scenario). However, having HTTPS listeners on a couple of central management servers in order to use those as jump hosts could be more viable. Even though it would possibly be needed to either use CredSSP or Kerberos delegation to avoid double hop scenarios (or maybe not, since we do not use Kerberos from the worker to the management server?).

Although using jump hosts might seem more cumbersome than Azure Automation where regular HTTP-based remoting just works, this might be the best alternative when it comes to Functions in my opinion.

[eamonoreilly]

Thanks Jan for trying this solution out and for the feedback. I agree that generally it would make sense to have a couple of jump boxes configured in the environment and then use those to access the rest of the environment. You can see this in practice on https://docs.microsoft.com/en-us/azure/azure-functions/functions-hybrid-powershell#managing-other-systems-on-premises where we remote to a jump box machine and then manage the rest of the environment from there.

The customer could set up the session configuration to run under AD credentials using the -RunAsCredential so it can connect using those credentials to other machines. Thanks again and please let me know if there is anything else we can do within functions to make automation scenarios easier for customers. Eamon

[thierrybrunet2c]

PSRemoting from Powershell Function on consumption plan to Windows VM [over WSMan] works great for us. Have to be careful in installing Powershell 7 rc2 everywhere and using -ConfigurationName "PowerShell.7.0.0-rc.2. We use this extensively on Prod and several lower environments. However, remoting to Linux VM [over SSH] does not work (we get ssh.exe not found error message. Under kudu, ssh.exe runs fine, but not when the function executes. We tried to install ssh.exe on the function app root but that does not help. Our work around at this moment is to implement some form of nested invoke-command {...} for PSremoting like so : AzPwshFunction --> Windows VM [WSMan] --> Linux VM [ssh]. The Windows VM acts as a PSremoting [SSH] proxy for the Az Powershell Function in order to relay invoke commands for the Linux VM. Make sure the trustedhosts file on the Windows VM user account (under .ssh folder) is updated otherwise the inner nested invoke-command [SSH] to Linux will block the script execution without showing any error message (very bad!!!). Tried all available options to avoid checking the ECDSA key fingerprint but to no avail. Would be nice to have this -noxxxCheck option.

Are we missing something for making ssh.exe available for the Linux psremoting from function execution to succeed ? We are using function runtime V2 as runtime V3 breaks. May be this has been fixed now but have not had a chance to try-out yet.

I have edited my original text to clarify WSMan or SSH when doing PSRemoting.

Btw, I confirm that our nested remoting invoke works flawlessly [Consumption plan] for carrying some critical automation tasks (distributed store eventual constancy convergence scheduled verification ). Indeed will update our code when the ssh.exe issue is fixed , run time 3.0 works with Powershell Functions and fan-out durable function is GA. In the meanwhile, we had to make our own orchestrator and implement time-out recovery.

The Windows VM that performs the SSH proxy for the Linux box is not really a jump box as it already exists in each of our 7 environments. The Windows VM does not store any data or code, just need to have the Pwsh 7 registered for remoting and the trustedhosts file updated.

[janegilring]

@eamonoreilly On the flip side, having jump boxes kind of invalidates the concept of "serverless" though :) Azure File Sync will soon get support for adding a storage account/file share to an Active Directory in order to enable a hybrid scenario where NTFS authentication will just work against an Azure File Share. If it would be possible to add an Azure Function worker to an Active Directory domain available via a VNet in a similar fashion, I think much of the need for jump boxes would go away. Not sure whether this is possible at, just thinking out loud...maybe you could check with the Azure Storage team how they did it.

[JustinGrote]

PSRemoting from Powershell Function on consumption plan to Windows VM

Important to emphasize this is using Powershell over SSH, not over WSMan.

[thierrybrunet2c]

PSRemoting from Powershell Function on consumption plan to Windows VM

Important to emphasize this is using Powershell over SSH, not over WSMan.

I have edited my original text accordingly. Thanks for requesting this essential clarification. Btw, I confirm that our nested remoting invoke works flawlessly [Consumption plan]. Indeed will update our code when the ssh.exe issue is fixed and run time 3.0 works with Powershell Functions

[thierrybrunet2c]

@eamonoreilly On the flip side, having jump boxes kind of invalidates the concept of "serverless" though :) Azure File Sync will soon get support for adding a storage account/file share to an Active Directory in order to enable a hybrid scenario where NTFS authentication will just work against an Azure File Share. If it would be possible to add an Azure Function worker to an Active Directory domain available via a VNet in a similar fashion, I think much of the need for jump boxes would go away. Not sure whether this is possible at, just thinking out loud...maybe you could check with the Azure Storage team how they did it.

In our workaround for the ssh.exe fail issue when attempting ssh PSRemote to a Linux VM from an AzPwshFunction, the Windows VM that performs the SSH proxy for the Linux box is not really a jump box as it already exists in each of our 7 environments. The Windows VM does not store any data or code, just need to have the Pwsh 7 registered for remoting and the trustedhosts file updated.

[janegilring]

I also ran into the error you mentioned when trying to remote into a Linux VM from an Azure Function: The term 'ssh.exe' is not recognized as the name of a cmdlet, function, script file, or operable program

So this means SSH-based remoting is not supported in Azure Functions due to ssh.exe not being available - is supporting this on the roadmap? Or will it be blocked for security reasons?

Also, what is the reason for Windows being the only supported Operating System for the PowerShell runtime? Will this be supported at some point? Windows is the only supported Operating System for your selection of runtime stack.

[weq]

It's been atleast 2 years now. Any update when pwsh is going to be treated like a first class citizen in Azure Functions...?

[JustinGrote]

As of today, you can now run powershell functions on linux, you can also set up SSH as part of your cold-start in Windows to make ssh remoting work, and very soon (hopefully 7.3) the remoting functionality will be exposed to allow for custom remoting transports.

Being a user, I feel it is as well treated as any of the other isolated runtimes (javascript/go/etc.), but feature development has definitely slowed.

[weq]

PSremoting to Linux just works as intended when using it anywhere else than an Azure function (making the assumption that the subsystem is configured ofc). It is just super annoying that simple stuff still requires us to use modules like posh-ssh instead of the native methods. Even more so when we’ve gotten tools like scp in basically any modern os now.

It just falls between 2 chairs instead of a proper top down approach imo. The hard coding of ssh.exe and why that made sense should be documented. If we can’t use the full native features I’d hope it be at least documented. This could be just be discovered for all with a simple automatic test. What works and what doesn’t.

On Thu, 3 Mar 2022 at 17:24 Justin Grote @.***> wrote:

As of today, you can now run powershell functions on linux, you can also set up SSH as part of your cold-start in Windows to make ssh remoting work, and very soon (hopefully 7.3) the remoting functionality will be exposed to allow for custom remoting transports.

Being a user, I feel it is as well treated as any of the other isolated runtimes (javascript/go/etc.), but feature development has definitely slowed.

[JustinGrote]

The hard coding of ssh.exe and why that made sense should be documented.

This aspect isn't Azure Functions responsibility, this is Powershell SSH remoting subsystem itself. There was a big fight about it about a year back and the PS team decided not to remove the hardcoding and instead open up the transport. That work is coming along here: https://github.com/PowerShell/PowerShell/pull/16923

[weq]

So you are saying that psremoting from something like Ubuntu has ssh.exe bundled and that is still what is being used? Or is it relying on just ssh native executable on linux? I just find it oddly weird that what I'd consider to be a base feature is not available in an azure function but available everywhere else I use the same version of PS. Cause if so then I'd sorta argue it could still be the functions responsibility, to have all prerequisites in the sandbox to use all the base features. Or what am I missing here…?

On Thu, 3 Mar 2022 at 22:27 Justin Grote @.***> wrote:

The hard coding of ssh.exe and why that made sense should be documented.

This aspect isn't Azure Functions responsibility, this is Powershell SSH remoting subsystem itself. There was a big fight about it about a year back and the PS team decided not to remove the hardcoding and instead open up the transport. That work is coming along here: PowerShell/PowerShell#16923 https://github.com/PowerShell/PowerShell/pull/16923

[JustinGrote]

SSH remoting in Powershell relied on 'ssh.exe' being in the path, that is hardcoded into the code. A poor design decision IMHO but it is what it is. https://github.com/PowerShell/PowerShell/blob/0302b1f48e250c9387f976b78c9f4b6434ac9420/src/System.Management.Automation/engine/remoting/common/RunspaceConnectionInfo.cs#L2180-L2184

[JustinGrote]

So if you use the linux functions worker in consumption, it has SSH preinstalled. The windows one doesn't but you can install it yourself. IMHO function workers should be as lean as possible and you bring your own dependencies.

[weq]

I can now see that my remarks are totally unjust cause it is my own fault. I trusted my TF to deploy my function app to have correctly deployed a linux environment which it clearly has not. 😳