search

darkoperator / Posh-SSH

PowerShell Module for automating tasks on remote systems using SSH
BSD 3-Clause "New" or "Revised" License
975 stars 226 forks source link

Could not load file or assembly 'Newtonsoft.Json, Version=13.0.0.0 #469

Open DisasteR opened 2 years ago

DisasteR commented 2 years ago

Hi,

I am trying to load Posh-SSH from a vmware/powerclicore docker image but cannot load it.

PS /root> Install-Module -Name Posh-SSH -RequiredVersion 3.0.4 -Verbose
VERBOSE: Using the provider 'PowerShellGet' for searching packages.
VERBOSE: The -Repository parameter was not specified.  PowerShellGet will use all of the registered repositories.
VERBOSE: Getting the provider object for the PackageManagement Provider 'NuGet'.
VERBOSE: The specified Location is 'https://www.powershellgallery.com/api/v2' and PackageManagementProvider is 'NuGet'.
VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Posh-SSH'' for ''.
VERBOSE: Total package yield:'1' for the specified package 'Posh-SSH'.
VERBOSE: Performing the operation "Install-Module" on target "Version '3.0.4' of module 'Posh-SSH'".
VERBOSE: The installation scope is specified to be 'CurrentUser'.
VERBOSE: The specified module will be installed in '/root/.local/share/powershell/Modules'.
VERBOSE: The specified Location is 'NuGet' and PackageManagementProvider is 'NuGet'.
VERBOSE: Downloading module 'Posh-SSH' with version '3.0.4' from the repository 'https://www.powershellgallery.com/api/v2'.
VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Posh-SSH'' for ''.
VERBOSE: InstallPackage' - name='Posh-SSH', version='3.0.4',destination='/tmp/1212229673'
VERBOSE: DownloadPackage' - name='Posh-SSH', version='3.0.4',destination='/tmp/1212229673/Posh-SSH.3.0.4/Posh-SSH.3.0.4.nupkg', uri='https://www.powershellgallery.com/api/v2/package/Posh-SSH/3.0.4'
VERBOSE: Downloading 'https://www.powershellgallery.com/api/v2/package/Posh-SSH/3.0.4'.
VERBOSE: Completed downloading 'https://www.powershellgallery.com/api/v2/package/Posh-SSH/3.0.4'.
VERBOSE: Completed downloading 'Posh-SSH'.
VERBOSE: InstallPackageLocal' - name='Posh-SSH', version='3.0.4',destination='/tmp/1212229673'
VERBOSE: Validating the 'Posh-SSH' module contents under '/tmp/1212229673/Posh-SSH.3.0.4' path.
VERBOSE: Test-ModuleManifest successfully validated the module manifest file '/tmp/1212229673/Posh-SSH.3.0.4'.
VERBOSE: Module 'Posh-SSH' was installed successfully to path '/root/.local/share/powershell/Modules/Posh-SSH/3.0.4'.

PS /root> Import-Module -Name Posh-SSH -Verbose
VERBOSE: Loading module from path '/root/.local/share/powershell/Modules/Posh-SSH/3.0.4/Posh-SSH.psd1'.
VERBOSE: Loading 'Assembly' from path '/root/.local/share/powershell/Modules/Posh-SSH/3.0.4/Assembly/Newtonsoft.Json.dll'.
VERBOSE: Loading 'Executable' from path '/root/.local/share/powershell/Modules/Posh-SSH/3.0.4/Assembly/Newtonsoft.Json.dll'.
VERBOSE: Loading 'Assembly' from path '/root/.local/share/powershell/Modules/Posh-SSH/3.0.4/Assembly/Renci.SshNet.dll'.
VERBOSE: Loading 'Executable' from path '/root/.local/share/powershell/Modules/Posh-SSH/3.0.4/Assembly/Renci.SshNet.dll'.
VERBOSE: Loading 'Assembly' from path '/root/.local/share/powershell/Modules/Posh-SSH/3.0.4/Assembly/SshNet.Security.Cryptography.dll'.
VERBOSE: Loading 'Executable' from path '/root/.local/share/powershell/Modules/Posh-SSH/3.0.4/Assembly/SshNet.Security.Cryptography.dll'.
VERBOSE: Loading 'FormatsToProcess' from path '/root/.local/share/powershell/Modules/Posh-SSH/3.0.4/Format/SSHSession.Format.ps1xml'.
VERBOSE: Loading 'FormatsToProcess' from path '/root/.local/share/powershell/Modules/Posh-SSH/3.0.4/Format/SFTPSession.Format.ps1xml'.
VERBOSE: Loading 'FormatsToProcess' from path '/root/.local/share/powershell/Modules/Posh-SSH/3.0.4/Format/Renci.SshNet.SshCommand.Format.ps1xml'.
VERBOSE: Loading 'FormatsToProcess' from path '/root/.local/share/powershell/Modules/Posh-SSH/3.0.4/Format/Renci.SshNet.Sftp.SftpFile.Format.ps1xml'.
Import-Module: Could not load file or assembly 'Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'.

Here is the versions of powershell

Name                           Value
----                           -----
PSVersion                      7.1.5
PSEdition                      Core
GitCommitId                    7.1.5
OS                             Linux 5.13.0-39-generic #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

I hope you have a solution.

regards,

darkoperator commented 2 years ago

Was able to replicate on Linux, for some reason on Windows and Mac OS it is not a problem. Looking in to it .

DisasteR commented 2 years ago

FYI it work with version 3.0.0 and dont work with 2.3.x

darkoperator commented 2 years ago

Problem only happens on Linux and is related to references, even when I specify the latest file VS puts in it properties for the reference am older verdion. Still looking how to fix it. On Windows and Mac OS it works

DisasteR commented 2 years ago

Hope you'll find a solution

nthxny commented 2 years ago

Same error occuring for me on Windows in PowerShell version 6 and 7

darkoperator commented 2 years ago

Can you provide a bit more info to the to replicate? Version of Linux, output of $PSversiontable and how the module was installed and version

Sent from my iPhone

On Aug 2, 2022, at 12:28 AM, Anthony @.***> wrote:

 Same error occuring for me on Windows in PowerShell version 6 and 7

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

weq commented 2 years ago

Can confirm that the issue affects Linux Azure Functions as well. Anything higher than 3.0.0 will error out.

darkoperator commented 2 years ago

Can you provide output and specific versions

Sent from my iPhone

On Aug 8, 2022, at 8:23 AM, Raymond Siring @.***> wrote:

 Can confirm that the issue affects Linux Azure Functions as well. Anything higher than 3.0.0 will error out.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

darkoperator commented 2 years ago

I know the version before the last one had the issue and the Azure team contacted me and the latest release addressed it. I have not been able to replicate this with the latest version that is why Im asking about specific information of the environment to be able to replicate

weq commented 2 years ago 
using namespace System.Net
param($Request, $TriggerMetadata)
$hostname = "someHostname"
$resourceGroup = "someResourceGroup"
$storageAccountName = 'someStorageAccountName'
$containerName = 'certificates'
$tempFolder = [System.IO.Path]::GetTempPath()
Uninstall-AzureRm
try {
    $subscription = Get-AzSubscription | Where-Object name -eq 'someSubscriptionName'
    Set-AzContext -SubscriptionObject $subscription
    Import-Module Az.KeyVault, Az.Storage, Az.Accounts, Posh-SSH
    Get-Module
    $tempFolder = [System.IO.Path]::GetTempPath()
    $keyFileContent = Get-AzKeyVaultSecret -vaultname "someVaultName" -secretname "someSecretName" -asplaintext
    $keyFile = Join-Path -Path $tempFolder -ChildPath "key-file.priv"
    Set-Content -Path $keyFile -Value $keyFileContent
    $credential = new-object system.management.automation.pscredential('azurefunctions',(new-object system.security.securestring))
    Get-SCPItem -path "/tmp/someCert.tld.pem" -destination $tempFolder -Credential $credential -KeyFile $keyFile -ComputerName $hostname -AcceptKey -Force -PathType file
    Get-SCPItem -path "/tmp/someCert.tld.key" -destination $tempFolder -Credential $credential -KeyFile $keyFile -ComputerName $hostname -AcceptKey -Force -PathType file
    $privateKey = Get-ChildItem -Path "$tempFolder/someCert.tld.key"
    $pem = Get-ChildItem -Path "$tempFolder/someCert.tld.pem"
    $storageAccountKeys = Get-AzStorageAccountKey -ResourceGroupName $resourceGroup -Name $storageAccountName
    $primaryKey = $storageAccountKeys | Where-Object keyname -EQ 'key1' | Select-Object -ExpandProperty value
    $storageContext = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $primaryKey
    Set-AzStorageBlobContent -Container $containerName -File $privateKey -Blob "someCert.tld.key" -Context $storageContext -Force
    Set-AzStorageBlobContent -Container $containerName -File $pem -Blob "someCert.tld.pem" -Context $storageContext -Force
    Remove-Item -Path $privateKey -Force
    Remove-Item -Path $pem -Force
    Remove-Item -Path $keyFile
    $body = 'OK'
}
catch {
    $body = $_.Exception.Message
}
Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
        StatusCode = [HttpStatusCode]::OK
        Body       = $body
})

This is a stripped down version what I'm using. You can see the modules I utilize etc. Anything else you can think of you need...?

darkoperator commented 2 years ago

Can you pull $PSVersionTable of the PS that Azure uses? Also a Get-Module -ListAvaliable -name Posh-SSH to see the version number of the module. So that I can try to replicate. The version of Posh-SSH should be 3.0.6

weq commented 2 years ago

They utilize 7.0.11 in Azure Functions.

darkoperator commented 2 years ago

and you are sure you are running on 3.0.6 of the module?

image

Just tried a new build of Ubuntu Linux and it is working as it is supposed to.

Tested in Azure Shell and it worked also with the latest 3.06, in fact that version of PowerShell is a very old one that could be th issue since the latest both in Ubuntu and in Cloud shell is 7.2.5 

PS /home/carlos> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.2.5
PSEdition                      Core
GitCommitId                    7.2.5
OS                             Linux 5.4.0-1086-azure #91~18.04.1-Ubuntu SMP Thu Jun 23 20:33:05 UTC 2022
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

PS /home/carlos> uname                           
Linux
PS /home/carlos> uname -a
Linux cc-c7b525dc-9d87b9fb-5w7bz 5.4.0-1086-azure #91~18.04.1-Ubuntu SMP Thu Jun 23 20:33:05 UTC 2022 x86_64 GNU/Linux
PS /home/carlos> Get-Module -ListAvailable -Name posh-ssh

    Directory: /home/carlos/.local/share/powershell/Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Manifest   3.0.6                 Posh-SSH                            Desk      {Get-SCPItem, Get-SFTPItem, New-SFTPSession, New-SSHSession…}
jborean93 commented 2 years ago

The problem here is that Posh-SSH is bundling it's own copy of Newtonsoft.json at version 13.x.y. PowerShell itself also ships Newtonsoft.json which is at 13.x.y for pwsh 7.2 but pwsh 7.1 and 7.0 ship Newtonsoft.json 12.x.y. When using this module on older PowerShell versions you get this conflict because PowerShell has already loaded the older json dll causing a conflict when Posh-SSH goes to load it's own copy of 13.x.y.

Is there a reason why Posh-SSH is bundling it's own copy of Newtonsoft (and other dlls). I see the following with the 3.0.6 install

PS /home/jborean> ls -al /home/jborean/.local/share/powershell/Modules/Posh-SSH/3.0.6/Assembly/
total 3404
drwxrwxr-x. 1 jborean jborean     344 Aug  9 05:04 .
drwxrwxr-x. 1 jborean jborean     150 Aug  9 05:04 ..
-rw-rw-r--. 1 jborean jborean  701992 Mar 18  2021 Newtonsoft.Json.dll
-rw-rw-r--. 1 jborean jborean   43520 Jun 27 16:06 PoshSSH.dll
-rw-rw-r--. 1 jborean jborean  801792 Jun 23 20:36 Renci.SshNet.dll
-rw-rw-r--. 1 jborean jborean   34816 Oct 23  2017 SshNet.Security.Cryptography.dll
-rw-rw-r--. 1 jborean jborean   20856 Jan 25  2021 System.Buffers.dll
-rw-rw-r--. 1 jborean jborean 1608160 Dec  2  2020 System.Management.Automation.dll
-rw-rw-r--. 1 jborean jborean  141184 Jan 25  2021 System.Memory.dll
-rw-rw-r--. 1 jborean jborean  115856 Jan 25  2021 System.Numerics.Vectors.dll

Because PowerShell ships with Newtonsoft.Json.dll and System.Management.Automation.dll you really should avoid shipping your own to avoid conflicts like this. Adding PrivateAssets="all" to your csproj when you reference these dependencies will mean they are downloaded for development but they are not part of the final published module. This is needed so that it loads the assembly that pwsh will have already loaded avoiding any potential conflicts, say pwsh uses 12.x.y but Posh-SSH uses 13.x.y.

If you need a specific feature in newer versions then you should look at AssemblyLoadContext to load that assembly in a special context that doesn't affect anything outside the code and can load whatever it wishes. If you don't and are happy using whatever pwsh has then just specify it as a PrivateAsset and rely on what pwsh already provides you.

darkoperator commented 2 years ago

Good to know :) since I'm not a C# developer (this is actually my first and only C# project) :) I would have never figured it out on my own. Will look into this. Thanks for the information.

jborean93 commented 2 years ago

If you need help I'm more than happy to review a PR or answer any questions. Your first binary module is always going to be full of questions and it's great having someone to bounce your questions off of. If you are in the pwsh Discord you can find my under the same nickname, there are also plenty of other people there who are very helpful and more knowledgeable than me on this topic.

Just an FYI here is one of my binary modules where 2 libraries are shipped with pwsh https://github.com/jborean93/PSOpenAD/blob/2cf7dd1b4c14f47ab1cc710d20ca6879a0578127/src/PSOpenAD.csproj#L14-L15. By adding PrivateAssets="all" they won't be included in my built module and will rely on whatever pwsh supplies.

MVKozlov commented 2 years ago

@jborean93, the module loads its assembly into the current ps session and AssemblyLoadContext is unfortunately not available on windows .net

The module tries to be compatible with both ps 5.1 and ps7+ so it may be that PrivateAsset='All' is the only solution. If we don't include e.g. Newtonsoft.Json.dll, where do we get it from? it is not a Windows/Linux standard library.

jborean93 commented 2 years ago

the module loads its assembly into the current ps session

This is the problem, once the assembly is loaded you can't load the same project at a different major version. PowerShell 7.1 ships with Newtonsoft.json at 12.0.0 while this module tries to load Newtonsoft.json at 13.0.0. It's not a problem on PowerShell 7.2 as that loads 13.0.0 but there is no guarantee that it will continue to align in the future and people still running LTS PowerShell versions won't be able to use your new module versions without hacking in workarounds. Assemblies are also process wide and not per PowerShell session which normally isn't problematic but can be when you start dealing with Runspaces.

Luckily even if you want to target both WinPS and PS you have a few options available to you:

The 2nd is probably the easiest option in terms of code changes but I personally prefer the first option. It allows you even define compile time options for each build allowing you do to things like this contrived example

public static string TestMethod()
{
#if DOTNET_FRAMEWORK
    return "ran from dotnet framework";
#else
    return "ran from dotnet";
#endif
}

This is great if want to implement stuff on new code and keep the old stuff for backwards compatibility until you finally drop that version. It would also allow you to use an ALC for the netcoreapp3.1 dll while keeping the net472 one code just loading it directly but that won;'t be needed if you define the csproj correctly for each target. The downside to this approach is a more complicated build setup but that's a once off problem rather than ongoing issue.

MVKozlov commented 2 years ago

So, if you can help @darkoperator with compiling and deploying the multitarget module, that would be awesome, because I'm not a real C# developer either :)

as a fast "assembly resolver" I can suggest this, but it should be tested

--- Posh-SSH.psm1       Fri May 20 01:24:12 2022
+++ Posh-SSH.psm1.new      Tue Aug 09 10:52:34 2022
@@ -1,4 +1,7 @@
-if ($PSVersionTable.PSVersion.Major -eq 5) {
+try {
+       [void][Newtonsoft.Json.JsonConverter]
+}
+catch {
     Add-Type -Path "$PSScriptRoot/Assembly/Newtonsoft.Json.dll"
 }
jborean93 commented 2 years ago

I can definitely try and submit a PR but one thing I notice is you use Visual Studio (.sln) whereas I'm mostly used to just VSCode. I can certainly create a PR that does things how I do it but I don't want to step on others toes with my personal choices.

jborean93 commented 2 years ago

Here is a WIP PR that does the first step to create separate binaries https://github.com/darkoperator/Posh-SSH/pull/471.

SeniorConsulting commented 2 years ago

This may or may not be of assistance, but there's a related bug on over here: https://github.com/microsoftgraph/msgraph-sdk-powershell/issues/1439

The recommendation here was to upgrade to PS7.2.5

darkoperator commented 2 years ago

I would have zero issue telling people to upgrade to PS7.2.5 :) especially since the builds of PowerShell were old. But I do realize I need to add tests and auto building to the project given how many people inside of MS and outside rely on it. The "Let me share this I build for this side project" is now a tool many depend on.

darkoperator commented 1 year ago

Try the latest release I pushed out yesterday, tested on Ununtu x64

Sent from my iPhone

On Jun 29, 2022, at 10:12 AM, Benjamin SAIZ @.***> wrote:

 Hope you'll find a solution

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.