Support credentials for PSGalleryModule and PSGalleryNuget

[alphakilo45]

We have run into an issue pulling dependencies from a private Nuget repository hosted in Azure DevOps Artifacts. The Find-NugetPackage function used by the PSGalleryNuget dependency type errors with the following: TF400813: Resource not available for anonymous access. Client authentication required. Note: The actual downloading of the package still works because it uses the nuget.exe CLI which can save credentials. This means that the "latest" version can't be used in this scenario.

The PSGalleryModule dependency type fails with a warning as well: "WARNING: Cannot access 'https://<AccountName>.pkgs.visualstudio.com/_packaging/<RepositoryName>/nuget/v2'. Are you missing 'Credential' parameter in the cmdlet?"

In both cases I think providing credentials for the repository would be useful. I'm more than happy to submit a PR with those changes if there is interest.

[alphakilo45]

To further detail the changes now that I have them implemented for our own use - I added a -Credentials parameter to Invoke-PSDepend that accepts a hashtable of named PSCredential objects. The individual dependency providers (specifically PSGalleryModule and PSGalleryNuget for now) will now accept a Credential parameter which is the name of the PSCredential object to use. PSDepend then looks up the credential and applies it to any requests made by the dependency provider.

Let me know if this sounds useful to anyone else and I'll submit a PR for a more in-depth review.

[johnmccrae]

@alphakilo45 Would you mind sharing out what/how you did that? I have that exact scenario I need to overcome.

[alphakilo45]

@johnmccrae Sure - you can see the changes in credentials branch on my fork of the project: https://github.com/alphakilo45/PSDepend/commits/credentials Beware though, I haven't merged in any changes since creating the fork

[byron-edmonds]

@alphakilo45 great suggestion. I've just run into exactly the same requirements. Thanks for sharing your changes.

[rlvandaveer]

@RamblingCookieMonster Is it possible to get this baked into the project? I just started using this module and find it very useful but I need to be able to support credentialed feeds. I'd rather not fork it, create a new pipeline, and host the module myself if I can help it. I'm willing to contribute on this though looking at @alphakilo45's changes, it looks complete.

[RamblingCookieMonster]

Arg, missed this issue, code looks reasonable but like you mentioned, there have been changes to that file - if you can get these changes into the current codebase, it looks good to me!!

[alphakilo45]

Absolutely - I can get things merged later this week and submit a PR (unless @rlvandaveer gets around to it first).

[rlvandaveer]

That's great. I may have time to tackle this week. I'll look at the branches and see what I think it'll take to resolve this issue given the state of master now.

[rlvandaveer]

I'm looking over PSGalleryNuget.ps1 and I'm wondering how much, if anything, needs to change. Nuget supports adding credentialed sources. It seems that Nuget should handle it as long as you have your source setup correctly. Any disagreement? I would think just updating the docs might be enough.

One items that should change in PSGalleryNuget.ps1 is to make it cross platform. I think that all references to nuget.exe should be changed to nuget i.e.,

if(-not (Get-Command Nuget.exe -ErrorAction SilentlyContinue))
{
    Write-Error "PSGalleryNuget requires Nuget.exe.  Ensure this is in your path, or explicitly specified in $ModuleRoot\PSDepend.Config's NugetPath.  Skipping [$DependencyName]"
}

and

    Invoke-ExternalCommand nuget.exe -Arguments $NugetParams

I'm working on a Mac in PowerShell Core and PSDepend works well, but obviously only because I haven't used a nuget dependency yet.

[rlvandaveer]

@RamblingCookieMonster, is Find-NugetPackage.ps1 still necessary? It appears to only exist in order to do a search for all package versions. As of the version I have installed, 4.8.2.5835, nuget appears to do this: nuget list <search terms> -source '<URL here>' -allversions

Why am I asking? I think it would be ideal to avoid configuring credentials in source control if possible. If we can rely on just nuget and credentialed sources configured in nuget.config, then that job becomes easier.

[rlvandaveer]

@RamblingCookieMonster, is Find-NugetPackage.ps1 still necessary? It appears to only exist in order to do a search for all package versions. As of the version I have installed, 4.8.2.5835, nuget appears to do this: nuget list -source '<URL here>' -allversions

Why am I asking? I think it would be ideal to avoid configuring credentials in source control if possible. If we can rely on just nuget and credentialed sources configured in nuget.config, then that job becomes easier.

Never mind. I just executed nuget list and I can totally see why that script exists. It's unfortunate since it would be really nice to just rely on native nuget logic.

[rlvandaveer]

So, let's talk about credential strategy. I thought about what @alphakilo45 did and the shortcoming is that it leads to a user likely saving credentials to source code. I think there are some ways to save an encrypted file in Windows PowerShell but my early tests via PowerShell Core haven't been fruitful.

I had a thought that instead of directly committing the credentials to the depend.psd1 or requirements.psd1, that we could instead include a reference to the userid and password variables. You could then get the variables using Get-Variable or Get-Item env: to pass to the web call or construct a 'PSCredential' instance. How you configure them upstream is up to you. E.g.,

        @{
            PSDeploy = @{
                DependencyType = 'PSPrivateGallery'
                Source = 'https://dev.azure.com/orgname/_packages/projectname/nuget/v2'
                Credentials = @{
                                UsernameVariable = 'env:username'
                                PasswordVariable = 'env:password'
                }
            }
        }

Is that naive? Is there a better way?

[ChrisLGardner]

For Windows you might want to look at BetterCredentials for storing creds or just export them as clixml, they'll only be usable by the same user on the same machine. For PSCore it's harder as secure string isn't really secure on non-windows (its just a string there).

On Mon, Jun 3, 2019, 21:26 rlvandaveer notifications@github.com wrote:

So, let's talk about credential strategy. I thought about what @alphakilo45 https://github.com/alphakilo45 did and the shortcoming is that it leads to a user likely saving credentials to source code. I think there are some ways to save an encrypted file in Windows PowerShell but my early tests via PowerShell Core haven't been fruitful.

I had a thought that instead of directly committing the credentials to the depend.psd1 or requirements.psd1, that we could instead include a reference to the userid and password variables. You could then get the variables using Get-Variable. How you configure them upstream is up to you. E.g.,

    @{
        PSDeploy = @{
            DependencyType = 'PSGalleryNuget'
            Source = 'https://dev.azure.com/orgname/_packages/projectname/nuget/v2'
            Credentials = @{
                            UsernameVariable = 'env:username'
                            PasswordVariable = 'env:password'
            }
        }
    }

Is that naive? Is there a better way?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/RamblingCookieMonster/PSDepend/issues/94?email_source=notifications&email_token=AEAOUFSIPLUHJH5Q24HBZY3PYVV6HA5CNFSM4GASDKA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW2N4ZQ#issuecomment-498392678, or mute the thread https://github.com/notifications/unsubscribe-auth/AEAOUFV2BQTARQF45TAEX6LPYVV6HANCNFSM4GASDKAQ .

[alphakilo45]

I agree that we don't want credentials in the .depend.psd1 files. The intention was was to provide a hash (keyName -> credential) of the credentials to your call to Invoke-PSDepend. The .depend.psd1 files then reference the key needed for each module and the matching credential object is passed to the dependency provider.

[rlvandaveer]

I agree that we don't want credentials in the .depend.psd1 files. The intention was was to provide a hash (keyName -> credential) of the credentials to your call to Invoke-PSDepend. The .depend.psd1 files then reference the key needed for each module and the matching credential object is passed to the dependency provider.

Ah, I see it. I hadn't gotten to the module invocation yet. I read the changes top down and when I saw Dependency.Credentials I thought that it was configured via the dependency declaration. I think that makes sense and is likely enough.

[rlvandaveer]

I have created PR #100. It doesn't yet have any unit tests.

[rlvandaveer]

I believe PR #100 is done. Unit tests have been added and I have tested retrieving from a private repo using Nuget and PowerShellGet based modules.

[pauby]

It's not clear how the -Credentials parameter is supposed to work (to me at least). And despite trying numerous different ways I simply cannot get it to work.

$d = @{ 
    module = @{ 
        Name = 'module-name'
        Parameters = @{ 
            Repository = 'internal-repo'
        } 
    }
}

$d | Invoke-PSDepend -import -install -force -Credentials @{ module = $creds } -v
# also tried:
# $d | Invoke-PSDepend -import -install -force -Credentials @{ 'module-name' = $creds } -v
# $d | Invoke-PSDepend -import -install -force -Credentials @{ 'internal-repo' = $creds } -v

I have my internal repo on an Azure DevOps Artifacts feed and the error is:

WARNING: Could not get response from query 'https://XXX.pkgs.visualstudio.com/_packaging/XXX/nuget/v2/FindPackagesById()?id='module-name'&$skip=0&$top=40'.
PackageManagement\Find-Package : No match was found for the specified search criteria and module name 'module-name'. Try Get-PSRepository to see all available registered module repositories.
At /opt/microsoft/powershell/6/Modules/PowerShellGet/PSModule.psm1:8850 char:9
+         PackageManagement\Find-Package @PSBoundParameters | Microsoft ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : ObjectNotFound: (Microsoft.PowerShel\u2026Cmdlets.FindPackage:FindPackage) [Find-Package], Exception
+ FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.FindPackage

If I run Find-Package and give it the -Credential $creds parameter it works fine. So the issue seems to be matching something with the -Credentials hash table.

I am running this in a CentOS Docker Container (mcr.microsoft.com/powershell:centos-7) under PowerShell Core. Output of $PSVersionTable:

Name                           Value
----                           -----
PSVersion                      6.2.1
PSEdition                      Core
GitCommitId                    6.2.1
OS                             Linux 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Suggestions?

[johnmccrae]

@pauby What do get if you execute Get-PSRepository? I found that for my private repo I had to register it like this. Your URL above is different and I don't know if that's an issue or not.

Register-PSRepository -Name 'MyRepo' -SourceLocation 'https://pkgs.dev.azure.com/my-org/_packaging/stuff/nuget/v2/' -PublishLocation 'https://pkgs.dev.azure.com/my-org/_packaging/stuff/nuget/v2/' -InstallationPolicy Trusted -Credential $credential -Verbose

[pauby]

@johnmccrae I get:

Name                      InstallationPolicy   SourceLocation
----                      ------------------   --------------
internal-repo             Trusted              https://XXX.pkgs.visualstudio.com/_packaging/XXX/nuget/v2
PSGallery                 Trusted              https://www.powershellgallery.com/api/v2

If I use find-module -repository internal-repo -credential $creds I get a list of the modules. So it's registered correctly.

Something I didn't mention above (because I simply forgot) is that I am running this in a Docker container locally under PowerShell Core. I am going to go back and update my original message with that and the $PSVersionTable.

[johnmccrae]

Now I gotcha.. ok, so when I try to update from my internal repo I call this:

Update-Module -Name 'internal-repo' -Credential $credential The $credential object is the userid and token used above to register the repo.

What I'm finding is that after I create the internal repo i now get warnings and sometimes errors when I install or update modules between PSGallery and Internal-Repo. For all PSGallery or other external modules I now need to specify -Repository on the Install-Module command. Update-Module works correctly after installation

[pauby]

How do I get that to work / get that functionality with PSDepend?

[johnmccrae]

OK, I'm a dork. I was responding to the wrong thing. Sorry.

[pauby]

If I run install-package -name 'module-name' -Source 'https://XXX.pkgs.visualstudio.com/_packaging/XXX/nuget/v2' -Credential $creds -providername powershellget it actually installs it.

[alphakilo45]

@pauby I think the piece that's missing is that you need to specify the key for the credentials in your dependency. For example from the PSDepend tests:

'jenkins' = @{
        DependencyType = 'PSGalleryNuget'
        Version = 'latest'
        Target = 'TestDrive:/PSDependPesterTest'
        Parameters = @{
            Force = $true
        }
        Credential = "imaginaryCreds"
    }

That key (in this case "imaginaryCreds") then needs to match the key in the hashtable passed in on the -Credentials parameter. (ex: $d | Invoke-PSDepend -import -install -force -Credentials @{ 'imaginaryCreds' = $creds })

[pauby]

I had a look at some of the code. Unfortunately I've not got a lot of familiarity with it and it appears to use a global variables so it's difficult to track stuff down.

I get the error message:

WARNING: No credential found for the specified name . Was the dependency misconfigured?

So I found that. Iyou look at the error message above and the code, $Name is empty. So I traced the code back to where the Resolve-Credential function is being called and it's only in one function here and here. If we look for $CredentialName in the code you can find it further up. The Get-GlobalOption function is found further up again.

Looking at the Get-GlobalOption function it looks like $Output = $Default would be executed and as $Default is $null that's what's returned. Because I don't know the code well enough I'm not sure what should be returned here.

I could be reading this code completely wrong and be way off base with the issue I have. It's late and I'm tired so would really appreciate any suggestions.

[pauby]

@alphakilo45 That has fixed it! The above is still an issue as it doesn't fail gracefully there (but tries to display a $null which I'm assuming is because it doesn't match anything in the Credentials hashtable.

I will see if I can raise a PR to update the docs tomorrow as they don't mention Credentials at all and the function help is ambiguous.

Thank you again!