search

PowerShell / PSResourceGet

PSResourceGet is the package manager for PowerShell
https://www.powershellgallery.com/packages/Microsoft.PowerShell.PSResourceGet
MIT License
485 stars 92 forks source link

Unable to publish-module to NuGet with a required module dependency #843

Open mmisztal1980 opened 1 year ago

mmisztal1980 commented 1 year ago

Prerequisites

Steps to reproduce

I'm working on emitting powershell modules from a github repo to nuget. My repo contains 2 modules:

B has a dependency on previously published version of A, therefore I've added it to it's RequiredModules . I've encountered a problem during Publish-Module and I'm attempting to solve it. If my understanding is correct, in order to successfully publish B with Publish-Module, A needs to be installed locally 1st as indicated here.

In my CI I'm using PowershellGet 3.0.17-beta17.

Expected behavior

I'm expecting `Install-Module` on A to succeed while installing B locally from a path on the local filesystem of a GitHub actions runner (ubuntu-latest), so that I can execute `Publish-Module` on B in the next step.

Actual behavior

The pipeline fails as seen here

Line |
   2 |  ./Publish.ps1 `
     |  ~~~~~~~~~~~~~~~
     | Failed to publish module CloudTek.Automation.K8S :   The specified
     | RequiredModules entry 'CloudTek.Automation.Shell' in the module manifest
     | '/home/runner/work/automation/automation/src/CloudTek.Automation.K8S/CloudTek.Automation.K8S.psd1' is invalid. Try again after updating this entry with valid values.

Error details

Exception             : System.Exception: No match was found for the specified 
                        search criteria and module name 'CloudTek.Automation.Sh
                        ell'. Try Get-PSRepository to see all available registe
                        red module repositories.
TargetObject          : Microsoft.PowerShell.PackageManagement.Cmdlets.InstallP
                        ackage
CategoryInfo          : ObjectNotFound: (Microsoft.PowerShel…lets.InstallPackag
                        e:InstallPackage) [Install-Package], Exception
FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageMan
                        agement.Cmdlets.InstallPackage
ErrorDetails          : 
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at Install-Module<Process>, /home/runner/.local/share/p
                        owershell/Modules/PowerShellGet/2.2.5/PSModule.psm1: li
                        ne 9711
                        at <ScriptBlock>, /home/runner/work/automation/automati
                        on/src/CloudTek.Automation.K8S/PrePublish.ps1: line 16
                        at <ScriptBlock>, /home/runner/work/automation/automati
                        on/src/CloudTek.Automation.K8S/PrePublish.ps1: line 7
                        at <ScriptBlock>, /home/runner/work/automation/automati
                        on/scripts/Publish.ps1: line 31
                        at <ScriptBlock>, /home/runner/work/_temp/476525ff-140e
                        -429e-810c-50cafc565387.ps1: line 2
                        at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {0, 1}

Environment data

Key   : PSVersion
Value : 7.2.6
Name  : PSVersion

Key   : PSEdition
Value : Core
Name  : PSEdition

Key   : GitCommitId
Value : 7.2.6
Name  : GitCommitId

Key   : OS
Value : Linux 5.15.0-1022-azure #27~20.04.1-Ubuntu SMP Mon Oct 17 02:03:50 UTC 
        2022
Name  : OS

Key   : Platform
Value : Unix
Name  : Platform

Key   : PSCompatibleVersions
Value : {1.0, 2.0, 3.0, 4.0…}
Name  : PSCompatibleVersions

Key   : PSRemotingProtocolVersion
Value : 2.3
Name  : PSRemotingProtocolVersion

Key   : SerializationVersion
Value : 1.1.0.1
Name  : SerializationVersion

Key   : WSManStackVersion
Value : 3.0
Name  : WSManStackVersion

Visuals

No response

anamnavi commented 1 year ago

@mmisztal1980 I see you've updated to PowerShellGet V3 (with version 3.0.17-beta17), but I should note that PowerShellGetV3 was a rewrite and the cmdlets' names have changed from PowerShellGetV2. So Publish-Module is now Publish-PSResource. So if you can try running Publish-PSResource -Path <path_To_.psd1 or path_to_folder_with_.psd1> and let us know if this still reproduces that would be great, thanks!

Can you also share the module manifest (.psd1) file's contents with us please? Thanks!

mmisztal1980 commented 1 year ago

@anamnavi I'm getting a similar result: https://github.com/cloud-tek/automation/actions/runs/3436020892/jobs/5729064700

The manifest can be seen here: https://github.com/cloud-tek/automation/blob/main/src/CloudTek.Automation.K8S/CloudTek.Automation.K8S.psd1

Please note that versioning is achieved using gitversion in CI

anamnavi commented 1 year ago

@mmisztal1980 thanks for sharing this info! We'll work to resolve this soon

mmisztal1980 commented 1 year ago

@anamnavi looking forward to hear from you. Meanwhile, can you provide some guidance on how this should be done? I'll try to make sure that my impl is as close to your guidance as possible

anamnavi commented 1 year ago

@mmisztal1980 my apologies for the confusion, specifying ModuleVersion with your RequiredModules should work actually. Here's an example of a package we have that has RequiredModules specified: https://www.powershellgallery.com/packages/PowerShellGet/2.2.5/Content/PowerShellGet.psd1

Could you try using single quotes around the ModuleName and ModuleVersion values. Also could you remove the ; at the end of ModuleVersion (the last key value pair) in your example. So your RequiredModules value should look like this:

  RequiredModules = @(
    @{
      ModuleName = 'CloudTek.Automation.Shell';
      ModuleVersion = '0.1.119'
    }
  )
mmisztal1980 commented 1 year ago

@anamnavi I've applied your suggestions. Unfortunately it's failing.

https://github.com/cloud-tek/automation/actions/runs/3527879669/jobs/5917407395

Line |
   2 |  ./Publish.ps1 `
     |  ~~~~~~~~~~~~~~~
     | Failed to publish module CloudTek.Automation.K8S :   The specified
     | RequiredModules entry 'CloudTek.Automation.Shell' in the module manifest
     | '/home/runner/work/automation/automation/src/CloudTek.Automation.K8S/CloudTek.Automation.K8S.psd1' is invalid. Try again after updating this entry with valid values. Run 'Test-ModuleManifest' to validate the module manifest.
anamnavi commented 1 year ago

@mmisztal1980 Your syntax of the .psd1 is now correct and that's not the issue. The issue is that it needs to find the RequiredModule you specify on your system. Looking at your failed job it first fails to install CloudTek.Automation.Shell Version="0.1.119" on L96. That's why it later fails to publish the CloudTek.Automation package which takes a dependency on CloudTek.Automation.Shell...it should be able to install that package from nuget.org.

You could install this with PowerShellGet V3 using the following commands: Register-PSResourceRepository -Name "NuGetGallery" -Uri "https://api.nuget.org/v3/index.json" -Trusted Install-PSResource -Name "CloudTek.Automation.Shell" -Version "0.1.119" -Repository NuGetGallery -Verbose

Once you resolve that issue and CloudTek.Automation.Shell is installed, the publish step should work.

I understand the PowerShell message saying the .psd1 entry is invalid is not clear/intuitive enough and should instead indicate that the requiredModule entry package was not installed...we'll fix this aspect.

mmisztal1980 commented 1 year ago

@anamnavi let me update the code and try it out really quick

mmisztal1980 commented 1 year ago

@anamnavi it appears to have worked https://github.com/cloud-tek/automation/actions/runs/3568486063/jobs/5997407163

I have some questions:

mmisztal1980 commented 1 year ago

@anamnavi a problem I'm noticing in CI is:

Q:

mmisztal1980 commented 1 year ago

@anamnavi FYI, I've updated the upstream

mmisztal1980 commented 1 year ago

@anamnavi I believe the issue may be hidden in registering the local PSResourceRepository

https://github.com/cloud-tek/automation/actions/runs/3615952444/jobs/6093460572

After registration, Get-PSResourceRepository yields:

Name                            Uri                                                                          Trusted Pri
                                                                                                                     ori
                                                                                                                     ty
----                            ---                                                                          ------- ---
CloudTek.Automation.Shell-local file:///home/runner/work/automation/automation/src/CloudTek.Automation.Shell True    10
PSGallery                       https://www.powershellgallery.com/api/v2                                     True    20
nuget                           https://api.nuget.org/v3/index.json                                          True    40
NuGetGallery                    https://api.nuget.org/v3/index.json                                          True    50

However running Find-PSResource -Name "CloudTek*" against those repositories does not detect the local package in the CloudTek.Automation.Shell-local repository

Name                      Version   Prerelease Repository   Description
----                      -------   ---------- ----------   -----------
CloudTek.Automation.Shell 0.1.152.0            nuget        This package contains reusable powershell automation for ex…
CloudTek.Automation.K8S   0.1.68.0             nuget        This package contains reusable kubernetes powershell automa…
CloudTek.Build            0.11.3.0             nuget        Package Description
CloudTek.Automation.Shell 0.1.152.0            NuGetGallery This package contains reusable powershell automation for ex…
CloudTek.Automation.K8S   0.1.68.0             NuGetGallery This package contains reusable kubernetes powershell automa…
CloudTek.Build            0.11.3.0             NuGetGallery Package Description

Installing explicitly from that repo fails as well

  Install-PSResource -RequiredResource @{
    "$($_.ModuleName)" = @{
      version = "$($_.ModuleVersion)"
      repository = "$($_.ModuleName)-local"
     }
  } -Verbose -Debug;

Are there any preconditions I should be aware of?

# $path = "/home/runner/work/automation/automation/src/CloudTek.Automation.Shell"
function Register-LocalPSResourceRepository([string] $path, [string]$name) {
  [string]$p = Resolve-Path -Path $path;
  Write-Host "Registering PSRepository ($name : $p) ..." -ForegroundColor Gray;
  [hashtable[]]$repositories = @(
    @{ Name = $name; Uri = "$p"; Trusted = $true; Priority = 10 }
  );

  Register-PSResourceRepository -Repository $repositories -Verbose;
}
anamnavi commented 1 year ago

@mmisztal1980 thanks for raising these questions, I will work on making the documentation clear for such examples/limitations.

The intended flow when finding a package from a local repository is as follows:

  1. Register the local repository. This will be a folder where package .nupkg files are stored. (Register-PSResourceRepository -Name "MyLocalRepo" -Uri "C:\Users\johndoe\Documents\RepoFolder")
  2. Ensure the .nupkg file for the package is present at this location. You could either manually move the .nupkg file to the repository path or you can use Save-PSResource which basically downloads the package from a repository to a location of your specified choice. (Save-PSResource -Name "CloudTek.Automation.Shell" -Repository NuGetGallery -Path "C:\Users\johndoe\Documents\RepoFolder" -AsNupkg). The -AsNupkg is important otherwise it'll download and extract the contents of the package and you'll have a folder of contents in the repository path, but you want the .nupkg file.
  3. Now you can search for the package from your local repository. (Find-PSResource -Name "CloudTek.Automation.Shell" -Repository MyLocalRepo). Because a .nupkg file is present at the local repository path it will find that package in your local repository.

So to answer your 3rd question the expected structure of your local repository's path is that it should contain .nupkg file for the package you wish to find there. If your local repository should have 10 packages there would be 10 .nupkg files. Let me know if this helps

mmisztal1980 commented 1 year ago

I think I was missing step 2. Let me get to work. I'll update you tonight

mmisztal1980 commented 1 year ago

@anamnavi Unfortunately there still seems to be an issue.

pipeline run: https://github.com/cloud-tek/automation/actions/runs/3625290829/jobs/6113171752 code: https://github.com/cloud-tek/automation/blob/main/src/CloudTek.Automation.K8S/PrePublish.ps1

What I've done so far:

[string]$repository =  "CloudTek.Automation.Shell-local";
[string]$path = Resolve-Path -Path "$PSScriptRoot/../CloudTek.Automation.Shel";
 Register-LocalPSResourceRepository -name $repository -path $path;
 Save-PSResource `
    -Name "CloudTek.Automation.Shell" `
    -Repository $repository `
    -Version "<current gitversion>" `
    -Path $path ` 
    -AsNupkg -Verbose;

The step fails with:

Package 'CloudTek.Automation.Shell' with requested version range '[0.1.160, 0.1.160]' could not be installed as it was not found in any registered repositories
mmisztal1980 commented 1 year ago

Am I packing the module in a wrong way? I basically want to pack the contents of the module folder into a .nupkg file so it becomes visible

mmisztal1980 commented 1 year ago

@anamnavi I've just noticed that your step 2 assumes that the dependency package has been previously published to the target repository. This is a problem. It may take N minutes for the package to become visible in the remote repository, which will impact CI. Instead, I'd like to pack this package locally and install it from a local source. How can I do "pack" on it?

mmisztal1980 commented 1 year ago

@anamnavi I've figured out how to publish local .nupkg modules . 

  Get-PSRepository -Name "local";

  [hashtable] $publishArgs = @{
    Repository = "local"
    Path = "$PSScriptRoot/../src/$module"
  };

  Publish-Module @publishArgs;

Find-Module works as expected. I'll refactor my solution to adjust for this approach and continue from there

PS /Users/mmisztal/Checkout/cloudtek/automation> find-module -Repository local
WARNING: The file extension '/Users/mmisztal/Checkout/cloudtek/automation/packages/.gitkeep' is not valid. The required file extension is '.nupkg'.

Version              Name                                Repository           Description
-------              ----                                ----------           -----------
1.2.3                CloudTek.Automation.Shell           local                This package contains reusable powershell automatio
mmisztal1980 commented 1 year ago

@anamnavi I'm making progress. I've redesigned the pipeline to:

While doing so I'm testing some of the functionality.

Q:

mmisztal1980 commented 1 year ago

@anamnavi I've successfully published both packages 🎉

image

https://github.com/cloud-tek/automation/actions/runs/3671046905

Before doing so, I run into:

Line |
   2 |  ./Publish.ps1 `
     |  ~~~~~~~~~~~~~~~
     | Failed to publish module CloudTek.Automation.K8S :   Dependency
     | 'CloudTek.Automation.Shell' was not found in repository 'nuget'.  Make
     | sure the dependency is published to the repository before publishing
     | this module.

Thankfully Publish-PSResource -SkipDependenciesCheck managed to solve this for me.

Please feel free to review my pipeline & code and verify if this is the way you intend the process to be with v3 for repos emitting multiple co-dependent modules which have their shared version set during CI

mmisztal1980 commented 1 year ago

@anamnavi I've noticed another interesting issue:

Write-Error: /home/runner/work/_temp/bee1aece-6a76-4428-bf17-8b30840c5fc7.ps1:2
Line |
   2 |  ./Publish-Local.ps1 `
     |  ~~~~~~~~~~~~~~~~~~~~~
     | Failed to publish module locally, CloudTek.Automation.Shell :   The
     | 'ModuleVersion' member is not valid in the module manifest file
     | '/home/runner/work/automation/automation/src/CloudTek.Automation.Shell/CloudTek.Automation.Shell.psd1': 
Cannot convert value "0.2.0-beta0002" to type "System.Version". 
Error: "Input string was not in a correct format."

What is the correct format? From what I can see, the current pre-release version of PowershellGet is 3.0.17-beta17 . I've tried both:

Update I've found the docs for prerelease modules: https://learn.microsoft.com/en-us/powershell/scripting/gallery/concepts/module-prerelease-support?view=powershell-7.3

Q: How do I specify a required module as a pre-release dependency?

  RequiredModules = @(
    @{
      ModuleName = "CloudTek.Automation.Shell";
      ModuleVersion = "0.0.0"
    }

This obiect does not have a PreRelease property, correct?

mmisztal1980 commented 1 year ago

@anamnavi I've emitted pre-release packages for 0.2.0-beta.7 but got a false positive image

https://github.com/cloud-tek/automation/actions/runs/3678970392/jobs/6222881440

Am I missing anything? Apart from settings PreRelease in psd1 is there anything else I should be aware of?

anamnavi commented 1 year ago

@mmisztal1980

When publishing the dependency package, in the .psd1 file specify ModuleVersion = '0.2.0' and then in the PSData -> Prerelease = 'beta7' (I would recommend that over beta.7).

Now when publishing the parent package (which has the dependency) in your .psd1 file, for the RequiredModules property ModuleVersion could you try either skip specifying the version (leave ModuleVersion empty and it should get the latest) or specify the version with prerelease label there (i.e 0.2.0-beta7). Does that get the prerelease version of the dependency package?

Wrt to which version to use when publishing, I hope this helps, otherwise if you can share the command used in publish.ps1 script that would help.

Yes unfortunately using wildcard ('*') with package name to search a local repository is not supported. One potential solution is that if you wish to see all packages in the local repo you could do ls at the repo uri (if it's a file path). For example if I have repository MyLocalRepo with Uri C:\Users\johndoe\Documents\RepoFolder I could do ls C:\Users\johndoe\Documents\RepoFolder and with recursive search you could get version too. Wildcard support for local repositories is something we hope to fix in upcoming releases.

Let me know if you have other questions, thanks

mmisztal1980 commented 1 year ago

Is my understanding correct?

  RequiredModules = @(
    @{
      ModuleName = "CloudTek.Automation.Shell";
      ModuleVersion = "0.2.0-beta7"
    }
anamnavi commented 1 year ago

@mmisztal1980 yes, can you try that? First ensure that CloudTek.Automation.Shell with version 0.2.0-beta7 is present.

mmisztal1980 commented 1 year ago

I just examined my manifest, version rendering failed for some reason, I'm going to write some Pester tests to verify it's rock-solid. I'll keep you posted

mmisztal1980 commented 1 year ago

@anamnavi I've written some PESTER tests, and I'm running into an issue I can't pinpoint yet.

My test renders a Test02.psd1 module manifest:

Test02.psd1 ```pwsh # # Module manifest for module 'Test02' # # Generated by: CloudTek # # Generated on: 09/25/2022 # @{ # Script module or binary module file associated with this manifest. # RootModule = '' # Version number of this module. ModuleVersion = "0.9.0" # Supported PSEditions CompatiblePSEditions = 'Core' # ID used to uniquely identify this module GUID = '936efc6d-dc78-4f10-ac35-c21f2c14b092' # Author of this module Author = 'CloudTek' # Company or vendor of this module CompanyName = 'CloudTek' # Copyright statement for this module Copyright = '' # Description of the functionality provided by this module Description = 'This is a module used for testing.' # Minimum version of the PowerShell engine required by this module PowerShellVersion = '7.2' # Name of the PowerShell host required by this module # PowerShellHostName = '' # Minimum version of the PowerShell host required by this module # PowerShellHostVersion = '' # Minimum version of Microsoft .NET Framework required by this module. This prerequisite is valid for the PowerShell Desktop edition only. # DotNetFrameworkVersion = '' # Minimum version of the common language runtime (CLR) required by this module. This prerequisite is valid for the PowerShell Desktop edition only. # ClrVersion = '' # Processor architecture (None, X86, Amd64) required by this module # ProcessorArchitecture = '' # Modules that must be imported into the global environment prior to importing this module RequiredModules = @( @{ ModuleName = "CloudTek.Automation.Shell"; ModuleVersion = "0.9.0-beta5" } ) # Assemblies that must be loaded prior to importing this module # RequiredAssemblies = @() # Script files (.ps1) that are run in the caller's environment prior to importing this module. # ScriptsToProcess = @() # Type files (.ps1xml) to be loaded when importing this module # TypesToProcess = @() # Format files (.ps1xml) to be loaded when importing this module # FormatsToProcess = @() # Modules to import as nested modules of the module specified in RootModule/ModuleToProcess NestedModules = @("./Script.psm1") # Functions to export from this module, for best performance, do not use wildcards and do not delete the entry, use an empty array if there are no functions to export. FunctionsToExport = @("Deploy-HelmTemplate", "Kubectl-Apply") # Cmdlets to export from this module, for best performance, do not use wildcards and do not delete the entry, use an empty array if there are no cmdlets to export. CmdletsToExport = @() # Variables to export from this module VariablesToExport = @() # Aliases to export from this module, for best performance, do not use wildcards and do not delete the entry, use an empty array if there are no aliases to export. AliasesToExport = @() # DSC resources to export from this module # DscResourcesToExport = @() # List of all modules packaged with this module # ModuleList = @() # List of all files packaged with this module # FileList = @() # Private data to pass to the module specified in RootModule/ModuleToProcess. This may also contain a PSData hashtable with additional module metadata used by PowerShell. PrivateData = @{ PSData = @{ # Tags applied to this module. These help with module discovery in online galleries. # Tags = @() # A URL to the license for this module. # LicenseUri = '' # A URL to the main website for this project. ProjectUri = 'https://github.com/cloud-tek/automation' RepositoryUrl = 'https://github.com/cloud-tek/automation' # A URL to an icon representing this module. IconUri = 'https://avatars.githubusercontent.com/u/35167581?s=400&u=ca5fdf8da213a9ab3edd83813b5f3491dea70f6c&v=4' # ReleaseNotes of this module # ReleaseNotes = '' # Prerelease string of this module Prerelease = 'beta5' # Flag to indicate whether the module requires explicit user acceptance for install/update/save # RequireLicenseAcceptance = $false # External dependent modules of this module # ExternalModuleDependencies = @() } # End of PSData hashtable } # End of PrivateData hashtable # HelpInfo URI of this module # HelpInfoURI = '' # Default prefix for commands exported from this module. Override the default prefix using Import-Module -Prefix. # DefaultCommandPrefix = '' } ```

The error I'm getting is:

The variable cannot be validated because the value  is not a valid value for the Version variable.

Please note that I'm getting this with:

I'm looking though the source code but haven't been able to pinpoint the issue just yet. Any suggestions?

anamnavi commented 1 year ago

@mmisztal1980 thanks for providing such thorough info...I looked into it and it appears to be a limitation with RequiredModules's ModuleVersion property where it can't take a prerelease version specifically (i.e 0.9.0-beta will not work, but 0.5.0 would work). The ModuleVersion is the lowest possible version that would work, so if you specify ModuleVersion = 0.8.9 that would be lower than 0.9.0-beta and would work. Could you try that, thanks.

mmisztal1980 commented 1 year ago

@anamnavi I'm not sure I understand

To clarify:

RequiredModules = @(
    @{
      ModuleName = "CloudTek.Automation.Shell";
      ModuleVersion = "0.9.0" # <-- this needs to be without the suffix, correct?
    }
  )

I'm specifically aiming at the scenario where both the current module and dependencies share the version as they are the outputs of the same build, which is subject to GitVersion versioning. This includes feature/pre-release branches which are supposed to emit prerelease modules

Edit

Your suggestion worked, however I'm not sure it is my desired option. Q: Can my pre-release package require another pre-release package?

This issue would imply that it is not possible, however I still feel it needs to be asked

alerickson commented 1 year ago

@mmisztal1980 we'll consider this for future releases.