Cannot build PowerShell module with nested folders/files

avishnyakov commented 4 years ago

Steps to reproduce

git clone https://github.com/SubPointSolutions/uplift-powershell.git
cd uplift-powershell\invoke-uplift
pwsh -c invoke-build Clean, PrepareModule, VersionModule, BuildModule, InstallModule, ValidateInstalledModule

Expected behavior

Can build and publish module

Actual behavior

Cannot build neither publish module

Environment data

Name                           Value
----                           -----
PSVersion                      6.2.4
PSEdition                      Core
GitCommitId                    6.2.4
OS                             Microsoft Windows 10.0.18363
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

> Get-Module -ListAvailable PowerShellGet,PackageManagement
ModuleType Version    Name                                PSEdition ExportedCommands
---------- -------    ----                                --------- ----------------
Script     1.4.6      PackageManagement                   Desk      {Find-Package, Get-Packa…

    Directory: C:\program files\powershell\6\Modules

ModuleType Version    Name                                PSEdition ExportedCommands
---------- -------    ----                                --------- ----------------
Script     1.3.2      PackageManagement                   Desk      {Find-Package, Get-Packa… Script     2.1.3      PowerShellGet

> Get-PackageProvider -ListAvailable
Name                     Version          DynamicOptions  
----                     -------          --------------  
NuGet                    3.0.0.1          Destination, Exc
PowerShellGet            2.1.3.0          PackageManagemen

Submitting this report as a continuation of https://github.com/PowerShell/PowerShellGet/issues/478 In particular, these comments:

https://github.com/PowerShell/PowerShellGet/issues/478#issuecomment-585079070

It seems that dotnet pack is stuck while packing a .csproj project with 5+ files due to tons of warning produces by the dotnet pack. The dotnet.exe is stuck in the process explorer. Killing it unblocks the buil process producing errors and output mentioned in #478

This does not happen with a single-powershell file module, only with 5+ and more files, especially in subfodlers.

Same happens in linux container, unfortionatly.

Can get another repro but steps mentioned in this ticket should be enought. Let me know what else I can help with.

SydneyhSmith commented 4 years ago

Thanks for opening this issue... @avishnyakov have you had this issue with other versions of dotnet? Also, have you had any success with using the nuget or dotnet commands to publish?

avishnyakov commented 4 years ago

Yes, 3.1 and 2.2 at least.

I am containerizing build just right now so I can try different combinations of PowerShellGet and dotnet. I should be able to find the right combination of these. My module used to be going alright with AppVeyor long time ago, like a year (the uplift-powershell module) but not anymore.

Will share more context and updates today.

avishnyakov commented 4 years ago

Trying different containers, dotnet and package versions. Now getting even more confused.

Using container azuresdk/azure-powershell-core was probably not the best idea. Failing while trying to package nuget. Old packages, old nuget.

Name              Version
----              -------
PackageManagement 1.1.7.2
PowerShellGet     1.6.7

Name          Version
----          -------
NuGet         2.8.5.210
PowerShellGet 1.6.7.0

.NET Core SDK (reflecting any global.json):
 Version:   2.2.402 OR
 Version:   3.1.101

ERROR: Failed to generate the compressed file for module ''.
At /opt/microsoft/powershell/6/Modules/PowerShellGet/PSModule.psm1:10814 char:17
+ ...             Publish-PSArtifactUtility @PublishPSArtifactUtility_Param …

Using mcr.microsoft.com/powershell:latest container with dotnet-sdk-3.1, dotnet-sdk-2.1 and dotnet-sdk-2.2 gives another error related to dates. New packages and nuget 3.0+

https://github.com/NuGet/Home/issues/7001

Name          Version
----          -------
NuGet         3.0.0.1
PowerShellGet 2.1.3.0

Name              Version
----              -------
PackageManagement 1.3.2
PowerShellGet     2.1.3

 Version:   3.1.101

and the failure is

Microsoft (R) Build Engine version 16.4.0+e901037fe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.
  Restore completed in 3.67 sec for /tmp/fdd904cb-bf81-4114-b7fc-5d99aa6a48cc/Temp.csproj.
  Temp -> /tmp/fdd904cb-bf81-4114-b7fc-5d99aa6a48cc/bin/Debug/netcoreapp2.0/NotUsed.dll
/usr/share/dotnet/sdk/3.1.101/Sdks/NuGet.Build.Tasks.Pack/build/NuGet.Build.Tasks.Pack.targets(198,5): error : The DateTimeOffset specified cannot be converted into a Zip file timestamp. (Parameter 'value') [/tmp/fdd904cb-bf81-4114-b7fc-5d99aa6a48cc/Temp.csproj]      
'.
At /opt/microsoft/powershell/6/Modules/PowerShellGet/PSModule.psm1:10784 char:17
+ ...             Publish-PSArtifactUtility @PublishPSArtifactUtility_Param ...

avishnyakov commented 4 years ago

I am getting more and more confused.

From what I see, packaging is now failing due to newest versions of nuget/dotnet and deterministic changes which lock dates of the files or something.

Based on the following discussion, several ideas to fix this:

https://github.com/dotnet/core/issues/3388#issuecomment-532360477
dotnet pack -p:Deterministic=false
The compiler respects the true property, and starting with 3.0 so does NuGet at pack time.

Trying other combinations of dotnet/nuget/packages and containers. This is a bit frustrating, tbh, several days walking on the minefield.

avishnyakov commented 4 years ago

Ha! Same-same issue by @bo67192 https://github.com/PowerShell/PowerShellGet/issues/575

avishnyakov commented 4 years ago

Bingo. Found working combination of dotnet and nuget.

@bo67192, FYI

dotnet-sdk-2.0.2 works well. It has to do with nuget being under version 5. With 2.0.2 it comes like this:

NuGet Command Line 4.4.0.3

I would suggest to get a docker container to build powershell modules and pin version to ensure stabilkty. Something like this will do:

FROM  mcr.microsoft.com/powershell:6.2.4-ubuntu-16.04

RUN apt-get update

RUN apt-get install -y apt-transport-https
RUN apt-get install -y wget 

RUN wget -q https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
RUN dpkg -i packages-microsoft-prod.deb

RUN apt-get update

# no need, seems to be already there
# RUN pwsh -c Register-PSRepository -Default

RUN pwsh -c Install-Module -Name InvokeBuild -Force
RUN pwsh -c Install-Module -Name Pester -Force

RUN apt-get install -y dotnet-sdk-2.0.2

Moreover, I would suggest to improve CI/CD and testing before releasing PowerShellGet. Saw CI runs different builds for various configs. Maybe docker-izing or at least running docker-ized builds against different dotnet version will add some value from the regression testing perspective?

avishnyakov commented 4 years ago

@SydneyhSmith, up. Just making sure this is acknowledged by the Microsoft team.

alerickson commented 4 years ago

@avishnyakov thanks for the extensive testing and suggestions! I highly agree with the suggestion to move onto docker containers.

PowerShell / PowerShellGetv2