Faster way to install all AWS modules

KirkMunro commented 3 years ago

I appreciate what you have tried to do by using the AWS.Tools.Installer module as an easy way to install and update modules (likely in an attempt to correct some of what is wrong with the PowerShellGet way of managing installed modules), and I also appreciate that you have taken your uber-module and broken it up into smaller modules that will provide faster load times; however, you seem to be missing an easy, (as) fast (as possible) way to install all of the existing AWS.Tools.* modules.

In the background right now I am running the following command to install all AWS.Tools.* modules:

find-module AWS.Tools.* | where Name -notmatch 'Installer$' | install-module -Scope AllUsers

This is dreadfully slow, likely in part because each individual AWS.Tools.* module has dependencies on the same modules, which means looking up those dependencies for each module installed this way, one at a time, resulting in a terrible installation performance. I must pull down the entire module bundle so that I simply have what I need on my system when I need it for automation purposes. I am not willing to simply go back to an installer again and again for bite-sized installation of modules. If I go back to install something that is missing, I'll update the entire thing.

Note that, because of some of the challenges that are left unresolved with PowerShellGet, some of us "upgrade" modules and clean up older versions simply by removing what is installed on a system via recursive file system removal, and then re-installing the modules that we have removed. This is how I upgrade module bundles (Az, Microsoft.Graph, and now AWS.Tools), so that I don't have extra bits that I don't need kicking around.

I went looking in AWS.Tools.Installer for a way to install the entire module in a smart and fast way, but there doesn't seem to be such an option. I also went looking for a AWS.Tools module that would simply act as the link between all AWS.Tools.* modules, so that you could install AWS.Tools using Install-Module AWS.Tools, and you would get the whole thing, in a manner smart enough to handle dependencies well, but there is no AWS.Tools module.

Describe the Feature

I want an easy and fast way to install all modules in the AWS.Tools bundle. Ideally I would like it to be via PowerShellGet, rather than an AWS.Tools.Installer module, so that I have a consistent method to deal with module management.

Is your Feature Request related to a problem?

Yes. It is difficult to install AWS.Tools.* modules in a manner consistent with other module bundles, in a performant (or at least as fast as possible) way.

Proposed Solution

Create an AWS.Tools module that is nothing but a manifest module, with each of the AWS.Tools.* modules listed in that manifest in the RequiredModules section.

Describe alternatives you've considered

I considered using AWS.Tools.Installer, but that wants me to list each and every module in the bundle. While I could write a script to build the list of module names and then paste that array into the command to install the modules, that work should not be necessary. Also, I prefer consistency, especially given that PowerShellGet is currently in pre-release for version 3, which should give us all a better foundation for managing modules -- hopefully one that is easier to contribute to as open source.

Additional Context

I believe I have provided enough detail already.

Environment

No specific requirements: just easy, fast installation of the AWS module bundle on a system where a lot of automation is run.

[ ] :wave: I may be able to implement this feature request
[ ] :warning: This feature might incur a breaking change

This is a :rocket: Feature Request

github-actions[bot] commented 2 years ago

We have noticed this issue has not recieved attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.

Lujade commented 1 year ago

Reopen please. Installing modules takes ages.

ashishdhingra commented 1 year ago

Needs review with the team.

HP-85 commented 1 year ago

If anyone is up for creating a universal ps1 install script for AWS.Tools as a simple way to download, install (or upgrade) all AWS PS modules - I put together these 2 sets of commands. It took between 15 seconds (t4g.micro Amazon Linux 2023 arm64) and 30 seconds (t2.xlarge windows 10) to execute,

Advantages:

AWS.Tools does not require the 'Import-Module AWSPowerShell.NetCore' command
AWS Modules auto-load quickly based on the AWS command.
No need search for and install modules such as 'Install-AWSToolsModule AWS.Tools.EC2,AWS.Tools.S3'.

User-installed modules: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_psmodulepath?view=powershell-7.3

"User-installed modules: On Windows, modules installed in the CurrentUser scope are typically stored in the $HOME\Documents\WindowsPowerShell\Modules folder."

"On non-Windows systems, modules installed in the CurrentUser scope are stored in the $HOME/.local/share/powershell/Modules"

Windows:

# Windows:  Tested on Windows PS 5.1 and PS 7
Invoke-WebRequest -Uri https://sdk-for-net.amazonwebservices.com/ps/v4/latest/AWS.Tools.zip -OutFile $env:USERPROFILE\Documents\WindowsPowerShell\AWS.Tools.zip
Expand-Archive -Path $env:USERPROFILE\Documents\WindowsPowerShell\AWS.Tools.zip $env:USERPROFILE\Documents\WindowsPowerShell\Modules -Force
Remove-Item $env:USERPROFILE\Documents\WindowsPowerShell\AWS.Tools.zip

Linux / macOS:

# Linux / macOS :  Tested on PS 7.3.4 on Ubuntu 22.04, Amazon Linux 2023, macOS 13
Invoke-WebRequest -Uri https://sdk-for-net.amazonwebservices.com/ps/v4/latest/AWS.Tools.zip -OutFile $env:HOME/.local/share/powershell/AWS.Tools.zip 
Expand-Archive -Path $env:HOME/.local/share/powershell/AWS.Tools.zip -DestinationPath $env:HOME/.local/share/powershell/Modules/ -Force
Remove-Item $env:HOME/.local/share/powershell/AWS.Tools.zip

HP-85 commented 10 months ago

Updated use single script for PowerShell 7 on Linux, Mac, and Windows, and PowerShell 5 on Windows.

# Install AWS.Tools for PowerShell on Windows, Mac, or Linux
# Installs to user's PS Modules folder so installation does not require eleveated prompt
# Deletes existing AWS.Tools module directories to clear older versions
# coolcrai 2023-10-14

# Linux and macOS PS7
If ($IsLinux -eq $true -or $IsMacOS -eq $true) {
    $PSMUserPath = $env:PSModulePath -split ':' | Where-Object { $_ -like "$env:HOME*" }
}

# Windows PS5 PS7  (PS5 does not support $IsWindows, so check for $env:OS instead)
If ($env:OS -eq "Windows_NT") {
    $PSMUserPath = $env:PSModulePath -split ';' | Where-Object { $_ -like "*$env:HOMEPATH*" }
}

Write-Host 'Installing AWS.Tools to '  -NoNewline
Write-Host $PSMUserPath
#need if 'null' write-host 'error' and quit

Measure-Command {
    #Delete current AWS Modules:
    #If you see errors like 'Access to the path 'aws-crt-auth.dll' is denied.' quit and re-leaunch PowerShell
    If (Test-Path $PSMUserPath/Aws.Tools.*) {
        Remove-Item -Path $PSMUserPath/Aws.Tools.* -Force -Recurse 
    }
    #Create PSModulePath Dir if doesn't exist - common after installing PS7 on Windows 
    If (-not (Test-Path $PSMUserPath)) { 
        New-Item -Path $PSMUserPath -Type Directory
    }
    #Download .zip file
    If ((Get-Host).Version.Major -eq 7) {
        Invoke-WebRequest `
            -Uri https://sdk-for-net.amazonwebservices.com/ps/v4/latest/AWS.Tools.zip `
            -OutFile $PSMUserPath/AWS.Tools.zip 
    }
    If ((Get-Host).Version.Major -eq 5) {
        Start-BitsTransfer  `
            -Source https://sdk-for-net.amazonwebservices.com/ps/v4/latest/AWS.Tools.zip `
            -Destination $PSMUserPath/AWS.Tools.zip
    }
    # Need else if neither - write-host 'error' and quit

    #Install modules
    Expand-Archive -Path $PSMUserPath/AWS.Tools.zip -DestinationPath $PSMUserPath -Force
    #Delete AWS.Tools.zip file
    Remove-Item $PSMUserPath/AWS.Tools.zip
}
#the end

HP-85 commented 8 months ago

AWS Tools for PowerShell - cross platform install script

PowerShell script to install all AWS.Tools modules in PowerShell 7 on Linux, Mac, and Windows, or PowerShell 5 on Windows.

Readme.md https://github.com/HP-85/atp

AWS.Tools.Install.ps1 https://github.com/HP-85/atp/blob/main/AWS.Tools.Install.ps1

Install directly from repo:

Invoke-RestMethod -Uri "github.com/HP-85/atp/raw/main/AWS.Tools.Install.ps1" | Invoke-Expression

aws / aws-tools-for-powershell