search

aws / aws-tools-for-powershell

The AWS Tools for PowerShell lets developers and administrators manage their AWS services from the PowerShell scripting environment.
Apache License 2.0
235 stars 78 forks source link

Autopagination not working #147

Closed lgaudreau closed 3 years ago

lgaudreau commented 4 years ago

While running a command like 'get-ec2snapshot' in a region where there are more than 1000 snapshots, the command fails with

Get-EC2Snapshot: You have exceeded the output limit for an unpaginated response. Retry your request using pagination parameters.

Running with -maxresults returns a warning that the command is deprecated, but the solution presented (to pipe the results to Select-Object -First ##) also fails since it requires returning all objects first.

This was not happening in AWSPowerShell 3.3.563.1 at least.

matteo-prosperi commented 4 years ago

I think this can be fixed by setting a default value of 1000 for -MaxResult. I will mark this as a bug.

I believe previous versions of the module are similarly affected, they would just return a partial output without complaining. You should be able to restore this behavior by passing -MaxResult 1000 and ignoring the warning.

Information for the implementer of the fix: AWS.Tools will have a different behavior and should be tested as well.

lgaudreau commented 4 years ago

I've installed AWS.Tools version 4.0.5.0 and have run into the following behaviour:

With 15,000 snapshots in the account, running 'Get-EC2Snapshot -Owner self' returns the same error: Get-EC2Snapshot: You have exceeded the output limit for an unpaginated response. Retry your request using pagination parameters. Using -MaxResult 1000 returns the first 1000, but setting -NextToken is not working as expected either - the second run only returns 467 items, not another 1000.

ashishdhingra commented 3 years ago

Hi @lgaudreau,

Good afternoon.

I was going through the issue backlog and came across the issue. The -NextToken appears to work fine for me:

Get-EC2Snapshot -MaxResult 10 -NextToken $null 
Get-EC2Snapshot -MaxResult 10 -NextToken $AWSHistory.LastServiceResponse.NextToken

Output of Get-InstalledModule | Select Version,Name

Version Name
------- ----
4.1.2.0 AWS.Tools.EC2
4.1.2.0 AWS.Tools.IdentityManagement
4.1.2.0 AWS.Tools.Glacier
4.1.2.0 AWS.Tools.S3
4.1.2.0 AWS.Tools.SQS
4.1.2.0 AWS.Tools.Common
4.1.2.0 AWS.Tools.SimpleNotificationService
4.1.2.0 AWSPowerShell
1.0.2.0 AWS.Tools.Installer
2.0.0.0 AWSLambdaPSCore
4.1.2.0 AWSPowerShell.NetCore

Output of Get-AWSPowerShellVersion

AWS Tools for Windows PowerShell
Version 3.3.119.0
Copyright 2012-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Amazon Web Services SDK for .NET
Core Runtime Version 3.3.17.0
Copyright 2009-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Release notes: https://aws.amazon.com/releasenotes/PowerShell

This software includes third party software subject to the following copyrights:
- Logging from log4net, Apache License
[http://logging.apache.org/log4net/license.html]

Please try with the latest version of Powershell Tools/Modules and confirm if the issue is resolved.

Thanks, Ashish

lgaudreau commented 3 years ago

Hi Ashish,

Using -NextToken works correctly as a workaround. Auto-pagination is still not working using the latest version:

Output of Get-EC2Snapshot -Owner self Get-EC2Snapshot: You have exceeded the output limit for an unpaginated response. Retry your request using pagination parameters.

Output of Get-InstalledModule Select Version,Name Version Name

4.1.2.0 AWS.Tools.Common 4.1.2.0 AWS.Tools.EC2 1.0.2.0 AWS.Tools.Installer 4.1.2.0 AWS.Tools.RDS 4.1.2.0 AWS.Tools.Route53 4.1.2.0 AWS.Tools.S3 4.1.2.0 AWS.Tools.WAFV2 4.1.2.0 AWS.Tools.WorkDocs 4.1.2.0 AWS.Tools.WorkSpaces

Output of Get-AWSPowerShellVersion AWS Tools for PowerShell Version 4.1.2.0 Copyright 2012-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Amazon Web Services SDK for .NET Core Runtime Version 3.5.1.23 Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

Release notes: https://github.com/aws/aws-tools-for-powershell/blob/master/CHANGELOG.md

This software includes third party software subject to the following copyrights:

Thanks,

Lincoln

From: Ashish Dhingra notifications@github.com Sent: Wednesday, October 14, 2020 3:26 PM To: aws/aws-tools-for-powershell aws-tools-for-powershell@noreply.github.com Cc: lgaudreau lincolng@outlook.com; Mention mention@noreply.github.com Subject: Re: [aws/aws-tools-for-powershell] Autopagination not working (#147)

Hi @lgaudreauhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flgaudreau&data=02%7C01%7C%7Cfecf12128b4b43c4678608d8707708c7%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637383003827386214&sdata=ZbCOpl1JTjWo7oKZJ9eVZVBClJshSSwkghjvgVZA%2By0%3D&reserved=0,

Good afternoon.

I was going through the issue backlog and came across the issue. The -NextToken appears to work fine for me:

Get-EC2Snapshot -MaxResult 10 -NextToken $null Get-EC2Snapshot -MaxResult 10 -NextToken $AWSHistory.LastServiceResponse.NextToken

Output of Get-InstalledModule | Select Version,Name

Version Name

4.1.2.0 AWS.Tools.EC2 4.1.2.0 AWS.Tools.IdentityManagement 4.1.2.0 AWS.Tools.Glacier 4.1.2.0 AWS.Tools.S3 4.1.2.0 AWS.Tools.SQS 4.1.2.0 AWS.Tools.Common 4.1.2.0 AWS.Tools.SimpleNotificationService 4.1.2.0 AWSPowerShell 1.0.2.0 AWS.Tools.Installer 2.0.0.0 AWSLambdaPSCore 4.1.2.0 AWSPowerShell.NetCore

Output of Get-AWSPowerShellVersion

AWS Tools for Windows PowerShell Version 3.3.119.0 Copyright 2012-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Amazon Web Services SDK for .NET Core Runtime Version 3.3.17.0 Copyright 2009-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Release notes: https://aws.amazon.com/releasenotes/PowerShell

This software includes third party software subject to the following copyrights:

Please try with the latest version of Powershell Tools/Modules and confirm if the issue is resolved.

Thanks, Ashish

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faws%2Faws-tools-for-powershell%2Fissues%2F147%23issuecomment-708612084&data=02%7C01%7C%7Cfecf12128b4b43c4678608d8707708c7%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637383003827386214&sdata=fyLICnHhdIA1hrSDt4s%2FaTgyTk98gjavIGSiSf5HDPo%3D&reserved=0, or unsubscribehttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKWFK6E27FHX6VAFBYZCELDSKX3N3ANCNFSM4MJ6IN2Q&data=02%7C01%7C%7Cfecf12128b4b43c4678608d8707708c7%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637383003827386214&sdata=cfPs%2Bzpvt6JvFqO6bknqgGdtbzlZgq0M7af3nPCX1tU%3D&reserved=0.

ashishdhingra commented 3 years ago

Hi @lgaudreau,

Thanks for your reply. I think we already have a feature request https://github.com/aws/aws-tools-for-powershell/issues/117 to support auto-pagination for AWS.Tools.*. Please confirm if we close this issue in favor of the other feature request.

Thanks, Ashish

lgaudreau commented 3 years ago

Thank Ashish,

That's fine, yes! This issue can be closed with the feature request open.

Lincoln

From: Ashish Dhingra notifications@github.com Sent: Thursday, October 15, 2020 1:52 PM To: aws/aws-tools-for-powershell aws-tools-for-powershell@noreply.github.com Cc: lgaudreau lincolng@outlook.com; Mention mention@noreply.github.com Subject: Re: [aws/aws-tools-for-powershell] Autopagination not working (#147)

Hi @lgaudreauhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flgaudreau&data=04%7C01%7C%7Cf6849f35d2d5412a527408d87133168f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637383811511355523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bftexKH5oql6a05qSyVfynJqpi6y8LyVxY32omiG6sM%3D&reserved=0,

Thanks for your reply. I think we already have a feature request #117https://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faws%2Faws-tools-for-powershell%2Fissues%2F117&data=04%7C01%7C%7Cf6849f35d2d5412a527408d87133168f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637383811511355523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tyaXMOhfIm4uteWE%2BJWZt0FLG0ZPAho8TYfwMpjopeQ%3D&reserved=0 to support auto-pagination for AWS.Tools.*. Please confirm if we close this issue in favor of the other feature request.

Thanks, Ashish

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faws%2Faws-tools-for-powershell%2Fissues%2F147%23issuecomment-709490523&data=04%7C01%7C%7Cf6849f35d2d5412a527408d87133168f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637383811511365517%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rBdqRtb9XZnXcT0B5MSBEaOWAXicSF%2BvesNez7zBc10%3D&reserved=0, or unsubscribehttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKWFK6A5YHZ5XEW7FDWH2J3SK4ZF5ANCNFSM4MJ6IN2Q&data=04%7C01%7C%7Cf6849f35d2d5412a527408d87133168f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637383811511365517%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Y8aet2vDeRFmPnHoe%2BD7wkjKK9JFSWQ6EdCcxxYGuyk%3D&reserved=0.