search

aws / aws-tools-for-powershell

The AWS Tools for PowerShell lets developers and administrators manage their AWS services from the PowerShell scripting environment.
Apache License 2.0
238 stars 79 forks source link

Support automated pagination for operations returning multiple lists. #30

Closed recumbent closed 4 years ago

recumbent commented 5 years ago

If I make a call to the aws cli along the lines of the following:

aws logs filter-log-events --log-group-name '/my/log/group' --start-time "$(date -d "-1 day" +%s)000" --filter-pattern '{ $.level != INFO }'

Then I see all the messages matching the filter for the last 24 hours (for my typical use case about half a dozen).

If I make an equivalent call using the tools for powershell

Get-CWLFilteredLogEvent -LogGroupName "/my/log/group" -StartTime $filterStart -Filter "{ $.level != INFO }" (where $filterStart is an equivalent start time to the above).

I see rather fewer messages. I also get a continuation token (took me way too long to work that out...)

Expected Behavior

The behavior should be consistent across the CLI and the tools for Powershell - so in this case powershell should return all of the filtered messages for the time period in the same way that the CLI does

Current Behavior

Powershell returns a NextToken where the CLI does not and lists 4 searched log streams (per call) where the CLI lists 10 in a single result.

Context

Firstly it seems to me that the purpose of using "filtered" is to be able to more trivially find the interesting data from larger list of log messages. If I need to start worrying about NextTokens for what is ultimately 7kb of data then it ceases to be trivial.

Secondly I'm far more comfortable with Powershell than Bash (and more able to further play with the output), but need to be able to work on either - to be able to switch seamlessly between the two is helpful. If the behavior of the two toolkits is inconsistent then its much harder to do so.

Your Environment Include as many relevant details about the environment where the bug was discovered.

matteo-prosperi commented 5 years ago

Hello, thank you for reaching out about this. Automated pagination is not enabled for FilterLogEvents because the operation returns multiple lists. This is unfortunately not a supported case for automated pagination. We will keep this use case in mind for future improvements.

recumbent commented 5 years ago

@matteo-prosperi thank you for responding.

So my understanding is clear - the challenge is that the underlying API returns multiple lists - specifically in this case a list of events and a list of searched log streams - and as a consequence automated paging is not (currently) supported.

This is the reason for the difference between the behavior of the CLI and the Tools for Powershell.

Makes sense.

matteo-prosperi commented 5 years ago

@recumbent You can have a look here for documentation about our current pagination approach https://aws.amazon.com/blogs/developer/output-pagination-with-aws-tools-for-powershell/.

I am also looking into improvements to address this issue but I am not able to commit to anything yet.

matteo-prosperi commented 4 years ago

Hello, thanks for opening this issue. We changed the pagination algorithm to be more robust and flexible in the AWS.Tools modules starting with today's release (3.3.618). You can find more details in https://github.com/aws/aws-tools-for-powershell/issues/61.

At this time, we are not considering to port these changes to the AWSPowerShell and AWSPowerShell.NetCore modules due to backward compatibility concerns.