As you can see 2nd request has different log_group_name compared with 1st request.
# 1st request
Show request data {:log_group_name=>"/aws/lambda/prefix-application", :log_stream_name_prefix=>"20"}
# 2nd request
Show request data {:log_group_name=>"/aws/lambda/prefix", :next_token=>"Kwfr0bfVKkYk4OrYazFRiVorMi4af2evRctbOLW6c5RNyVDQcDvPglPXsnYYrtXo3GcFfLfJLaU45Sc4QjHOsph1n4h5aqrhjMFpWH9ZejG5VLgfeAHAwN-Ao-diSx-qHIV2ZnZhJPuOn4h6gQ0nxeEYJql2XBpBlqodRyE2OeI0np_pq2klv69FA2QxBJPHtUTUeXRGsVpWeFiK8dVMeVUNlOUyn6nmKc8O1G9TKlE1rkUIpa6m2tzlVNUs1BI7TOGYGYwBCmo_JQk3I9x3p11RUCCwQw0XbJAcdds_lAFy-6JTbVT-5sMg654ipoySDlFckRMwd67Q0Y-hWdcGniJF_1r6atGj0MtXe3FQgcQijeqOuhceRUpuYc7mVu4iVabaVQbeeGTojEkskvowCfXTHzFAkdFO3e2owZuikbAHEw61UJhNgTYPNFFfbSJw4MW-uHn_H4kWPugZYdQkCmPJvENsGg9W12B5HtOsA8M", :log_stream_name_prefix=>"20"}
2021-01-14 07:21:52 +0000 [warn]: thread exited by unexpected error plugin=Fluent::Plugin::CloudwatchLogsInput title=:in_cloudwatch_logs_runner error_class=Aws::CloudWatchLogs::Errors::InvalidParameterException error="Pagination token in request is invalid. Only token from previous list call or null is valid."
#<Thread:0x0000561204f25b40@in_cloudwatch_logs_runner /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/plugin_helper/thread.rb:70 run> terminated with exception (report_on_exception is true):
/usr/local/bundle/gems/aws-sdk-core-3.111.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Pagination token in request is invalid. Only token from previous list call or null is valid. (Aws::CloudWatchLogs::Errors::InvalidParameterException)
from /usr/local/bundle/gems/aws-sdk-core-3.111.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
from /usr/local/bundle/gems/aws-sdk-core-3.111.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
from /usr/local/bundle/gems/aws-sdk-core-3.111.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
from /usr/local/bundle/gems/aws-sdk-core-3.111.0/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
from /usr/local/bundle/gems/aws-sdk-core-3.111.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
from /usr/local/bundle/gems/aws-sdk-core-3.111.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
from /usr/local/bundle/gems/aws-sdk-core-3.111.0/lib/seahorse/client/request.rb:72:in `send_request'
from /usr/local/bundle/gems/aws-sdk-cloudwatchlogs-1.38.0/lib/aws-sdk-cloudwatchlogs/client.rb:1071:in `describe_log_streams'
from /usr/local/bundle/gems/fluent-plugin-cloudwatch-logs-0.13.0/lib/fluent/plugin/in_cloudwatch_logs.rb:295:in `block in describe_log_streams'
from /usr/local/bundle/gems/fluent-plugin-cloudwatch-logs-0.13.0/lib/fluent/plugin/in_cloudwatch_logs.rb:310:in `throttling_handler'
from /usr/local/bundle/gems/fluent-plugin-cloudwatch-logs-0.13.0/lib/fluent/plugin/in_cloudwatch_logs.rb:288:in `describe_log_streams'
from /usr/local/bundle/gems/fluent-plugin-cloudwatch-logs-0.13.0/lib/fluent/plugin/in_cloudwatch_logs.rb:303:in `block in describe_log_streams'
from /usr/local/bundle/gems/fluent-plugin-cloudwatch-logs-0.13.0/lib/fluent/plugin/in_cloudwatch_logs.rb:310:in `throttling_handler'
from /usr/local/bundle/gems/fluent-plugin-cloudwatch-logs-0.13.0/lib/fluent/plugin/in_cloudwatch_logs.rb:288:in `describe_log_streams'
from /usr/local/bundle/gems/fluent-plugin-cloudwatch-logs-0.13.0/lib/fluent/plugin/in_cloudwatch_logs.rb:189:in `block in run'
from /usr/local/bundle/gems/fluent-plugin-cloudwatch-logs-0.13.0/lib/fluent/plugin/in_cloudwatch_logs.rb:185:in `each'
from /usr/local/bundle/gems/fluent-plugin-cloudwatch-logs-0.13.0/lib/fluent/plugin/in_cloudwatch_logs.rb:185:in `run'
from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
So this PR is to paste log_group_name in 2nd request.
Here is fluentd config
I guess because of response data is reached to
max_results
https://docs.aws.amazon.com/sdk-for-ruby/v2/api/Aws/CloudWatchLogs/Client.html soPagination token
is created for 2nd request.As you can see 2nd request has different log_group_name compared with 1st request.
So this PR is to paste log_group_name in 2nd request.