Some functions produce their pagination markers in the result

[lvh]

I'm assuming this is unintentional and possibly means pagination doesn't work for those fns right now. I've found:

• #'amazonica.aws.simplesystemsmanagement/list-documents (contains next-token)
• #'amazonica.aws.identitymanagement/list-policies (contains marker)

[lvh]

#'amazonica.aws.cloudformation/describe-stack-resources takes a :starting-token; I'm not sure it has working pagination.

[mcohen01]

i'm not sure i follow....should these bean properties (e.g. next-token) not be included in the return value?

[lvh]

I was under the (perhaps mistaken?) impression that amazonica handled pagination for me?

[mcohen01]

not that i'm aware of :)

[lvh]

I guess I was confused by the kinesis firehose thing :) Sorry!

Do you have any opinions on whether pagination should exist, and what it should look like?

[mcohen01]

Do you have any opinions on whether pagination should exist, and what it should look like?

Honestly, I have no recollection of ever seeing a notion of pagination in the api before. Does it exist in other services besides Kinesis Firehose?

[lvh]

Yes. The two in the ticket do that. Most describe- and list- operations I've seen are paginated. Pagination is a little tricky; you have to get the next token to figure out if you're done. You can't e.g. just check if the current page has less entries than the maximum; I'm told (names withheld to protect the guilty, but they work at AWS) some services will actually do that. Can happen because e.g. sharding.

[lvh]

(Example: describe-instances: http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html)

It only tends to do that for large amounts of items, so maybe I'm a little unusual in that I care about that :)

[mcohen01]

You are most certainly correct!

$ grep -nro "public String getNextToken" aws-sdk-java | grep Result\.java | wc -l 307

Hmm....well, i think the nicest api would be to simply let the user specify max-results greater than 1000 (cursory glance at the java api suggests 1000 is the page limit), and transparently issue further requests until no more next-tokens. Not sure how thorny it would be to implement that though....

[liath]

Just in case it's helpful to anyone else coming here: I found a relevant snippit for walking results that provide isTruncated/contMarker pagination: https://github.com/apeckham/s3-index-clj/blob/e039050f629d69207991600ad9c8f5c57b900c58/src/s3_index/core.clj#L6

[blischalk]

Tokens for pagination in list-resource-record-sets of the route53 api:

http://docs.aws.amazon.com/cli/latest/reference/route53/list-resource-record-sets.html

do not seem to be supported by Amazonica. Is this accurate? It seems that a NextToken should be present in the results and a :starting-token parameter accepted for the subsequent request.

Passing :no-paginate as mentioned in the documentation as a way to disable pagination also seems to be un-supported currently.

[mcohen01]

Amazonica reflects on the Java SDK. The CLI most likely takes different parameters and returns different results. Is there a :next-record-identifier key in the result from (list-resource-record-sets)? If so, pass that as a :start-record-identifier key in the next call.

[blischalk]

Negative. It does not. I receive the following:

(keys (r53/list-resource-record-sets :hosted-zone-id "TheIdGoesHere"))
(:max-items :is-truncated :resource-record-sets :next-record-name :next-record-type)

Passing :next-record-name and :next-record-type to subsequent calls also do not seem to trigger pagination to work. Neither does passing :next-record-name as the :start-record-identifier or next-record-identifier.

[mcohen01]

What about passing :next-record-name as :start-record-name?

[blischalk]

That seems to do the trick. Thanks!