AWSSDK.S3.list_objects_v2 continuation token does not paginate

[tclements]

AWSSDK.S3.list_objects_v2 cannot use continuation-token when making paginating requests. This seems similar to #10

Relevant package versions

(@v1.4) pkg> st
Status `~/.julia/environments/v1.4/Project.toml`
  [4f1ea46c] AWSCore v0.6.14
  [1c724243] AWSS3 v0.6.12
  [0d499d91] AWSSDK v0.5.0

MWE using AWSSDK.S3.list_objects_v2

using AWSCore, AWSSDK.S3
aws = aws_config(region="us-east-1")
bucket = "XXXX"
prefix = "XXX/XX=XXX/XXX=XXX/XX=XXXX"
s3req = AWSSDK.S3.list_objects_v2( aws,Bucket=bucket,prefix=prefix)
# returns 1000 keys 

# next request try with continuation 
s3req = AWSSDK.S3.list_objects_v2(
    aws,
    Bucket=bucket,
    prefix=prefix,
    continuation-token=string(s3req["NextContinuationToken"]),
)
ERROR: syntax: invalid keyword argument name "(continuation - token)"
Stacktrace:
 [1] top-level scope at REPL[469]:1

which is not a valid keyword in Julia. Trying the what was suggested in #10 with headers

# next request
d = ["prefix"=>prefix,"Bucket"=>bucket,"headers"=>["continuation-token"=>string(s3req["NextContinuationToken"])]]
s3req = AWSSDK.S3.list_objects_v2(aws,d)

just returns the first 1000 keys of the original request. I've tried every combination of ContinuationToken, Continuation_Token, continuation_token, etc.. to get pagination working.

Please let me know if I am missing something obvious in using the continuation-token option.

[mattBrzezinski]

Haha, I believe I had just stumbled upon this while re-working on my unification of AWSCore.jl and this package into one unified package AWS.jl.

As you mentioned Julia does not support hyphens in variable names, however AWS uses this style for arguments. What we need to do is something like this here.

Basically pass in these arguments which AWS expects in a Julia-friendly format, then convert the query string before making the request. In my example above I've default to using underscores (continuation_token), since that seems both the easiest and most appropriate.

Tomorrow I can make a patch fix (unless you'd like to yourself), add some tests based off of your MWE and release the new version!

[tclements]

Awesome, thanks for the quick response!

[mattBrzezinski]

@tclements a work around to unblock yourself would be to do the below. I will still be working on a patch / discussing as I am not a fan of this styling.

s3req = AWSSDK.S3.list_objects_v2(
    aws,
    Bucket=bucket,
    prefix=prefix,
    var"continuation-token"=string(s3req["NextContinuationToken"]),
)

[tclements]

@mattBrzezinski thanks! var trick works for the moment. Looking forward to AWS.jl!