cloudyr / aws.s3

Amazon Simple Storage Service (S3) API Client
https://cloud.r-project.org/package=aws.s3
381 stars 148 forks source link

Use NextMarker instead of tail() to find NextMarker #414

Open cboettig opened 2 years ago

cboettig commented 2 years ago

Previously, function erroneously assumed that XML response always had <Content> elements listed last, and merely used tail() to get the Key of the last element for use as NextMarker.

This assumption relying on ordering rather than explicitly using element names fails when clients (Redhat's CEPH S3 interface, an open source product widely used in research data centers) return XML that lists metadata fields. Moreover, the S3 response actually provides a field called NextMarker for this very purpose, which ought to be used instead of attempting to find the last key. My pull request simply updates the code to use r$NextMarker to find the NextMarker instead of using tail(r, 1)[["Contents"]][["Key"]])

Please ensure the following before submitting a PR: