paws-r / paws

Paws, a package for Amazon Web Services in R
https://www.paws-r-sdk.com
Other
315 stars 37 forks source link

Improve performance of restxml_unmarshal #762

Closed DyfanJones closed 6 months ago

DyfanJones commented 6 months ago

This PR has some significant performance enhancements for xml_unmarshal. This ultimately makes working with AWS S3 faster.

# Mock response output to avoid any discrepancy from internet connections
httr_env <- asNamespace("httr")
unlockBinding("VERB", httr_env)
unlockBinding("GET", httr_env)

# NOTE: `resp.RDS` is the response from AWS S3. 
# Currently stored locally to avoid internet connection discrepancies
httr_env$VERB <- function(...) {
  resp <- readRDS("resp.RDS")
  return(resp)
}
httr_env$GET <- function(...) {
  resp <- readRDS("resp.RDS")
  return(resp)
}

library(paws)
# set region to us-east-2 to avoid PermanentRedirect
client <- s3(config(credentials(anonymous = T), region = "us-east-2"))

bm <- bench::mark(
   aws_s3 = aws.s3::get_bucket(bucket, region = "us-east-2"),
   paws.common_v0.7.1 = client$list_objects_v2(bucket),
   # paws.common_v0.7.1.9000 = client$list_objects_v2(bucket),
   check = F,
   iterations = 100
)

# A tibble: 3 × 13
  expression                   min   median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc total_time result memory                 time             gc                
  <bch:expr>              <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl> <int> <dbl>   <bch:tm> <list> <list>                 <list>           <list>            
1 paws.common_v0.7.1.9000   20.5ms   22.2ms     42.3     1.42MB     16.4    72    28       1.7s <NULL> <Rprofmem [572 × 3]>   <bench_tm [100]> <tibble [100 × 3]>
2 aws_s3                   130.8ms  155.9ms      6.35    3.32MB     23.0   100   362     15.75s <NULL> <Rprofmem [6,190 × 3]> <bench_tm [100]> <tibble [100 × 3]>
3 paws.common_v0.7.1        38.4ms   51.8ms     18.4    23.82MB     13.3   100    72      5.43s <NULL> <Rprofmem [4,895 × 3]> <bench_tm [100]> <tibble [100 × 3]>

image

codecov[bot] commented 6 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 85.22%. Comparing base (83a5017) to head (485d453).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #762 +/- ## ========================================== - Coverage 85.24% 85.22% -0.03% ========================================== Files 213 213 Lines 15208 15145 -63 ========================================== - Hits 12964 12907 -57 + Misses 2244 2238 -6 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.