search

DyfanJones / noctua

Connect R to Athena using paws SDK (DBI Interface)
https://dyfanjones.github.io/noctua/
Other
46 stars 5 forks source link

Speeding up `noctua` installation by cutting down unnecessary pieces #217

Open emmansh opened 7 months ago

emmansh commented 7 months ago

Is there a way to install just a subset of necessary parts/dependencies of the noctua library to support only:

con <- DBI::dbConnect(noctua::athena(), 
                      region = "us-east-1",
                      s3_staging_dir = "foo",
                      schema_name = "bar",
                      work_group = "baz"
                      )

Asking because my use-case is installing noctua package on AWS SageMaker jupyter notebook (with R kernel), which takes approx. 25 minutes of installation for just noctua. I believe the most time-consuming part is the paws dependency that has to be built. What's more, the installation on AWS SageMaker is not persistent, so each time I initialize the notebook instance I need to re-install.
This practically means that every day I need to waste 25 minutes before I can start doing actual work. I was wondering if possible to pick a minimal subset of paws or other dependencies to support just the piece of code above.

Worth noting that other packages such as tidyverse ones are installing in a matter of seconds - few minutes.

DyfanJones commented 7 months ago

This should speed up the installation of noctua.

install.packages('noctua', repos = c(pawsr = 'https://paws-r-builds.s3.amazonaws.com/packages/0.3.0/', CRAN = 'https://cloud.r-project.org'))

In short it will grab the latest binaries of paws from paws's Aws S3 bucket and then install the rest from the cran :)

DyfanJones commented 7 months ago

Let me know how this works for you :)

emmansh commented 7 months ago

Wow that was a quick response. Thanks will let you know!