pachadotdev / analogsea

Digital Ocean R client
https://pacha.dev/analogsea/
Apache License 2.0
154 stars 24 forks source link

Sanity Check: Simplest way to spin up a `future` cluster #167

Closed CerebralMastication closed 3 years ago

CerebralMastication commented 6 years ago

This is less an 'issue' and more a 'code review' sort of discussion. I'm working with O'Reilly & Paul Teeter to update the R Cookbook for a 2nd edition. One recipe I'd really like to have is "execute code in parallel in the cloud". It seems to me that future with furrr is a really good fit as it makes the purrr approach parallel.

Towards that end I'm trying to figure out the most drop dead simple recipe for spinning up a cluster for use with future. Thanks to some good ideas from Andrew Heiss I have cooked up the following:

library(analogsea)
library(tidyverse)
library(furrr)

cluster_tag <- 'r_cluster' 
cluster_prefix <- 'node-'
number_of_nodes <- 4

my_droplets <- docklets_create(names = paste0(cluster_prefix, as.character(1:number_of_nodes)),
                               region = "sfo2",
                               size = "1gb",
                               tags = cluster_tag)

# pull the ip addresses for the droplets
ips <- droplets(tag=cluster_tag) %>%
  map(function(x) x$networks$v4[[1]]$ip_address) %>%
  unlist

# Path to private SSH key that matches key uploaded to DigitalOcean
# it looks like ~ does not work here... not sure why
ssh_private_key_file <- "/Users/jal/.ssh/id_rsa"

# Connect and create a cluster
cl <- makeClusterPSOCK(
  # vector of IPs in our cluster
  workers = ips,

  # DigitalOcean droplets use root for user
  user = "root",

  # use the key connected to Digital Ocean
  rshopts = c(
    "-o", "StrictHostKeyChecking=no",
    "-o", "IdentitiesOnly=yes",
    "-i", ssh_private_key_file
  ),

  # run Rscript in the tidyverse docker
  rscript = c("sudo","docker","run","--net=host","rocker/tidyverse","Rscript"),

    rscript_args = c(
    # Install furrr (future too) 
    "-e", shQuote("install.packages('furrr')")
  ),
  homogeneous=FALSE, 
  verbose=TRUE
)

My question for you all is simply: Can I make this more simple? This work. It makes a parallel backend. I can explain each step. But can it be even more simple?

Thanks!

sckott commented 5 years ago

@CerebralMastication crap, so this must be the longest i've gone w/o replying to an issue. many apologies.

I can't think of anything to make simpler.

Is this still the up to date version of your thinking on this?

CerebralMastication commented 5 years ago

I think so...

sckott commented 5 years ago

cool, anything else to discuss on this ?