search

MarkEdmondson1234 / markedmondson.me-hugo

Files for the website
0 stars 2 forks source link

r-at-scale-on-google-cloud-platform/ #10

Open utterances-bot opened 3 years ago

utterances-bot commented 3 years ago

R at scale on the Google Cloud Platform · Mark Edmondson

Current thinking on what I consider the optimal way to work with R on Google Cloud Platform

https://code.markedmondson.me/r-at-scale-on-google-cloud-platform/

nathhenry3 commented 3 years ago

This is very useful, thank you!

I am currently building a Shiny app with the shinyMobile package, in which users can enter individual data values that are then stored in a Postgres database in the cloud. This data is then used to plot graphs which are unique to each user. Is this a use case that could be covered using Docker + Cloud Run? I.e. would I still run into issues with websockets etc to enable stateful running of the shiny application?

MarkEdmondson1234 commented 3 years ago

Hi @nathhenry3 - I think that's very doable since I guess the traffic to each app is relatively low. The connections from the server.R are unaffected and operate as normal. The websocket stuff is only about how the ui.R communicates to server.R and that issue is solved.

awaterpolo commented 2 years ago

Hi Mark, thanks so much for your useful tutorials.

I'm attempting to set up a cluster of VMs using the googleComputeEngineR package. I'm using this code which works fine (I have set built a custom image that is stored on our company's project repository).

my_docker <- gce_tag_container("custom-image", project = "my-project")

vms <- gce_vm_cluster(vm_prefix = "rrr-cluster-",cluster_size = 3, ssh_args = list("/home/directory/file.pub", "/home/directory") , docker_image=my_docker)

The docker image loads successfully to the initial vms, however, when I attempt to use the custom image in the cluster with this code:

plan(cluster, workers = as.cluster(vms, docker_image = my_docker))

it only works if the container registry image folder is set to public in the google container registry.

Otherwise I get this message:

Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication. See 'docker run --help'.

It appears that I need to add some type of authentication. I'm a little lost though as to how to implement this into the plan function. Any resources you can point me to (i looked at the advanced-authentication website obviously but I'm very new to docker so I don't really know to to decipher it).

Thanks!

MarkEdmondson1234 commented 2 years ago

Hi @awaterpolo - Hmm good question, I would say its either not supported yet or you need to pass args that will be passed into gce_vm_template(). The reason its not been an issue usually for me is that the Docker image installs packages only and not private info, and the R scripts used are the ones that load in any sensitive data (such as a googleCloudStorageR call at the start of the script)