search

MarkEdmondson1234 / serverless-R-API-appengine

Describes how to deploy an R API using Plumber onto App Engine and using Cloud Endpoints
MIT License
77 stars 7 forks source link

Scaling plumber instances to run long-running scripts on Kubernetes #3

Closed pedreen closed 5 years ago

pedreen commented 5 years ago

Hi @MarkEdmondson1234 , I have scripts that run mathematical models that usually take a long time to run. At first I used seven physical machines to split the work, but since R is single thread it takes a while, some of these codes take hours to execute. The solution I found was with Docker and Google Cloud Platform, and through its repository: https://github.com/MarkEdmondson1234/serverless-R-API-appengine, I was successful in creating my Dockers, but if I wanted to Scaling this horizontally I believe the best way is through Kubernetes, paralleling these containers and taking charge loads. I followed the walkthrough of this tutorial: https://code.markedmondson.me/r-on-kubernetes-serverless-shiny-r-apis-and-scheduled-scripts/, however at the time of setting the plumber in place of opencpu did not succeed. Here is the example I tried to upload using an endpoint that shows me a simple Sys.Date ().

API Kubernetes.zip

MarkEdmondson1234 commented 5 years ago

I need a reproduceable example to help, I can't walk through code on my own. Pretty much the only info I have is that plumber did not succeed. Help me help you :)

pedreen commented 5 years ago

Mark I saw that you added a tutorial with plumber and kubernetes in you page. I will try to replicate this example and next that I come to tell you if worked. Thanks!

pedreen commented 5 years ago

@MarkEdmondson1234 everything worked! I was able to access my endpoints. Thank you very much for your attention!

MarkEdmondson1234 commented 5 years ago

Great, glad it got sorted :)