search

ohsu-comp-bio / funnel

Funnel is a toolkit for distributed task execution via a simple, standard API.
https://ohsu-comp-bio.github.io/funnel
MIT License
121 stars 32 forks source link

Simplify deployment #40

Open jbingham opened 7 years ago

jbingham commented 7 years ago

It would be great to make deployment so easy that anyone can launch a Funnel service in < 5 min.

Perhaps Funnel could be packaged as a Docker image, with instructions on how to run it on your laptop or server.

On Google Cloud, the ideal might be to run as an AppEngine app, since most users would probably remain within the free tier. By contrast, running a VM just for Funnel could incur charges of $25/mo for a basic VM, which wouldn't scale as easily as AppEngine. With AppEngine, it's possible to write code that can work outside of AppEngine too.

buchanae commented 7 years ago

Totally agree. Ansible playbooks are another idea.

A docker image should be fairly straightforward. Mainly what we need there is a better build/release (automated) process.

Does AppEngine give us a persistent volume to write our database to? Or would we need a AppEngine database backend?

jbingham commented 7 years ago

For AppEngine, you'd typically either write to GCS or use Cloud SQL (MySQL or Postgres) or Datastore.

We've also looked into whether the service can be stateless apart from the queue -- ie, task status could be done with a file in the output directory and task history could be up to the user. The task IDs could be full GCS paths to the output, where you can find the logs and completion status.

Benefits: lighter weight, simpler implementation, lower cost, similar to SGE and traditional job schedulers which don't have persistent state for tasks that aren't in the queue or running Disadvantages: give up some niceness, especially if you want a UI to view completed stuff

buchanae commented 7 years ago

I haven't used Datastore, but I'm guessing it's the closest fit. The current database is BoltDB, which is a key/value store. We don't have a nice database interface yet, so that would be needed to start supporting multiple storage backends. That's something we've talked about, but hasn't been a priority.

A stateless server sounds cool, but also sounds like a different project than funnel. I'd prefer to put the emphasis on a great UI.

jbingham commented 7 years ago

Interesting. Is the main focus of Funnel to build a job manager UI that supports TES?

buchanae commented 7 years ago

The main focus is task execution infrastructure, but a nice UI is needed too. If necessary, I suppose the UI could be considered a separate service (that would probably need a database?).

If exploring a stateless server is important to you, shall we discuss it in a new, focused issue? There might be a solution that allows stateless mode to exist, or at least we can figure out whether it definitely doesn't fit.

jbingham commented 7 years ago

You're right -- I think this has grown into more than one issue.

Let's keep this thread about deployment. I'll move the UI and state discussion to a new thread.

buchanae commented 7 years ago

@jbingham For GCE, is there a recommended way for us to publicly share a Funnel release? Such as a public image family, or a snapshot, or a bucket, or something like cloud launcher?

App Engine is on the list, but I'm also interested in GCE.

buchanae commented 7 years ago

Added a PR with a Dockerfile in #108

Created a ohsucompbio/funnel repo on Docker Hub and pushed up a build from master.

A server could be started with:

docker run -p 8000:8000 -p 9090:9090 ohsucompbio/funnel server

But, because you need to properly route volumes and network ports to the docker containers, I've found that it's easier to try Funnel by downloading the binary from the website: https://ohsu-comp-bio.github.io/funnel/

Still, the docker container might be useful during deployment. I may experiment with deploying to GCE's container optimized image using the docker container.