search

hammerlab / cycledash

Variant Caller Analysis Dashboard and Data Management System
Other
36 stars 2 forks source link

Are we supporting only Hadoop clusters? #682

Open ihodes opened 9 years ago

ihodes commented 9 years ago

Doing so would

1) Reduce the number of people easily able to deploy Cycledash, BUT 2) Simplify our code, especially with 3) Our (inevitable?) move to using Impala 4) Building in greater support for using Spark, and other Hadoop tools

hammer commented 9 years ago

I think we should support more than Hadoop clusters. Hadoop has become quite good at hiding behind standard interfaces (e.g. NFS, ODBC). Let's do what we can to make CycleDash work with these interfaces rather than directly with Hadoop.

jaclynperrone commented 9 years ago

Thanks for adding this one!

ihodes commented 9 years ago

Other issues coming up/need to be addressed in making this decision.

  1. Serving BAMs from a RDBMS will be hard; Impala would work no problem.
  2. We'd like to run workers on the cluster to fix issues with workers in Celery failing on bigger files. We'd need to write workers to run on a non-Hadoop-style system as well if we want to support that.
  3. We use Postgres-specific SQL (\copy, some string processing, others) right now; we'd likely need to use Impala-specific SQL to use Impala (and Impala would have a data-loading setup); supporting both would be tedious.