PAOLT
commented
5 years ago Different approaches can be considered depending on the use case:
Different workloads might be generated by different user bases, as far as their reference use case changes:
- Researchers tend to look at population wide patterns, so they need to look at data from many individuals in different categories
- Clinical testers tend to look at a sample for a single individual mostly.
Based on the above differentiation it is important to understand at what extent those categories require the same tooling (HBase).
Provided that researchers need HBase at least for a part of their job, their HBase concerned use case can be supported by the same HBase cluster used for clinical testing?
In case it can:
- Scale-out the main cluster (same tables)
- Build and feed dedicated tables with their own resources (application driven)
In case it cannot a new cluster should be provisioned, and the following aspects should be evaluated:
- the overall ingestion process generates a data lake or is it going to feed HBase directly? In case a data lake exists, both clusters can source raw data from the same data lake.
- Is the researcher cluster good to start from raw data, or the clinical cluster performs some data processing tasks that are useful for researcher as well?
- In case the clinical data performed good work for researchers, then the most interesting process seems to be: Export data from clinical HBase and compress Move data to the researcher cluster Import data into the researcher cluster
This process has the benefit of:
- Research HBase import ready-to-go, compressed data (vs. re-indexing raw data)
- The Export tool supports filters, so multiple files can be moved in parallel. It is not clear if it can align exported files to table partitions (vs. ingesting application logic)
- The data movement can be achieved with ADF vs Copy Activity that shows good base performance https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance. From published metrics it is clear that moving 2PB is challenging and good performance tuning / parallelization should be evaluated.
- Alternatives could be based on Snapshots (looks less flexible) or Replication (that usually supports different scenarios – i.e. DR: is this a conversation at all?)
It is highly likely that researchers will need a different toolset – i.e. Spark or equivalent analytic platforms. Azure analytics platforms (i.e. HDInsight, Azure DataBricks, Azure Data Lake, Azure Datawarehouse) can share the same data lake (if any) with HBase and have good ways to source data from there.
- In case (again) HBase performs good work for researchers, or a data lake doen't exists, the same considerations as above apply (export)
- In case the above point don't hold, the given an analytics platform should source data from the data lake raw files and build its own analytic indexes
martinpeck
lawrencegripper
wbari
Many clinical genomic projects have researchers associated to study all the genome data collected. In this scenario we can identify two main big use cases:
We need to investigate how we make the different big data snapshots available to researchers in an optimal way.
Option 1 - Replication OpenCGA implements export/import functionality. Data could be exported every 6 months and imported in a second installation in read-only mode. By doing this there are two copies of the data in different installations with different users and goals,
Option 2 - Only Spark OpenCGA can export data in Apache Parquet format to be analysed with Spark. OpenCB Oskar (https://github.com/opencb/oskar) implements a genomic analysis spark library and notebooks for researchers to analyse data. This option is easier to setup and there is less data replicated, but researcher will need to query the OpenCGA installation for some queries, there could be some problems between users.