Test: Map build (HDFS, Spark, HBase, ZK)

[timrobertson100]

This test will evaluate the use of Stackable to run the GBIF map build job.

A custom Spark job will process coordinates found in the GBIF occurrence data into views stored as HFiles on HDFS that are suitable for subsequent bulk loading into HBase. A simple Java-based vector tile server sitting on HBase will serve map views to the browser.

It is envisaged that this test scenario will cover:

• A deployment of Kubernetes on 4 bare metal machines, each with 12+ local drives (leftover old production Hadoop hardware)
• Set up of HDFS making use of those local drives (paying attention to data locality concerns)
  • A copy of a subsampled GBIF occurrence view will be stored on HDFS in an Avro file
• Set up Zookeeper and HBase
• Creating a partitioned, empty HBase table
• Modification of the GBIF Maps Spark code to process the Avro file
  • The existing version uses Hive, but this will be adapted to just read the Avro file as input
  • The Spark job will use Zookeeper to discover the table partitioning of HBase
  • The result is HFiles stored on HDFS
• A manual bulk load of the HFiles into HBase
• Running a vector tile server to serve up the map views

The same procedure should be runnable on a laptop.

Once complete we expect to have gained an understanding of the configuration in Stackable and what is required to adapt our code to the later versions of HDFS, Spark, and HBase (Edited to add: we will update our code to run on the latest versions of everything).

[fmendezh]

be aware that Stackable supports Spark 3.2.1 and 3.3.0 only, the code we have works for 2.3.0 https://docs.stackable.tech/spark-k8s/stable/index.html

[zaultooz]

Commit: https://github.com/gbif/stackable/commit/184be0338bb51cf3f204ee1620d05ea6d46a50d5 contains the generic charts used for creating the cluster and deploying: HDFS, zookeeper, Hbase, Spark application and Vectortile server.

Env specific information has been left out.