search

ibmcb / cbtool

Cloud Rapid Experimentation and Analysis Toolkit
Apache License 2.0
77 stars 49 forks source link

Refresh the Spark workload (and other repairs) #436

Closed mraygalaxy closed 1 month ago

mraygalaxy commented 1 month ago

  1. The Spark workload was pretty old. spark-bench is no longer maintained. I didn't remove it, but I focused on GATK instead. We updated all the software as well as GATK and fixed all of the datasets that were required.

  2. We introduced a new "medium" sized GATK workload (18GB) versus the small (70MB) vs. the large (150GB). This new medium size is a nice sweet spot.

  3. We modified Spark to be better vertically scalable by make the worker node settings grow with the size of the VMs.

  4. There were still python2 locations still being used in various places, so we fixed that.

  5. The base container was still not updated to Ubuntu22... fixed that.

  6. The linode driver had a bug where the VM hostnames were not being set. Fixed that.

  7. The python-daemon package had to be updated as the Orchestrator Dockerfile was not working...fixed that too.

  8. We also updated the Spark workload to use LOAD_LEVEL, which allows you to run multiple jobs simultaneously while also calculating the resources required to do so (horizontal scalability).

mraygalaxy commented 1 month ago

@maugustosilva Can you take a look?

maugustosilva commented 1 month ago

A very much needed update and fixup.

mraygalaxy commented 1 month ago

Thank you!