Currently batch-generate-edit-histograms.sh and batch-generate-footprints.sh have separate scripts from generate db backfill with uses batch-process.sh. With the addition of this more generic batch-process.sh we could just have separate json configurations for the instance/spark conf requirements for these processes and not maintain almost duplicate shell scripts.
It should be noted that as of writing, batch-process takes its instance type configuration via env var since the other scripts do as well. If this task is addressed, it likely makes sense to migrate this env var into a parameter on the batch-process.sh script.
Currently batch-generate-edit-histograms.sh and batch-generate-footprints.sh have separate scripts from generate db backfill with uses batch-process.sh. With the addition of this more generic batch-process.sh we could just have separate json configurations for the instance/spark conf requirements for these processes and not maintain almost duplicate shell scripts.
It should be noted that as of writing, batch-process takes its instance type configuration via env var since the other scripts do as well. If this task is addressed, it likely makes sense to migrate this env var into a parameter on the batch-process.sh script.