Replace the batch connect inversion of control framework with a library of bash functions

[ericfranz]

• get rid of template library https://github.com/OSC/ood_core/blob/master/lib/ood_core/batch_connect/template.rb and related files
• have each batch connect app have the full single script; simplify the script enough by providing many well defined functions
• the batch connect app's main script is the FULL job script and can be a template whose context has access to all the common configuration that https://github.com/OSC/ood_core/blob/master/lib/ood_core/batch_connect/template.rb currently enables

By creating a large library of small well named bash functions that every interactive app has access to, we can do many things:

1. Dramatically simplify the shell script that starts the interactive app. Instead of using a templating system with a main script that sources other scripts at different parts of the execution, we just generate a single script that calls these library functions.
2. Enable other sites to more easily customize and debug problems with their interactive apps.
3. Reduce the amount of code we have to maintain.

Examples of what could be changed into functions is looking at the bc_desktop iHPC app:

1. All of the desktop scripts. For example https://github.com/OSC/bc_desktop/blob/master/template/desktops/xfce.sh could be refactored to 6 functions:
   1. remove_preconfigured_monitors
   2. copy_default_panel_if_doesnt_exist
   3. disable_startup_services
   4. run_xfce4_terminal_as_login_shell
   5. launch_xfce4_desktop, which does the echo statements (starting, ended) and calls functions 1-4 https://github.com/OSC/bc_desktop/blob/e70e27117977fe2327c642ae7da1b192e3572f06/template/script.sh.erb#L9-L12
2. export_module_function https://github.com/OSC/bc_desktop/blob/e70e27117977fe2327c642ae7da1b192e3572f06/template/before.sh.erb

A similar thing may be possible with https://github.com/OSC/ood_core/blob/d21a1d62a41b1a3e6d05a28cfe1100bb931597f9/lib/ood_core/batch_connect/template.rb and https://github.com/OSC/ood_core/blob/d21a1d62a41b1a3e6d05a28cfe1100bb931597f9/lib/ood_core/batch_connect/templates/vnc.rb

┆Issue is synchronized with this Asana task by Unito

[ericfranz]

In the desktop case, the before, the script, and the after could be handled together like this:

export_module_function 

(
  launch_xfce4_desktop
) &
SCRIPT_PID=$!

# after function calls

We could also put many of the functions in the library under test: https://github.com/sstephenson/bats

There would still be a need for "interactive app plugins" to provide their own library of functions (setting up R or Jupyter, for example).

But with this approach, instead of using the plugin files as values to insert into a template that ood_core owns, the reversal could occur: the plugin files would contain the full script, and it would just be calling functions from the bash library that ood_core provides. We could still have the plugin be a template, so passing in configuration as necessary.

This type of exploration requires inlining multiple interactive app plugin code with the ood_core batch connect library code so that its all together. Some thought is required for this.