emnd from master fails with grlsd on ARCHER

[ibethune]

I started commenting on #219, but I find enough issues that it is worth starting a new issue, rather than confusing the issue with Elena's problems (since I don't have the same symptoms).

I tested this using the latest ensemblemd from the master branch, and RP release:

(extasy-test)mbp-ib:grlsd-on-archer ibethune$ radicalpilot-version 
0.37.10
(extasy-test)mbp-ib:grlsd-on-archer ibethune$ ensemblemd-version 
0.3.6-80-gc82ad5f
(extasy-test2)mbp-ib:grlsd-on-archer ibethune$ python extasy_gromacs_lsdmap.py --RPconfig archer.rcfg --Kconfig gromacslsdmap.wcfg

================================================================================
 EnsembleMD (0.3.6-80-gc82ad5f)                                                 
================================================================================

Starting Allocation                                                           ok
        Verifying pattern                                                     ok
        Starting pattern execution                                            ok
--------------------------------------------------------------------------------
Executing simulation-analysis loop with 1 iterations on 24 allocated core(s) on 'epsrc.archer'

Job waiting on queue...
Job is now running !
Waiting for pre_loop step to complete.                                      done
Iteration 1: Waiting for simulation tasks: md.gromacs to complete           done
Iteration 1: Waiting for analysis tasks: md.pre_lsdmap to complete2015-12-17 11:56:32,399: radical.enmd.simulation_analysis_loop.static.default: MainProcess                     : Thread-3       : ERROR   : ComputeUnit error: STDERR: gromacs(3):ERROR:105: Unable to locate a modulefile for 'gromacs/5.0.0'
sh: trjconv: command not found
, STDOUT: gromacs(3):ERROR:105: Unable to locate a modulefile for 'gromacs/5.0.0'
sh: trjconv: command not found

2015-12-17 11:56:32,400: radical.enmd.simulation_analysis_loop.static.default: MainProcess                     : Thread-3       : ERROR   : Pattern execution FAILED.
2015-12-17 11:56:32,400: radical.pilot       : MainProcess                     : Thread-3       : ERROR   : unit manager controller thread caught system exit -- forcing application shutdown
Traceback (most recent call last):
  File "/Users/ibethune/Desktop/extasy-test2/lib/python2.7/site-packages/radical/pilot/controller/unit_manager_controller.py", line 261, in run
    self.call_unit_state_callbacks(unit_id, new_state)
  File "/Users/ibethune/Desktop/extasy-test2/lib/python2.7/site-packages/radical/pilot/controller/unit_manager_controller.py", line 198, in call_unit_state_callbacks
    cb(self._shared_data[unit_id]['facade_object'], new_state)
  File "/Users/ibethune/Desktop/extasy-test2/lib/python2.7/site-packages/radical/ensemblemd/exec_plugins/simulation_analysis_loop/static.py", line 141, in unit_state_cb
    sys.exit(1)
SystemExit: 1
Execution interuptedTraceback (most recent call last):
  File "/Users/ibethune/Desktop/extasy-test2/lib/python2.7/site-packages/radical/ensemblemd/exec_plugins/simulation_analysis_loop/static.py", line 743, in execute_pattern
    resource._umgr.wait_units(uids)
  File "/Users/ibethune/Desktop/extasy-test2/lib/python2.7/site-packages/radical/pilot/unit_manager.py", line 697, in wait_units
    time.sleep (0.5)
KeyboardInterrupt

        Starting DeallocationResource allocation cancelled. You probably ran out of walltime
        done 

I have placed the contents of the pilot directory in /work/e290/e290/shared/iain/rp.session.mbp-ib.epcc.ed.ac.uk.ibethune.016786.0003-pilot.0000

A couple of things that I observed:

• RP appears to start up OK, at least CUs start executing
• The 1st CU unit.000000/STDERR contains an error: ModuleCmd_Load.c(226):ERROR:105: Unable to locate a modulefile for 'python' and so the CU should be failing, but according the enmd, the pre_loop step was completed successfully. There is some error handling missing here.
• The radical_pilot_launch_script.sh for unit.000000 contains # Pre-exec commands module load python which is the source of that error. It should be module load python-compute
• The next CUs unit.000001 thru 000008 are full of errors: run.sh: line 22: grompp: command not found run.sh: line 23: mdrun: command not found cat: confout.gro: No such file or directory However, according the enmd, again these CUs are seen as successful - so the error handling here needs to be improved.
• The final CU unit.000009 is also trying to load the incorrect modules: in radical_pilot_launch_script.sh there is # Pre-exec commands module load packages-archer module load gromacs/5.0.0 module load python-compute/2.7.6 This should just be 'module load gromacs' At least in these cases error causes the CU to fail according to enmd, which shuts down the workflow.

There are several fixes needed here:

• The pre_lsdmap.py and pre_grlsd_loop.py kernel plugins in radical/ensemblemd need updated to use the correct modules on ARCHER. Only gromacs.py has been updated so far.
• The extasy/helper_scripts/run.py needs to be updated to use the correct gromacs executables - in Gromacs 5.1 you can no longer use grompp, mdrun and trjconv but rather 'gmx grompp' 'gmx mdrun' and 'gmx trjconv' I think the old use was deprecated but still supported in 5.0.x and was dropped in 5.1.

[vivek-bala]

The updates were made directly to master and devel was out of sync from master. Merged the two. I have pushed the two fixes as well.

[vivek-bala]

devel and master should both be fixed now. Moving the error handling issues to another issue.

[ibethune]

Thanks, that's now working. I have a problem with the lsdmap CU, will raise a new issue for that later.