Closed antonst closed 8 years ago
Next time I rerun it, it got further but produced this:
2015-10-20 16:01:27,930: radical.repex : MainProcess : Thread-4 : INFO : ComputeUnit 'unit.000019' state changed to AgentStagingInputPending.
2015-10-20 16:01:27,931: radical.pilot : MainProcess : Thread-4 : INFO : [Callback]: unit unit.000021 state on pilot pilot.0000: AgentStagingInputPending.
2015-10-20 16:01:27,931: radical.repex : MainProcess : Thread-4 : INFO : ComputeUnit 'unit.000021' state changed to AgentStagingInputPending.
2015-10-20 16:01:27,931: radical.pilot : MainProcess : Thread-4 : INFO : [Callback]: unit unit.000020 state on pilot pilot.0000: AgentStagingInputPending.
2015-10-20 16:01:27,931: radical.repex : MainProcess : Thread-4 : INFO : ComputeUnit 'unit.000020' state changed to AgentStagingInputPending.
2015-10-20 16:01:28,463: radical.pilot : MainProcess : PilotLauncherWorker-1: INFO : Performing periodical health check for pilot.0000 (SAGA job id [fork://localhost/]-[753856.0])
2015-10-20 16:01:28,571: radical.pilot : MainProcess : PilotLauncherWorker-1: WARNING : could not reconnect to pilot for state check (failed to list jobs: (127)(LIST
sh: LIST: command not found
) (/home/antontre/ram/lib/python2.7/site-packages/saga/adaptors/shell/shell_job.py +911 (list) : % (ret, out))))
2015-10-20 16:01:28,571: radical.pilot : MainProcess : PilotLauncherWorker-1: DEBUG : giving up after 10 attempts
2015-10-20 16:01:28,598: radical.pilot : MainProcess : PilotLauncherWorker-1: DEBUG : Could not reconnect to pilot pilot.0000 multiple times - giving up
2015-10-20 16:01:28,598: radical.pilot : MainProcess : PilotLauncherWorker-1: WARNING : pilot pilot.0000 declared dead
2015-10-20 16:01:29,118: radical.pilot : MainProcess : Thread-1 : INFO : ComputePilot 'pilot.0000' state changed from 'Active' to 'Failed'.
2015-10-20 16:01:29,119: radical.pilot : MainProcess : Thread-1 : INFO : [Callback]: ComputePilot 'pilot.0000' state: Failed.
2015-10-20 16:01:29,123: radical.pilot : MainProcess : Thread-4 : INFO : [Callback]: unit unit.000018 state on pilot pilot.0000: Executing.
2015-10-20 16:01:29,120: radical.repex : MainProcess : Thread-1 : INFO : ComputePilot 'pilot.0000' state changed to Failed.
2015-10-20 16:01:29,124: radical.repex : MainProcess : Thread-4 : INFO : ComputeUnit 'unit.000018' state changed to Executing.
2015-10-20 16:01:29,125: radical.pilot : MainProcess : Thread-4 : INFO : [Callback]: unit unit.000019 state on pilot pilot.0000: Executing.
2015-10-20 16:01:29,124: radical.repex : MainProcess : Thread-1 : ERROR : Pilot error: [<radical.pilot.logentry.Logentry object at 0x7f76d8399150>, <radical.pilot.logentry.Logentry object at 0x7f76d83ba610>, <radical.pilot.logentry.Logentry object at 0x7f76d83ba6d0>, <radical.pilot.logentry.Logentry object at 0x7f76d83bafd0>, <radical.pilot.logentry.Logentry object at 0x7f76d83ba550>, <radical.pilot.logentry.Logentry object at 0x7f76d83ba590>, <radical.pilot.logentry.Logentry object at 0x7f76d83ba710>, <radical.pilot.logentry.Logentry object at 0x7f76d83baf10>, <radical.pilot.logentry.Logentry object at 0x7f76d83ba690>, <radical.pilot.logentry.Logentry object at 0x7f76d83ba650>]
2015-10-20 16:01:29,125: radical.repex : MainProcess : Thread-1 : ERROR : RepEx execution FAILED.
2015-10-20 16:01:29,125: radical.repex : MainProcess : Thread-4 : INFO : ComputeUnit 'unit.000019' state changed to Executing.
2015-10-20 16:01:29,125: radical.pilot : MainProcess : Thread-1 : ERROR : pilot manager controller thread caught system exit -- forcing application shutdown
Traceback (most recent call last):
File "/home/antontre/ram/lib/python2.7/site-packages/radical/pilot/controller/pilot_manager_controller.py", line 335, in run
self.call_callbacks(pilot_id, new_state)
File "/home/antontre/ram/lib/python2.7/site-packages/radical/pilot/controller/pilot_manager_controller.py", line 258, in call_callbacks
cb(self._shared_data[pilot_id]['facade_object'](), new_state)
File "/home/antontre/ram/lib/python2.7/site-packages/pilot_kernels/pilot_kernel.py", line 73, in pilot_state_cb
sys.exit(1)
SystemExit: 1
2015-10-20 16:01:29,127: radical.pilot : MainProcess : Thread-4 : INFO : [Callback]: unit unit.000021 state on pilot pilot.0000: Allocating.
2015-10-20 16:01:29,127: radical.pilot : MainProcess : Thread-1 : DEBUG : pworker Thread-1 stops launcher PilotLauncherWorker-1
2015-10-20 16:01:29,128: radical.repex : MainProcess : Thread-4 : INFO : ComputeUnit 'unit.000021' state changed to Allocating.
2015-10-20 16:01:29,129: radical.pilot : MainProcess : Thread-1 : DEBUG : launcher PilotLauncherWorker-1 stopping
2015-10-20 16:01:29,130: radical.pilot : MainProcess : Thread-4 : INFO : [Callback]: unit unit.000020 state on pilot pilot.0000: Allocating.
2015-10-20 16:01:29,130: radical.repex : MainProcess : Thread-4 : INFO : ComputeUnit 'unit.000020' state changed to Allocating.
Traceback (most recent call last):
File "/home/antontre/ram/bin/repex-amber", line 58, in <module>
pilot_kernel.run_simulation( replicas, pilot_object, session, md_kernel )
File "/home/antontre/ram/lib/python2.7/site-packages/pilot_kernels/pilot_kernel_pattern_s.py", line 209, in run_simulation
unit_manager.wait_units()
File "/home/antontre/ram/lib/python2.7/site-packages/radical/pilot/unit_manager.py", line 697, in wait_units
time.sleep (0.5)
KeyboardInterrupt
2015-10-20 16:01:29,626: radical.pilot : MainProcess : Thread-1 : DEBUG : launcher PilotLauncherWorker-1 stopped
2015-10-20 16:01:29,627: radical.pilot : MainProcess : Thread-1 : DEBUG : pworker Thread-1 stopped launcher PilotLauncherWorker-1
2015-10-20 16:02:31,153: radical.pilot : MainProcess : Thread-4 : INFO : [Callback]: unit unit.000019 state on pilot pilot.0000: AgentStagingOutputPending.
2015-10-20 16:02:31,155: radical.repex : MainProcess : Thread-4 : INFO : ComputeUnit 'unit.000019' state changed to AgentStagingOutputPending.
Strangely enough simulation continued and CU's were execution even after this point.
Guys, could it be that the pilot is simply timing out?
Closing as irrelevant now
with:
amber_path = "/tmp/amber14/bin/sander"