search

saga-project / BigJob

SAGA-based Pilot-Job Implementation for Compute and Data
http://saga-project.github.com/BigJob/
Other
19 stars 8 forks source link

Fork adaptor does not respect requested pilot size #100

Closed ashleyz closed 11 years ago

ashleyz commented 11 years ago

This pilot_compute_description:

pilot_compute_description = {
    "service_url":"fork://localhost",
    "working_directory" : os.path.join(os.getcwd(), "agent"),
    "number_of_processes" : 1, #we will edit this in real time/per run                                                                           
#    "project": "TG-MCB090174",                                                                                                                  
    "queue": "debug",
    "walltime": "240",
    }

with multiple identical 1-core CUs assigned to it has output like:

05/03/2013 03:57:46 PM - bigjob - DEBUG - update description of pilot job to: {'queue': 'debug', 'working_directory': '/home/ashley/bj-performance-experiments/agent', 'service_url': 'fork://localhost', 'number_of_processes': 1, 'walltime': '240'}

05/03/2013 03:57:48 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-baa8ba82-b42b-11e2-a352-0023143c1d50', 'SPMDVariation': 'single'}

But, it appears CUs are consumed 4 at a time, which is equivalent to the number of CPUs reported by my machine (2 physical cores + HT)

05/03/2013 03:58:04 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 32}
05/03/2013 03:58:06 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 25, 'Running': 4, 'New': 3}
05/03/2013 03:58:08 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 25, 'Running': 4, 'New': 3}
05/03/2013 03:58:10 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 25, 'Done': 4, 'New': 3}
05/03/2013 03:58:12 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 17, 'Running': 4, 'Done': 4, 'New': 7}
05/03/2013 03:58:14 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 17, 'Running': 4, 'Done': 4, 'New': 7}
05/03/2013 03:58:16 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 16, 'Running': 1, 'Done': 8, 'New': 7}
05/03/2013 03:58:18 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 8, 'Running': 4, 'Done': 8, 'New': 12}
05/03/2013 03:58:20 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 8, 'Done': 12, 'New': 12}

Debug text:

05/03/2013 03:58:01 PM - bigjob - INFO - Loading BigJob version: 0.5 on lapcat
05/03/2013 03:58:02 PM - bigjob - WARNING - slurm+ssh://<hostname> plugin not compatible with SAGA Bliss. Use slurm+ssh://<hostname>
05/03/2013 03:58:02 PM - bigjob - DEBUG - ['/home/ashley/saga-python-env/local/lib/python2.7/site-packages/BigJob-0.5-py2.7.egg/pilot/filemanagement/../../../webhdfs-py/', '/home/ashley/bj-performance-experiments', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/distribute-0.6.34-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/BigJob-0.5-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/pexpect-2.4-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/simplejson-2.0.9-py2.7-linux-x86_64.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/boto-2.2.2-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/globusonline_transfer_api_client-0.10.14-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/python_hostlist-1.14-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/google_api_python_client-1.1-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/saga_python-0.9.3-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/redis-2.2.4-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/virtualenv-1.9.1-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/threadpool-1.2.7-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/uuid-1.30-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/python_gflags-2.0-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/httplib2-0.8-py2.7.egg', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/colorama-0.2.5-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/distribute-0.6.34-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/BigJob-0.5-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/pexpect-2.4-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/simplejson-2.0.9-py2.7-linux-x86_64.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/boto-2.2.2-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/globusonline_transfer_api_client-0.10.14-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/python_hostlist-1.14-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/google_api_python_client-1.1-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/saga_python-0.9.3-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/redis-2.2.4-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/virtualenv-1.9.1-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/threadpool-1.2.7-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/uuid-1.30-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/python_gflags-2.0-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/httplib2-0.8-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7/site-packages/colorama-0.2.5-py2.7.egg', '/home/ashley/saga-python-env/lib/python2.7', '/home/ashley/saga-python-env/lib/python2.7/plat-x86_64-linux-gnu', '/home/ashley/saga-python-env/lib/python2.7/lib-tk', '/home/ashley/saga-python-env/lib/python2.7/lib-old', '/home/ashley/saga-python-env/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages', '/home/ashley/saga-python-env/lib/python2.7/site-packages', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/BigJob-0.5-py2.7.egg/bigjob', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/BigJob-0.5-py2.7.egg/pilot/impl/../..', '/home/ashley/saga-python-env/local/lib/python2.7/site-packages/BigJob-0.5-py2.7.egg/pilot/filemanagement/../..']
05/03/2013 03:58:02 PM - bigjob - WARNING - WebHDFS package not found.
05/03/2013 03:58:02 PM - bigjob - WARNING - Amazon S3 package not found.
05/03/2013 03:58:02 PM - bigjob - WARNING - iRods Storage package not found.
05/03/2013 03:58:02 PM - bigjob - DEBUG - Created Pilot Compute Service: redis://localhost/pcs/pcs-c3061116-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:02 PM - bigjob - DEBUG - start bigjob at: fork://localhost
05/03/2013 03:58:02 PM - bigjob - DEBUG - Utilizing Redis Backend
05/03/2013 03:58:02 PM - bigjob - DEBUG - Parsing URL: redis://localhost
05/03/2013 03:58:02 PM - bigjob - DEBUG - redis:// localhost None
05/03/2013 03:58:02 PM - bigjob - DEBUG - Connect to Redis: localhost Port: 6379
05/03/2013 03:58:02 PM - bigjob - DEBUG - init BigJob w/: redis://localhost
05/03/2013 03:58:02 PM - bigjob - DEBUG - initialized BigJob: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:02 PM - bigjob - DEBUG - create pilot job entry on backend server: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:02 PM - bigjob - DEBUG - update state of pilot job to: Unknown stopped: False
05/03/2013 03:58:02 PM - bigjob - DEBUG - update description of pilot job to: {'queue': 'debug', 'working_directory': '/home/ashley/bj-performance-experiments/agent', 'service_url': 'fork://localhost', 'number_of_processes': 1, 'walltime': '240'}
05/03/2013 03:58:02 PM - bigjob - DEBUG - set pilot state to: Unknown
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.context.myproxy
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.context.myproxy for saga.Context API with URL scheme(s) ['myproxy://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.context.x509
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.context.x509 for saga.Context API with URL scheme(s) ['x509://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.context.ssh
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.context.ssh for saga.Context API with URL scheme(s) ['ssh://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.context.userpass
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.context.userpass for saga.Context API with URL scheme(s) ['userpass://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.shell.shell_file
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_file for saga.namespace.Directory API with URL scheme(s) ['file://', 'local://', 'sftp://', 'gsiftp://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_file for saga.namespace.Entry API with URL scheme(s) ['file://', 'local://', 'sftp://', 'gsiftp://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_file for saga.filesystem.Directory API with URL scheme(s) ['file://', 'local://', 'sftp://', 'gsiftp://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_file for saga.filesystem.File API with URL scheme(s) ['file://', 'local://', 'sftp://', 'gsiftp://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.shell.shell_job
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_job for saga.job.Service API with URL scheme(s) ['fork://', 'local://', 'ssh://', 'gsissh://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_job for saga.job.Job API with URL scheme(s) ['fork://', 'local://', 'ssh://', 'gsissh://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.sge.sgejob
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.sge.sgejob for saga.job.Service API with URL scheme(s) ['sge://', 'sge+ssh://', 'sge+gsissh://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.sge.sgejob for saga.job.Job API with URL scheme(s) ['sge://', 'sge+ssh://', 'sge+gsissh://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.pbs.pbsjob
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.pbs.pbsjob for saga.job.Service API with URL scheme(s) ['pbs://', 'pbs+ssh://', 'pbs+gsissh://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.pbs.pbsjob for saga.job.Job API with URL scheme(s) ['pbs://', 'pbs+ssh://', 'pbs+gsissh://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.condor.condorjob
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.condor.condorjob for saga.job.Service API with URL scheme(s) ['condor://', 'condor+ssh://', 'condor+gsissh://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.condor.condorjob for saga.job.Job API with URL scheme(s) ['condor://', 'condor+ssh://', 'condor+gsissh://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.slurm.slurm_job
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.slurm.slurm_job for saga.job.Service API with URL scheme(s) ['slurm://', 'slurm+ssh://', 'slurm+gsissh://']
2013:05:03 15:58:02 140017895196480 saga.engine           : [INFO    ] Register adaptor saga.adaptors.slurm.slurm_job for saga.job.Job API with URL scheme(s) ['slurm://', 'slurm+ssh://', 'slurm+gsissh://']
2013:05:03 15:58:02 140017895196480 saga.adaptor.ssh      : [INFO    ] default SSH  context for cert  at /home/ashley/.ssh/id_rsa
2013:05:03 15:58:02 140017895196480 saga.adaptor.ssh      : [INFO    ] default SSH  context for cert  at /home/ashley/.ssh/id_dsa
2013:05:03 15:58:02 140017895196480 saga.DefaultSession   : [DEBUG   ] Adding defaults for context adaptors: ['saga.adaptor.x509', 'saga.adaptor.userpass', 'saga.adaptor.myproxy', 'saga.adaptor.ssh'] 
2013:05:03 15:58:02 140017895196480 PTYShellFactory       : [DEBUG   ] open master pty for [sh] [localhost] ashley: /usr/bin/env TERM=vt100 /bin/bash  -l -i'
2013:05:03 15:58:02 140017895196480 ShellJobService       : [DEBUG   ] PTYProcess: '/usr/bin/env TERM=vt100 /bin/bash -l -i'
2013:05:03 15:58:02 140017895196480 ShellJobService       : [INFO    ] running: /usr/bin/env TERM=vt100 /bin/bash -l -i
2013:05:03 15:58:02 140017895196480 ShellJobService       : [DEBUG   ] read : [  101] (xmodmap:  commandline:1:  bad keysym in remove modifier list 'Caps_Lock', no corresponding keycodes\n)
2013:05:03 15:58:02 140017895196480 ShellJobService       : [DEBUG   ] read : [   42] (xmodmap:  1 error encountered, aborting.\n)
2013:05:03 15:58:02 140017895196480 ShellJobService       : [DEBUG   ] read : [   44] (ashley@lapcat:~/bj-performance-experiments$ )
2013:05:03 15:58:02 140017895196480 ShellJobService       : [DEBUG   ] got initial shell prompt
2013:05:03 15:58:02 140017895196480 ShellJobService       : [DEBUG   ] PTYProcess: '/usr/bin/env TERM=vt100 /bin/bash -l -i'
2013:05:03 15:58:02 140017895196480 ShellJobService       : [INFO    ] running: /usr/bin/env TERM=vt100 /bin/bash -l -i
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] read : [  101] (xmodmap:  commandline:1:  bad keysym in remove modifier list 'Caps_Lock', no corresponding keycodes\n)
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] read : [   42] (xmodmap:  1 error encountered, aborting.\n)
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] read : [   44] (ashley@lapcat:~/bj-performance-experiments$ )
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] got initial shell prompt
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] run_sync: unset PROMPT_COMMAND ; stty -echo; PS1='PROMPT-$?->'; PS2=''; export PS1 PS2
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] write: [   77] (unset PROMPT_COMMAND ; stty -echo; PS1='PROMPT-$?->'; PS2=''; export PS1 PS2\n)
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] read : [   10] (PROMPT-0->)
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] write: [    1] (\n)
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] read : [   10] (PROMPT-0->)
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] got new shell prompt
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] run_sync: mkdir -p ~/.saga/adaptors/shell_job
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] write: [   36] (mkdir -p ~/.saga/adaptors/shell_job\n)
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] read : [   10] (PROMPT-0->)
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] PTYProcess: '/usr/bin/env TERM=vt100 /bin/bash -l -i'
2013:05:03 15:58:03 140017895196480 ShellJobService       : [INFO    ] running: /usr/bin/env TERM=vt100 /bin/bash -l -i
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] read : [  143] (xmodmap:  commandline:1:  bad keysym in remove modifier list 'Caps_Lock', no corresponding keycodes\nxmodmap:  1 error encountered, aborting.\n)
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] read : [   44] (ashley@lapcat:~/bj-performance-experiments$ )
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] got initial shell prompt
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] write: [  110] (cd ~ && exec /bin/cp  /home/ashley/.saga/adaptors/shell//staging.47269200 .saga/adaptors/shell_job/wrapper.sh\n)
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] copy done
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] run_sync: /bin/sh -c '/bin/sh ~/.saga/adaptors/shell_job/wrapper.sh $$ && kill -9 $PPID' || false
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] write: [   88] (/bin/sh -c '/bin/sh ~/.saga/adaptors/shell_job/wrapper.sh $$ && kill -9 $PPID' || false\n)
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] read : [   24] (PID: 17483\nPROMPT-0->\n)
2013:05:03 15:58:03 140017895196480 ShellJobService       : [DEBUG   ] got cmd prompt (0)(PID: 17483)
05/03/2013 03:58:03 PM - bigjob - DEBUG - setting walltime to: 240
05/03/2013 03:58:03 PM - bigjob - DEBUG - Use SSH backend for PilotData
05/03/2013 03:58:03 PM - bigjob - DEBUG - Security Context: None
05/03/2013 03:58:03 PM - bigjob - DEBUG - BigJob working directory: ssh://localhost//home/ashley/bj-performance-experiments/agent/bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:03 PM - bigjob - DEBUG - Create directory: //home/ashley/bj-performance-experiments/agent/bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:03 PM - bigjob - DEBUG - ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 localhost mkdir //home/ashley/bj-performance-experiments/agent/bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:03 PM - bigjob - DEBUG - Run ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 localhost mkdir //home/ashley/bj-performance-experiments/agent/bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50 Output: ["Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.\r\r\n"]
05/03/2013 03:58:04 PM - bigjob - WARNING - No file staging adaptor found.
05/03/2013 03:58:04 PM - bigjob - DEBUG - BJ Working Directory: /home/ashley/bj-performance-experiments/agent
05/03/2013 03:58:04 PM - bigjob - DEBUG - Adaptor specific modifications: fork
05/03/2013 03:58:04 PM - bigjob - DEBUG - Escape PBS
05/03/2013 03:58:04 PM - bigjob - DEBUG - 'import sys
import os
import urllib
import sys
import time
start_time = time.time()
home = os.environ.get("HOME")
#print "Home: " + home
if home==None: home = os.getcwd()
BIGJOB_AGENT_DIR= os.path.join(home, ".bigjob")
if not os.path.exists(BIGJOB_AGENT_DIR): os.mkdir (BIGJOB_AGENT_DIR)
BIGJOB_PYTHON_DIR=BIGJOB_AGENT_DIR+"/python/"
if not os.path.exists(BIGJOB_PYTHON_DIR): os.mkdir(BIGJOB_PYTHON_DIR)
BOOTSTRAP_URL="https://raw.github.com/saga-project/BigJob/master/bootstrap/bigjob-bootstrap.py"
BOOTSTRAP_FILE=BIGJOB_AGENT_DIR+"/bigjob-bootstrap.py"
#ensure that BJ in .bigjob is upfront in sys.path
sys.path.insert(0, os.getcwd() + "/../")
p = list()
for i in sys.path:
    if i.find(".bigjob/python")>1:
          p.insert(0, i)
for i in p: sys.path.insert(0, i)
print "Python path: " + str(sys.path)
print "Python version: " + str(sys.version_info)
try: import saga
except: print "SAGA and SAGA Python Bindings not found.";
try: import bigjob.bigjob_agent
except: 
    print "BigJob not installed. Attempt to install it."; 
    opener = urllib.FancyURLopener({}); 
    opener.retrieve(BOOTSTRAP_URL, BOOTSTRAP_FILE); 
    print "Execute: " + "python " + BOOTSTRAP_FILE + " " + BIGJOB_PYTHON_DIR
    os.system("/usr/bin/env")
    try:
        os.system("python " + BOOTSTRAP_FILE + " " + BIGJOB_PYTHON_DIR); 
        activate_this = os.path.join(BIGJOB_PYTHON_DIR, "bin/activate_this.py"); 
        execfile(activate_this, dict(__file__=activate_this))
    except:
        print "BJ installation failed. Trying system-level python (/usr/bin/python)";
        os.system("/usr/bin/python " + BOOTSTRAP_FILE + " " + BIGJOB_PYTHON_DIR); 
        activate_this = os.path.join(BIGJOB_PYTHON_DIR, "bin/activate_this.py"); 
        execfile(activate_this, dict(__file__=activate_this))
#try to import BJ once again
import bigjob.bigjob_agent
# execute bj agent
args = list()
args.append("bigjob_agent.py")
args.append("redis://localhost:6379")
args.append("bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost")
args.append("PilotComputeServiceQueue-pcs-c3061116-b42b-11e2-9ec1-0023143c1d50")
print "Bootstrap time: " + str(time.time()-start_time)
print "Starting BigJob Agents with following args: " + str(args)
bigjob_agent = bigjob.bigjob_agent.bigjob_agent(args)
'
05/03/2013 03:58:04 PM - bigjob - DEBUG - Working directory: /home/ashley/bj-performance-experiments/agent Job Description: <class 'saga.job.description.Description'> <bound method Description.as_dict of <saga.job.description.Description object at 0x2d8a110>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - Creating pilot job with description: <class 'saga.job.description.Description'> <bound method Description.as_dict of <saga.job.description.Description object at 0x2d8a110>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - Trying to submit pilot job to: fork://localhost
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] run_sync: BULK
LRUN
( cd /home/ashley/bj-performance-experiments/agent && /usr/bin/env python -c 'import sys
import os
import urllib
import sys
import time
start_time = time.time()
home = os.environ.get("HOME")
#print "Home: " + home
if home==None: home = os.getcwd()
BIGJOB_AGENT_DIR= os.path.join(home, ".bigjob")
if not os.path.exists(BIGJOB_AGENT_DIR): os.mkdir (BIGJOB_AGENT_DIR)
BIGJOB_PYTHON_DIR=BIGJOB_AGENT_DIR+"/python/"
if not os.path.exists(BIGJOB_PYTHON_DIR): os.mkdir(BIGJOB_PYTHON_DIR)
BOOTSTRAP_URL="https://raw.github.com/saga-project/BigJob/master/bootstrap/bigjob-bootstrap.py"
BOOTSTRAP_FILE=BIGJOB_AGENT_DIR+"/bigjob-bootstrap.py"
#ensure that BJ in .bigjob is upfront in sys.path
sys.path.insert(0, os.getcwd() + "/../")
p = list()
for i in sys.path:
    if i.find(".bigjob/python")>1:
          p.insert(0, i)
for i in p: sys.path.insert(0, i)
print "Python path: " + str(sys.path)
print "Python version: " + str(sys.version_info)
try: import saga
except: print "SAGA and SAGA Python Bindings not found.";
try: import bigjob.bigjob_agent
except: 
    print "BigJob not installed. Attempt to install it."; 
    opener = urllib.FancyURLopener({}); 
    opener.retrieve(BOOTSTRAP_URL, BOOTSTRAP_FILE); 
    print "Execute: " + "python " + BOOTSTRAP_FILE + " " + BIGJOB_PYTHON_DIR
    os.system("/usr/bin/env")
    try:
        os.system("python " + BOOTSTRAP_FILE + " " + BIGJOB_PYTHON_DIR); 
        activate_this = os.path.join(BIGJOB_PYTHON_DIR, "bin/activate_this.py"); 
        execfile(activate_this, dict(__file__=activate_this))
    except:
        print "BJ installation failed. Trying system-level python (/usr/bin/python)";
        os.system("/usr/bin/python " + BOOTSTRAP_FILE + " " + BIGJOB_PYTHON_DIR); 
        activate_this = os.path.join(BIGJOB_PYTHON_DIR, "bin/activate_this.py"); 
        execfile(activate_this, dict(__file__=activate_this))
#try to import BJ once again
import bigjob.bigjob_agent
# execute bj agent
args = list()
args.append("bigjob_agent.py")
args.append("redis://localhost:6379")
args.append("bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost")
args.append("PilotComputeServiceQueue-pcs-c3061116-b42b-11e2-9ec1-0023143c1d50")
print "Bootstrap time: " + str(time.time()-start_time)
print "Starting BigJob Agents with following args: " + str(args)
bigjob_agent = bigjob.bigjob_agent.bigjob_agent(args)
' ) 1>/home/ashley/bj-performance-experiments/agent/stdout-bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50-agent.txt 2>/home/ashley/bj-performance-experiments/agent/stderr-bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50-agent.txt 
LRUN_EOT
BULK_RUN
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] write: [ 2590] (BULK\nLRUN\n( cd /home/ashley/ ... ent.txt \nLRUN_EOT\nBULK_RUN\n)
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] read : [   10] (RUNNING \n)
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] read : [   11] (OK\n17632\n)
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] read : [   12] (PROMPT-0->\n)
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] ['RUNNING ', 'OK', '17632']
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] started job [fork://localhost]-[17632]
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] read : [   20] (OK\nBULK COMPLETED\n)
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] read : [   12] (PROMPT-0->\n)
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] run_sync: STATS 17632
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] write: [   12] (STATS 17632\n)
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] read : [   45] (OK\nSTATE : RUNNING\n\nSTART : 1367611084\n\n)
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] read : [   12] (PROMPT-0->\n)
2013:05:03 15:58:04 140017895196480 ShellJobService       : [DEBUG   ] ['OK', 'STATE : RUNNING', 'START : 1367611084']
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submission succeeded. Job ID: [fork://localhost]-[17632] 
05/03/2013 03:58:04 PM - bigjob - DEBUG - Create PilotCompute for BigJob: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - Total Jobs: 0 States: {}
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c412646a-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dab650>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c412646a-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4143768-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dab810>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4143768-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4154f22-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dab9d0>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4154f22-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c416bd44-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dabb90>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c416bd44-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c417b802-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dabd10>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c417b802-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c418ba72-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dabed0>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c418ba72-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c41987a4-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dabfd0>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c41987a4-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c41a4d4c-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dbd290>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c41a4d4c-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c41b74e2-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dbd450>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c41b74e2-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c41c8e90-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dbd610>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c41c8e90-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c41d73fa-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dbd7d0>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c41d73fa-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c41e69ea-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dbd990>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c41e69ea-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c41f5ada-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dbdb50>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c41f5ada-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4204b70-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dbdd10>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4204b70-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4216532-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dbded0>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4216532-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c42251c2-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2dbdfd0>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c42251c2-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4235c8e-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e58290>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4235c8e-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4243690-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e58450>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4243690-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4254558-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e58610>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4254558-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c426248c-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e587d0>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c426248c-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4271e00-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e58990>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4271e00-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4283c18-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e58b50>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4283c18-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4293618-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e58d10>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4293618-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c42aee22-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e58ed0>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c42aee22-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c42c808e-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e58fd0>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c42c808e-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c42dd588-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e6d290>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c42dd588-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c42f9f30-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e6d450>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c42f9f30-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4308800-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e6d610>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4308800-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4315a96-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e6d7d0>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4315a96-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4322fe8-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e6d990>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4322fe8-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c4332d58-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e6db50>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c4332d58-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 03:58:04 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost
05/03/2013 03:58:04 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-c3089f8a-b42b-11e2-9ec1-0023143c1d50:localhost:jobs:sj-c43431a8-b42b-11e2-9ec1-0023143c1d50
05/03/2013 03:58:04 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x2e6dd10>>
05/03/2013 03:58:04 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-c43431a8-b42b-11e2-9ec1-0023143c1d50', 'SPMDVariation': 'single'}
05/03/2013 03:58:04 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 03:58:04 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 32}
05/03/2013 03:58:06 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 25, 'Running': 4, 'New': 3}
05/03/2013 03:58:08 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 25, 'Running': 4, 'New': 3}
05/03/2013 03:58:10 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 25, 'Done': 4, 'New': 3}
05/03/2013 03:58:12 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 17, 'Running': 4, 'Done': 4, 'New': 7}
05/03/2013 03:58:14 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 17, 'Running': 4, 'Done': 4, 'New': 7}
05/03/2013 03:58:16 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 16, 'Running': 1, 'Done': 8, 'New': 7}
05/03/2013 03:58:18 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 8, 'Running': 4, 'Done': 8, 'New': 12}
05/03/2013 03:58:20 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 8, 'Done': 12, 'New': 12}
05/03/2013 03:58:22 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 14, 'Running': 4, 'Done': 12, 'Unknown': 2}
05/03/2013 03:58:24 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 14, 'Running': 4, 'Done': 12, 'Unknown': 2}
05/03/2013 03:58:26 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 14, 'Done': 16, 'Unknown': 2}
05/03/2013 03:58:28 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 12, 'Running': 4, 'Done': 16}
05/03/2013 03:58:30 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 8, 'Running': 4, 'Done': 20}
05/03/2013 03:58:32 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 8, 'Running': 4, 'Done': 20}
05/03/2013 03:58:34 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 8, 'Running': 4, 'Done': 20}
05/03/2013 03:58:36 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 4, 'Running': 4, 'Done': 24}
05/03/2013 03:58:38 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 4, 'Running': 4, 'Done': 24}
05/03/2013 03:58:40 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 4, 'Done': 28}
05/03/2013 03:58:42 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Running': 2, 'Done': 28}
05/03/2013 03:58:44 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Running': 2, 'Done': 28}
05/03/2013 03:58:46 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:58:48 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:58:50 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:58:52 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:58:54 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:58:56 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:58:58 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:00 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:02 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:04 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:06 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:08 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:10 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:12 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:14 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:16 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:18 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:20 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:22 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:24 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:26 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:28 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:30 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:32 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:34 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:36 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:38 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:40 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:42 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:44 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:46 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:48 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:50 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:52 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:54 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:56 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 03:59:58 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:00 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:02 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:04 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:07 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:09 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:11 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:13 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:15 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:17 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:19 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:21 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:23 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:25 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:27 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:29 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:31 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:33 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:35 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:37 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:39 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:41 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:43 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:45 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:47 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:49 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:51 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:53 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:55 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:57 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:00:59 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:01:01 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:01:03 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:01:05 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
05/03/2013 04:01:07 PM - bigjob - DEBUG - Total Jobs: 32 States: {'New': 2, 'Done': 30}
andre-merzky commented 11 years ago

Indeed, fork does not support multicore jobs. It should raise an exception, or use mpirun (but I am not sure if we really want to do the latter).

ashleyz commented 11 years ago

For clarification -- I am requesting a single-core pilot with fork://, and it appears to be returning a multi-core pilot. (The single-core CUS are consumed 4 at a time when I believe they should be consumed one at a time).

05/03/2013 03:58:06 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 25, 'Running': 4, 'New': 3}

I would like to be able to request single-core pilots for fork, and actually, being able to have multi-core pilots would be nice too -- I don't care about per-CU MPI, but just having multiple CUs executing simultaneously on the same pilot. Then I could benchmark fork:// pilots with 1-N cores with empty 1-core CUs, and see how that affects things.

drelu commented 11 years ago

Yes, the BJ agent detects the core count and except for Condor utilizes all available cores on a machines. Am 04.05.2013 10:52 schrieb "ashleyz" notifications@github.com:

For clarification -- I am requesting a single-core pilot with fork://, and it appears to be returning a multi-core pilot. (The single-core CUS are consumed 4 at a time when I believe they should be consumed one at a time).

05/03/2013 03:58:06 PM - bigjob - DEBUG - Total Jobs: 32 States: {'Unknown': 25, 'Running': 4, 'New': 3}

I would like to be able to request single-core pilots for fork, and actually, being able to have multi-core pilots would be nice too -- I don't care about per-CU MPI, but just having multiple CUs executing simultaneously on the same pilot. Then I could benchmark fork:// pilots with 1-N cores with empty 1-core CUs, and see how that affects things.

— Reply to this email directly or view it on GitHubhttps://github.com/saga-project/BigJob/issues/100#issuecomment-17434416 .

andre-merzky commented 11 years ago

Thanks Ashley, Andre, understood now -- so that seems to be a BigJob problem / feature, right? Anything we can do on our end to make the behavior more consistent with the job request?

ashleyz commented 11 years ago

It's my understanding that this needs to be fixed on the BJ end, so that any pilot requested with the fork:// adaptor does not automatically expand to consume all available cores.