search

saga-project / BigJob

SAGA-based Pilot-Job Implementation for Compute and Data
http://saga-project.github.com/BigJob/
Other
19 stars 8 forks source link

SAGA-Python + BigJob + SLURM do not request the proper number of cores for a pilot #103

Closed ashleyz closed 11 years ago

ashleyz commented 11 years ago

This may relate to #102

On Stampede, requesting 32-core BJ with SAGA-Python. Relevant part of the errorlog appears to be the following (BJ not forwarding on core request to SAGA-Python in job description?)

05/03/2013 06:15:16 PM - bigjob - DEBUG - Working directory: /home1/01414/ashleyz Job Description: <class 'saga.job.description.Description'> <bound method Description.as_dict of <saga.job.description.Description object at 0x35ae950>>
05/03/2013 06:15:16 PM - bigjob - DEBUG - Creating pilot job with description: <class 'saga.job.description.Description'> <bound method Description.as_dict of <saga.job.description.Description object at 0x35ae950>>
05/03/2013 06:15:16 PM - bigjob - DEBUG - Trying to submit pilot job to: slurm://localhost
2013:05:03 18:15:16 47898035154304 SLURMJobService       : [WARNING ] number_of_processes not specified in submitted SLURM job description -- defaulting to 1!
2013:05:03 18:15:16 47898035154304 SLURMJobService       : [DEBUG   ] SLURM script generated:
#"'!'"/bin/bash
#SBATCH -J \"SAGAPythonSLURMJob\"
#SBATCH -n 1
#SBATCH -D /home1/01414/ashleyz
#SBATCH -o /home1/01414/ashleyz/stdout-bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54-agent.txt
#SBATCH -e /home1/01414/ashleyz/stderr-bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54-agent.txt
#SBATCH -t 04:00:00
#SBATCH -p development
#SBATCH -A TG-MCB090174

Full log:

set to DEBUG
05/03/2013 06:15:11 PM - bigjob - INFO - Loading BigJob version: 0.5 on login2.stampede.tacc.utexas.edu
05/03/2013 06:15:12 PM - bigjob - DEBUG - ['/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/BigJob-0.5-py2.7.egg/pilot/filemanagement/../../../webhdfs-py/', '/home1/01414/ashleyz/bj-performance-experiments', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/saga_python-0.9.3-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/colorama-0.2.5-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/pexpect-2.4-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/simplejson-2.0.9-py2.7-linux-x86_64.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/boto-2.2.2-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/globusonline_transfer_api_client-0.10.14-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/python_hostlist-1.14-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/google_api_python_client-1.1-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/bliss-0.2.7-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/redis-2.2.4-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/threadpool-1.2.7-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/uuid-1.30-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/python_gflags-2.0-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/httplib2-0.8-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/paramiko_on_pypi-1.7.6-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/pycrypto_on_pypi-2.3-py2.7-linux-x86_64.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/BigJob-0.5-py2.7.egg', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/virtualenv-1.9.1-py2.7.egg', '/opt/apps/python/epd/7.3.2/modules/lib/python', '/opt/apps/python/epd/7.3.2/lib', '/home1/01414/ashleyz/saga-python-env/lib/python27.zip', '/home1/01414/ashleyz/saga-python-env/lib/python2.7', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/plat-linux2', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/lib-tk', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/lib-old', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/lib-dynload', '/opt/apps/python/epd/7.3.2/lib/python2.7', '/opt/apps/python/epd/7.3.2/lib/python2.7/plat-linux2', '/opt/apps/python/epd/7.3.2/lib/python2.7/lib-tk', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/BigJob-0.5-py2.7.egg/bigjob', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/BigJob-0.5-py2.7.egg/pilot/impl/../..', '/home1/01414/ashleyz/saga-python-env/lib/python2.7/site-packages/BigJob-0.5-py2.7.egg/pilot/filemanagement/../..']
05/03/2013 06:15:12 PM - bigjob - WARNING - WebHDFS package not found.
05/03/2013 06:15:12 PM - bigjob - DEBUG - Created Pilot Compute Service: redis://ILikeBigJob_wITH-REdIS@gw68.quarry.iu.teragrid.org:6379/pcs/pcs-4867ff08-e192-11e1-a694-00003e980000/pcs/pcs-4e81f168-b447-11e2-b156-d4ae52a0ea54
Beginning new run:
Run #1 of 6
Iteration #: 0
Number of pilots: 1
Number of compute services: 1
Number of jobs: 64
Number of processes per pilot: 32
Number of processes per job: 1
Expected time to run a job(SLEEP_LENGTH): 0
Binding type: computedata
Creating pilots.
05/03/2013 06:15:12 PM - bigjob - DEBUG - start bigjob at: slurm://localhost
05/03/2013 06:15:12 PM - bigjob - DEBUG - Utilizing Redis Backend
05/03/2013 06:15:12 PM - bigjob - DEBUG - Parsing URL: redis://ILikeBigJob_wITH-REdIS@gw68.quarry.iu.teragrid.org:6379/pcs/pcs-4867ff08-e192-11e1-a694-00003e980000
05/03/2013 06:15:12 PM - bigjob - DEBUG - redis:// gw68.quarry.iu.teragrid.org 6379
05/03/2013 06:15:12 PM - bigjob - DEBUG - Connect to Redis: gw68.quarry.iu.teragrid.org Port: 6379
05/03/2013 06:15:12 PM - bigjob - DEBUG - init BigJob w/: redis://ILikeBigJob_wITH-REdIS@gw68.quarry.iu.teragrid.org:6379/pcs/pcs-4867ff08-e192-11e1-a694-00003e980000
05/03/2013 06:15:12 PM - bigjob - DEBUG - initialized BigJob: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:12 PM - bigjob - DEBUG - create pilot job entry on backend server: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost
05/03/2013 06:15:12 PM - bigjob - DEBUG - update state of pilot job to: Unknown stopped: False
05/03/2013 06:15:13 PM - bigjob - DEBUG - update description of pilot job to: {'queue': 'development', 'number_of_processes': 32, 'project': 'TG-MCB090174', 'working_directory': '/home1/01414/ashleyz/', 'service_url': 'slurm://localhost', 'walltime': '240'}
05/03/2013 06:15:13 PM - bigjob - DEBUG - set pilot state to: Unknown
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.context.myproxy
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.context.myproxy for saga.Context API with URL scheme(s) ['myproxy://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.context.x509
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.context.x509 for saga.Context API with URL scheme(s) ['x509://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.context.ssh
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.context.ssh for saga.Context API with URL scheme(s) ['ssh://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.context.userpass
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.context.userpass for saga.Context API with URL scheme(s) ['userpass://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.shell.shell_file
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_file for saga.namespace.Directory API with URL scheme(s) ['file://', 'local://', 'sftp://', 'gsiftp://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_file for saga.namespace.Entry API with URL scheme(s) ['file://', 'local://', 'sftp://', 'gsiftp://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_file for saga.filesystem.Directory API with URL scheme(s) ['file://', 'local://', 'sftp://', 'gsiftp://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_file for saga.filesystem.File API with URL scheme(s) ['file://', 'local://', 'sftp://', 'gsiftp://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.shell.shell_job
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_job for saga.job.Service API with URL scheme(s) ['fork://', 'local://', 'ssh://', 'gsissh://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_job for saga.job.Job API with URL scheme(s) ['fork://', 'local://', 'ssh://', 'gsissh://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.sge.sgejob
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.sge.sgejob for saga.job.Service API with URL scheme(s) ['sge://', 'sge+ssh://', 'sge+gsissh://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.sge.sgejob for saga.job.Job API with URL scheme(s) ['sge://', 'sge+ssh://', 'sge+gsissh://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.pbs.pbsjob
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.pbs.pbsjob for saga.job.Service API with URL scheme(s) ['pbs://', 'pbs+ssh://', 'pbs+gsissh://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.pbs.pbsjob for saga.job.Job API with URL scheme(s) ['pbs://', 'pbs+ssh://', 'pbs+gsissh://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.condor.condorjob
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.condor.condorjob for saga.job.Service API with URL scheme(s) ['condor://', 'condor+ssh://', 'condor+gsissh://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.condor.condorjob for saga.job.Job API with URL scheme(s) ['condor://', 'condor+ssh://', 'condor+gsissh://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.slurm.slurm_job
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.slurm.slurm_job for saga.job.Service API with URL scheme(s) ['slurm://', 'slurm+ssh://', 'slurm+gsissh://']
2013:05:03 18:15:13 47898035154304 saga.engine           : [INFO    ] Register adaptor saga.adaptors.slurm.slurm_job for saga.job.Job API with URL scheme(s) ['slurm://', 'slurm+ssh://', 'slurm+gsissh://']
2013:05:03 18:15:13 47898035154304 saga.adaptor.ssh      : [INFO    ] default SSH  context for cert  at /home1/01414/ashleyz/.ssh/id_rsa
2013:05:03 18:15:13 47898035154304 saga.DefaultSession   : [DEBUG   ] Adding defaults for context adaptors: ['saga.adaptor.x509', 'saga.adaptor.userpass', 'saga.adaptor.myproxy', 'saga.adaptor.ssh'] 
2013:05:03 18:15:13 47898035154304 SLURMJobService       : [DEBUG   ] Opening shell of type: fork://localhost
2013:05:03 18:15:13 47898035154304 PTYShellFactory       : [DEBUG   ] open master pty for [sh] [localhost] ashleyz: /usr/bin/env TERM=vt100 /bin/bash  -l -i'
2013:05:03 18:15:13 47898035154304 SLURMJobService       : [DEBUG   ] PTYProcess: '/usr/bin/env TERM=vt100 /bin/bash -l -i'
2013:05:03 18:15:13 47898035154304 SLURMJobService       : [INFO    ] running: /usr/bin/env TERM=vt100 /bin/bash -l -i
2013:05:03 18:15:13 47898035154304 SLURMJobService       : [DEBUG   ] read : [  325] (---------------------- Project balances for user ashleyz ----------------------\n| Name           Avail SUs     Expires | Name           Avail SUs     Expires |\n| TG-SEE100004        -556  2013-08-15 | TG-MCB090174      239807  2013-06-30 | \n| TG-ASC120003      142772  2014-03-31 |                                      |\n)
2013:05:03 18:15:13 47898035154304 SLURMJobService       : [DEBUG   ] read : [  405] (------------------------ Disk quotas for user ashleyz -------------------------\n| Disk         Usage (GB)     Limit    %Used   File Usage       Limit   %Used |\n| /home1              0.9       5.0    18.41        51312      150000   34.21 |\n| /work               0.0     400.0     0.00            3     3000000    0.00 |\n-------------------------------------------------------------------------------\n)
2013:05:03 18:15:14 47898035154304 SLURMJobService       : [DEBUG   ] read : [   45] (ashleyz@login2:~/bj-performance-experiments$ )
2013:05:03 18:15:14 47898035154304 SLURMJobService       : [DEBUG   ] got initial shell prompt
2013:05:03 18:15:14 47898035154304 SLURMJobService       : [DEBUG   ] PTYProcess: '/usr/bin/env TERM=vt100 /bin/bash -l -i'
2013:05:03 18:15:14 47898035154304 SLURMJobService       : [INFO    ] running: /usr/bin/env TERM=vt100 /bin/bash -l -i
2013:05:03 18:15:14 47898035154304 SLURMJobService       : [DEBUG   ] read : [  325] (---------------------- Project balances for user ashleyz ----------------------\n| Name           Avail SUs     Expires | Name           Avail SUs     Expires |\n| TG-SEE100004        -556  2013-08-15 | TG-MCB090174      239807  2013-06-30 | \n| TG-ASC120003      142772  2014-03-31 |                                      |\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [  405] (------------------------ Disk quotas for user ashleyz -------------------------\n| Disk         Usage (GB)     Limit    %Used   File Usage       Limit   %Used |\n| /home1              0.9       5.0    18.41        51312      150000   34.21 |\n| /work               0.0     400.0     0.00            3     3000000    0.00 |\n-------------------------------------------------------------------------------\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [   45] (ashleyz@login2:~/bj-performance-experiments$ )
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] got initial shell prompt
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] run_sync: unset PROMPT_COMMAND ; stty -echo; PS1='PROMPT-$?->'; PS2=''; export PS1 PS2
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] write: [   77] (unset PROMPT_COMMAND ; stty -echo; PS1='PROMPT-$?->'; PS2=''; export PS1 PS2\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [   10] (PROMPT-0->)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] write: [    1] (\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [   10] (PROMPT-0->)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] got new shell prompt
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] Verifying existence of remote SLURM tools.
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] run_sync: which squeue
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] write: [   13] (which squeue\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [   27] (/usr/bin/squeue\nPROMPT-0->)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] run_sync: which sbatch
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] write: [   13] (which sbatch\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [   27] (/usr/bin/sbatch\nPROMPT-0->)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] run_sync: which scancel
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] write: [   14] (which scancel\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [   18] (/usr/bin/scancel\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [   10] (PROMPT-0->)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] run_sync: which scontrol
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] write: [   15] (which scontrol\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [   19] (/usr/bin/scontrol\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [   10] (PROMPT-0->)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] run_sync: mkdir -p $HOME/.saga/adaptors/slurm_job
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] write: [   40] (mkdir -p $HOME/.saga/adaptors/slurm_job\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [   10] (PROMPT-0->)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] got cmd prompt (0)()
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] No username provided in URL slurm://localhost, so we are going to find it with whoami
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] run_sync: whoami
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] write: [    7] (whoami\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [    9] (ashleyz\n)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] read : [   10] (PROMPT-0->)
2013:05:03 18:15:15 47898035154304 SLURMJobService       : [DEBUG   ] Username detected as: ashleyz
05/03/2013 06:15:15 PM - bigjob - DEBUG - setting walltime to: 240
05/03/2013 06:15:15 PM - bigjob - DEBUG - Use SSH backend for PilotData
05/03/2013 06:15:15 PM - bigjob - DEBUG - Security Context: None
05/03/2013 06:15:15 PM - bigjob - DEBUG - BigJob working directory: ssh://localhost//home1/01414/ashleyz/bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:15 PM - bigjob - DEBUG - Create directory: //home1/01414/ashleyz/bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:15 PM - bigjob - DEBUG - ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 localhost mkdir //home1/01414/ashleyz/bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:16 PM - bigjob - DEBUG - Run ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 localhost mkdir //home1/01414/ashleyz/bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54 Output: ["Warning: Permanently added 'localhost' (RSA) to the list of known hosts.\r\r\n"]
05/03/2013 06:15:16 PM - bigjob - WARNING - No file staging adaptor found.
05/03/2013 06:15:16 PM - bigjob - DEBUG - BJ Working Directory: /home1/01414/ashleyz
05/03/2013 06:15:16 PM - bigjob - DEBUG - Adaptor specific modifications: slurm
05/03/2013 06:15:16 PM - bigjob - DEBUG - Escape PBS
05/03/2013 06:15:16 PM - bigjob - DEBUG - 'import sys
import os
import urllib
import sys
import time
start_time = time.time()
home = os.environ.get("HOME")
#print "Home: " + home
if home==None: home = os.getcwd()
BIGJOB_AGENT_DIR= os.path.join(home, ".bigjob")
if not os.path.exists(BIGJOB_AGENT_DIR): os.mkdir (BIGJOB_AGENT_DIR)
BIGJOB_PYTHON_DIR=BIGJOB_AGENT_DIR+"/python/"
if not os.path.exists(BIGJOB_PYTHON_DIR): os.mkdir(BIGJOB_PYTHON_DIR)
BOOTSTRAP_URL="https://raw.github.com/saga-project/BigJob/master/bootstrap/bigjob-bootstrap.py"
BOOTSTRAP_FILE=BIGJOB_AGENT_DIR+"/bigjob-bootstrap.py"
#ensure that BJ in .bigjob is upfront in sys.path
sys.path.insert(0, os.getcwd() + "/../")
p = list()
for i in sys.path:
    if i.find(".bigjob/python")>1:
          p.insert(0, i)
for i in p: sys.path.insert(0, i)
print "Python path: " + str(sys.path)
print "Python version: " + str(sys.version_info)
try: import saga
except: print "SAGA and SAGA Python Bindings not found.";
try: import bigjob.bigjob_agent
except: 
    print "BigJob not installed. Attempt to install it."; 
    opener = urllib.FancyURLopener({}); 
    opener.retrieve(BOOTSTRAP_URL, BOOTSTRAP_FILE); 
    print "Execute: " + "python " + BOOTSTRAP_FILE + " " + BIGJOB_PYTHON_DIR
    os.system("/usr/bin/env")
    try:
        os.system("python " + BOOTSTRAP_FILE + " " + BIGJOB_PYTHON_DIR); 
        activate_this = os.path.join(BIGJOB_PYTHON_DIR, "bin/activate_this.py"); 
        execfile(activate_this, dict(__file__=activate_this))
    except:
        print "BJ installation failed. Trying system-level python (/usr/bin/python)";
        os.system("/usr/bin/python " + BOOTSTRAP_FILE + " " + BIGJOB_PYTHON_DIR); 
        activate_this = os.path.join(BIGJOB_PYTHON_DIR, "bin/activate_this.py"); 
        execfile(activate_this, dict(__file__=activate_this))
#try to import BJ once again
import bigjob.bigjob_agent
# execute bj agent
args = list()
args.append("bigjob_agent.py")
args.append("redis://ILikeBigJob_wITH-REdIS@gw68.quarry.iu.teragrid.org:6379")
args.append("bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost")
args.append("PilotComputeServiceQueue-pcs-4e81f168-b447-11e2-b156-d4ae52a0ea54")
print "Bootstrap time: " + str(time.time()-start_time)
print "Starting BigJob Agents with following args: " + str(args)
bigjob_agent = bigjob.bigjob_agent.bigjob_agent(args)
'
05/03/2013 06:15:16 PM - bigjob - DEBUG - Working directory: /home1/01414/ashleyz Job Description: <class 'saga.job.description.Description'> <bound method Description.as_dict of <saga.job.description.Description object at 0x35ae950>>
05/03/2013 06:15:16 PM - bigjob - DEBUG - Creating pilot job with description: <class 'saga.job.description.Description'> <bound method Description.as_dict of <saga.job.description.Description object at 0x35ae950>>
05/03/2013 06:15:16 PM - bigjob - DEBUG - Trying to submit pilot job to: slurm://localhost
2013:05:03 18:15:16 47898035154304 SLURMJobService       : [WARNING ] number_of_processes not specified in submitted SLURM job description -- defaulting to 1!
2013:05:03 18:15:16 47898035154304 SLURMJobService       : [DEBUG   ] SLURM script generated:
#"'!'"/bin/bash
#SBATCH -J \"SAGAPythonSLURMJob\"
#SBATCH -n 1
#SBATCH -D /home1/01414/ashleyz
#SBATCH -o /home1/01414/ashleyz/stdout-bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54-agent.txt
#SBATCH -e /home1/01414/ashleyz/stderr-bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54-agent.txt
#SBATCH -t 04:00:00
#SBATCH -p development
#SBATCH -A TG-MCB090174

/usr/bin/env python -c 'import sys
import os
import urllib
import sys
import time
start_time = time.time()
home = os.environ.get(\"HOME\")
#print \"Home: \" + home
if home==None: home = os.getcwd()
BIGJOB_AGENT_DIR= os.path.join(home, \".bigjob\")
if not os.path.exists(BIGJOB_AGENT_DIR): os.mkdir (BIGJOB_AGENT_DIR)
BIGJOB_PYTHON_DIR=BIGJOB_AGENT_DIR+\"/python/\"
if not os.path.exists(BIGJOB_PYTHON_DIR): os.mkdir(BIGJOB_PYTHON_DIR)
BOOTSTRAP_URL=\"https://raw.github.com/saga-project/BigJob/master/bootstrap/bigjob-bootstrap.py\"
BOOTSTRAP_FILE=BIGJOB_AGENT_DIR+\"/bigjob-bootstrap.py\"
#ensure that BJ in .bigjob is upfront in sys.path
sys.path.insert(0, os.getcwd() + \"/../\")
p = list()
for i in sys.path:
    if i.find(\".bigjob/python\")>1:
          p.insert(0, i)
for i in p: sys.path.insert(0, i)
print \"Python path: \" + str(sys.path)
print \"Python version: \" + str(sys.version_info)
try: import saga
except: print \"SAGA and SAGA Python Bindings not found.\";
try: import bigjob.bigjob_agent
except: 
    print \"BigJob not installed. Attempt to install it.\"; 
    opener = urllib.FancyURLopener({}); 
    opener.retrieve(BOOTSTRAP_URL, BOOTSTRAP_FILE); 
    print \"Execute: \" + \"python \" + BOOTSTRAP_FILE + \" \" + BIGJOB_PYTHON_DIR
    os.system(\"/usr/bin/env\")
    try:
        os.system(\"python \" + BOOTSTRAP_FILE + \" \" + BIGJOB_PYTHON_DIR); 
        activate_this = os.path.join(BIGJOB_PYTHON_DIR, \"bin/activate_this.py\"); 
        execfile(activate_this, dict(__file__=activate_this))
    except:
        print \"BJ installation failed. Trying system-level python (/usr/bin/python)\";
        os.system(\"/usr/bin/python \" + BOOTSTRAP_FILE + \" \" + BIGJOB_PYTHON_DIR); 
        activate_this = os.path.join(BIGJOB_PYTHON_DIR, \"bin/activate_this.py\"); 
        execfile(activate_this, dict(__file__=activate_this))
#try to import BJ once again
import bigjob.bigjob_agent
# execute bj agent
args = list()
args.append(\"bigjob_agent.py\")
args.append(\"redis://ILikeBigJob_wITH-REdIS@gw68.quarry.iu.teragrid.org:6379\")
args.append(\"bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost\")
args.append(\"PilotComputeServiceQueue-pcs-4e81f168-b447-11e2-b156-d4ae52a0ea54\")
print \"Bootstrap time: \" + str(time.time()-start_time)
print \"Starting BigJob Agents with following args: \" + str(args)
bigjob_agent = bigjob.bigjob_agent.bigjob_agent(args)
'
2013:05:03 18:15:16 47898035154304 SLURMJobService       : [DEBUG   ] Transferring SLURM script to remote host
2013:05:03 18:15:16 47898035154304 SLURMJobService       : [DEBUG   ] run_sync: echo "#"'!'"/bin/bash
#SBATCH -J \"SAGAPythonSLURMJob\"
#SBATCH -n 1
#SBATCH -D /home1/01414/ashleyz
#SBATCH -o /home1/01414/ashleyz/stdout-bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54-agent.txt
#SBATCH -e /home1/01414/ashleyz/stderr-bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54-agent.txt
#SBATCH -t 04:00:00
#SBATCH -p development
#SBATCH -A TG-MCB090174

/usr/bin/env python -c 'import sys
import os
import urllib
import sys
import time
start_time = time.time()
home = os.environ.get(\"HOME\")
#print \"Home: \" + home
if home==None: home = os.getcwd()
BIGJOB_AGENT_DIR= os.path.join(home, \".bigjob\")
if not os.path.exists(BIGJOB_AGENT_DIR): os.mkdir (BIGJOB_AGENT_DIR)
BIGJOB_PYTHON_DIR=BIGJOB_AGENT_DIR+\"/python/\"
if not os.path.exists(BIGJOB_PYTHON_DIR): os.mkdir(BIGJOB_PYTHON_DIR)
BOOTSTRAP_URL=\"https://raw.github.com/saga-project/BigJob/master/bootstrap/bigjob-bootstrap.py\"
BOOTSTRAP_FILE=BIGJOB_AGENT_DIR+\"/bigjob-bootstrap.py\"
#ensure that BJ in .bigjob is upfront in sys.path
sys.path.insert(0, os.getcwd() + \"/../\")
p = list()
for i in sys.path:
    if i.find(\".bigjob/python\")>1:
          p.insert(0, i)
for i in p: sys.path.insert(0, i)
print \"Python path: \" + str(sys.path)
print \"Python version: \" + str(sys.version_info)
try: import saga
except: print \"SAGA and SAGA Python Bindings not found.\";
try: import bigjob.bigjob_agent
except: 
    print \"BigJob not installed. Attempt to install it.\"; 
    opener = urllib.FancyURLopener({}); 
    opener.retrieve(BOOTSTRAP_URL, BOOTSTRAP_FILE); 
    print \"Execute: \" + \"python \" + BOOTSTRAP_FILE + \" \" + BIGJOB_PYTHON_DIR
    os.system(\"/usr/bin/env\")
    try:
        os.system(\"python \" + BOOTSTRAP_FILE + \" \" + BIGJOB_PYTHON_DIR); 
        activate_this = os.path.join(BIGJOB_PYTHON_DIR, \"bin/activate_this.py\"); 
        execfile(activate_this, dict(__file__=activate_this))
    except:
        print \"BJ installation failed. Trying system-level python (/usr/bin/python)\";
        os.system(\"/usr/bin/python \" + BOOTSTRAP_FILE + \" \" + BIGJOB_PYTHON_DIR); 
        activate_this = os.path.join(BIGJOB_PYTHON_DIR, \"bin/activate_this.py\"); 
        execfile(activate_this, dict(__file__=activate_this))
#try to import BJ once again
import bigjob.bigjob_agent
# execute bj agent
args = list()
args.append(\"bigjob_agent.py\")
args.append(\"redis://ILikeBigJob_wITH-REdIS@gw68.quarry.iu.teragrid.org:6379\")
args.append(\"bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost\")
args.append(\"PilotComputeServiceQueue-pcs-4e81f168-b447-11e2-b156-d4ae52a0ea54\")
print \"Bootstrap time: \" + str(time.time()-start_time)
print \"Starting BigJob Agents with following args: \" + str(args)
bigjob_agent = bigjob.bigjob_agent.bigjob_agent(args)
'" | sbatch
2013:05:03 18:15:16 47898035154304 SLURMJobService       : [DEBUG   ] write: [ 2751] (echo "#"'!'"/bin/bash\n#SBATCH ... job_agent(args)\n'" | sbatch\n)
2013:05:03 18:15:16 47898035154304 SLURMJobService       : [DEBUG   ] read : [  203] (-----------------------------------------------------------------\n              Welcome to the Stampede Supercomputer              \n-----------------------------------------------------------------\n\n)
2013:05:03 18:15:16 47898035154304 SLURMJobService       : [DEBUG   ] read : [   47] (--> Verifying valid submit host (login2)...OK\n)
2013:05:03 18:15:18 47898035154304 SLURMJobService       : [DEBUG   ] read : [   38] (--> Enforcing max jobs per user...OK\n)
2013:05:03 18:15:18 47898035154304 SLURMJobService       : [DEBUG   ] read : [   73] (--> Verifying availability of your home dir (/home1/01414/ashleyz)...OK\n)
2013:05:03 18:15:18 47898035154304 SLURMJobService       : [DEBUG   ] read : [   72] (--> Verifying availability of your work dir (/work/01414/ashleyz)...OK\n)
2013:05:03 18:15:18 47898035154304 SLURMJobService       : [DEBUG   ] read : [  250] (--> Verifying availability of your scratch dir (/scratch/01414/ashleyz)...OK\n--> Verifying access to desired queue (development)...OK\n--> Verifying job request is within current queue limits...OK\n--> Checking available allocation (TG-MCB090174)...)
2013:05:03 18:15:19 47898035154304 SLURMJobService       : [DEBUG   ] read : [    4] (OK\n)
2013:05:03 18:15:19 47898035154304 SLURMJobService       : [DEBUG   ] read : [   38] (Submitted batch job 693632\nPROMPT-0->)
2013:05:03 18:15:19 47898035154304 SLURMJobService       : [DEBUG   ] started job [slurm://localhost]-[693632]
2013:05:03 18:15:19 47898035154304 SLURMJobService       : [DEBUG   ] Batch system output:
-----------------------------------------------------------------
              Welcome to the Stampede Supercomputer              
-----------------------------------------------------------------

--> Verifying valid submit host (login2)...OK
--> Enforcing max jobs per user...OK
--> Verifying availability of your home dir (/home1/01414/ashleyz)...OK
--> Verifying availability of your work dir (/work/01414/ashleyz)...OK
--> Verifying availability of your scratch dir (/scratch/01414/ashleyz)...OK
--> Verifying access to desired queue (development)...OK
--> Verifying job request is within current queue limits...OK
--> Checking available allocation (TG-MCB090174)...OK
Submitted batch job 693632

2013:05:03 18:15:19 47898035154304 SLURMJobService       : [DEBUG   ] run_sync: scontrol show job 693632
2013:05:03 18:15:19 47898035154304 SLURMJobService       : [DEBUG   ] write: [   25] (scontrol show job 693632\n)
2013:05:03 18:15:19 47898035154304 SLURMJobService       : [DEBUG   ] read : [  891] (JobId=693632 Name=SAGAPythonSL ... e1/01414/ashleyz\n\nPROMPT-0->)
05/03/2013 06:15:19 PM - bigjob - DEBUG - Submission succeeded. Job ID: [slurm://localhost]-[693632] 
05/03/2013 06:15:19 PM - bigjob - DEBUG - Create PilotCompute for BigJob: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost
Waiting for pilots to be ready.
05/03/2013 06:15:19 PM - bigjob - DEBUG - Total Jobs: 0 States: {}
It took 0.117213010788 seconds for the pilots to finish queuing
Creating compute services.
05/03/2013 06:15:19 PM - bigjob - DEBUG - redis://localhost/bigjob
05/03/2013 06:15:19 PM - bigjob - DEBUG - CDS URL: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Create CDS directory at redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54
Creating and submitting compute units.
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cb057a-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cb30cc-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cb6024-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cb86c6-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cbac50-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cbde8c-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cc05f6-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cc2d42-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cc53ee-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cc7b6c-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52ccad1c-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52ccd422-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52ccfb14-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cd2206-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cd4998-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cda802-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cdd16a-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cdfbc2-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52ce2354-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52ce4a28-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52ce7624-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52ceabe4-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52ced3f8-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cefb3a-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cf218c-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cf4b1c-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cf71dc-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cf96f8-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cfbd68-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52cfe45a-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d00fd4-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d0370c-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d05e12-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d08892-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d0afa2-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d0daf4-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d101f0-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d128e2-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d1502e-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d176da-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d1a240-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d1c91e-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d1f0ec-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d217de-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d23eda-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d26a36-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d29128-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d2b82e-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d2df34-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d305f4-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d33132-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d3582e-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d37ef8-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d3a5e0-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d3ccfa-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d3f842-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d41f66-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d44522-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d4691c-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d48d02-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d4b5d4-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d4d96a-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d4fdc8-b447-11e2-b156-d4ae52a0ea54
05/03/2013 06:15:19 PM - bigjob - DEBUG - Created CU: redis://localhost/bigjob:cds-52caebb2-b447-11e2-b156-d4ae52a0ea54/cu-52d52186-b447-11e2-b156-d4ae52a0ea54
Done submitting compute units!
It took 0.0672359466553 seconds to submit all CUs
Waiting for compute units to finish
05/03/2013 06:15:19 PM - bigjob - DEBUG - ### START WAIT ###
05/03/2013 06:15:20 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:20 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:20 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:20 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:20 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:21 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Unknown
05/03/2013 06:15:21 PM - bigjob - DEBUG - Candidate PJs: []
05/03/2013 06:15:21 PM - bigjob - DEBUG - No resource found.
05/03/2013 06:15:22 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:22 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:22 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:22 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:22 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:22 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Unknown
05/03/2013 06:15:22 PM - bigjob - DEBUG - Candidate PJs: []
05/03/2013 06:15:22 PM - bigjob - DEBUG - No resource found.
05/03/2013 06:15:23 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:23 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:23 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:23 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:23 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:23 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Unknown
05/03/2013 06:15:23 PM - bigjob - DEBUG - Candidate PJs: []
05/03/2013 06:15:23 PM - bigjob - DEBUG - No resource found.
05/03/2013 06:15:24 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:24 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:24 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:24 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:24 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:24 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Unknown
05/03/2013 06:15:25 PM - bigjob - DEBUG - Candidate PJs: []
05/03/2013 06:15:25 PM - bigjob - DEBUG - No resource found.
05/03/2013 06:15:26 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:26 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:26 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:26 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:26 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:26 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Unknown
05/03/2013 06:15:26 PM - bigjob - DEBUG - Candidate PJs: []
05/03/2013 06:15:26 PM - bigjob - DEBUG - No resource found.
05/03/2013 06:15:27 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:27 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:27 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:27 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:27 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:27 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Unknown
05/03/2013 06:15:27 PM - bigjob - DEBUG - Candidate PJs: []
05/03/2013 06:15:27 PM - bigjob - DEBUG - No resource found.
05/03/2013 06:15:28 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:28 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:28 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:28 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:28 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:28 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Unknown
05/03/2013 06:15:28 PM - bigjob - DEBUG - Candidate PJs: []
05/03/2013 06:15:28 PM - bigjob - DEBUG - No resource found.
05/03/2013 06:15:29 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:29 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:29 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:29 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:29 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:29 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Unknown
05/03/2013 06:15:29 PM - bigjob - DEBUG - Candidate PJs: []
05/03/2013 06:15:29 PM - bigjob - DEBUG - No resource found.
05/03/2013 06:15:30 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:30 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:30 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:30 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:30 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:31 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Unknown
05/03/2013 06:15:31 PM - bigjob - DEBUG - Candidate PJs: []
05/03/2013 06:15:31 PM - bigjob - DEBUG - No resource found.
05/03/2013 06:15:32 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:32 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:32 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:32 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:32 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:32 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Unknown
05/03/2013 06:15:32 PM - bigjob - DEBUG - Candidate PJs: []
05/03/2013 06:15:32 PM - bigjob - DEBUG - No resource found.
05/03/2013 06:15:33 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:33 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:33 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:33 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:33 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:33 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Unknown
05/03/2013 06:15:33 PM - bigjob - DEBUG - Candidate PJs: []
05/03/2013 06:15:33 PM - bigjob - DEBUG - No resource found.
05/03/2013 06:15:34 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:34 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:34 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:34 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:34 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:34 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Unknown
05/03/2013 06:15:34 PM - bigjob - DEBUG - Candidate PJs: []
05/03/2013 06:15:34 PM - bigjob - DEBUG - No resource found.
05/03/2013 06:15:35 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:35 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:35 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:35 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:35 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:36 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Running
05/03/2013 06:15:36 PM - bigjob - DEBUG - Candidate PJs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:36 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 06:15:36 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost
05/03/2013 06:15:36 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost:jobs:sj-5c7db8b0-b447-11e2-b574-d4ae52a0ea54
05/03/2013 06:15:36 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x3637410>>
05/03/2013 06:15:36 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-5c7db8b0-b447-11e2-b574-d4ae52a0ea54', 'SPMDVariation': 'single'}
05/03/2013 06:15:36 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 06:15:37 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:37 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:37 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:37 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:37 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:37 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Running
05/03/2013 06:15:37 PM - bigjob - DEBUG - Candidate PJs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:37 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 06:15:37 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost
05/03/2013 06:15:37 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost:jobs:sj-5d63adca-b447-11e2-b574-d4ae52a0ea54
05/03/2013 06:15:37 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x3637590>>
05/03/2013 06:15:37 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-5d63adca-b447-11e2-b574-d4ae52a0ea54', 'SPMDVariation': 'single'}
05/03/2013 06:15:37 PM - bigjob - DEBUG - set job state to: Unknown
05/03/2013 06:15:38 PM - bigjob - DEBUG - Schedule CU
05/03/2013 06:15:38 PM - bigjob - DEBUG - __update_scheduler_resources
05/03/2013 06:15:38 PM - bigjob - DEBUG - Pilot-Jobs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:38 PM - bigjob - DEBUG - Schedule to PJ - # Avail PJs: 1
05/03/2013 06:15:38 PM - bigjob - DEBUG - B No pilot compute w/ affinity found... Looking for alternative pilot.
05/03/2013 06:15:38 PM - bigjob - DEBUG - BJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost State: Running
05/03/2013 06:15:39 PM - bigjob - DEBUG - Candidate PJs: [bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost]
05/03/2013 06:15:39 PM - bigjob - DEBUG - Submit CU to big-job
05/03/2013 06:15:39 PM - bigjob - DEBUG - add subjob to queue of PJ: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost
05/03/2013 06:15:39 PM - bigjob - DEBUG - create dictionary for job description. Job-URL: bigjob:bj-4ea7884c-b447-11e2-b156-d4ae52a0ea54:localhost:jobs:sj-5e528954-b447-11e2-b574-d4ae52a0ea54
05/03/2013 06:15:39 PM - bigjob - DEBUG - SJ Attributes: <class 'bigjob.bigjob_manager.description'> <bound method description.as_dict of <bigjob.bigjob_manager.description object at 0x3637710>>
05/03/2013 06:15:39 PM - bigjob - DEBUG - job dict: {'Executable': '/bin/sleep', 'NumberOfProcesses': 1, 'state': 'Unknown', 'Arguments': ['0'], 'Error': 'stderr.txt', 'Output': 'stdout.txt', 'job-id': 'sj-5e528954-b447-11e2-b574-d4ae52a0ea54', 'SPMDVariation': 'single'}
05/03/2013 06:15:39 PM - bigjob - DEBUG - set job state to: Unknown
oleweidner commented 11 years ago

Are you using saga-python 0.9.3 (installed with BigJob) or the latest develop branch?

oleweidner commented 11 years ago

Hi Ashley, I created a saga-python tickets for this: https://github.com/saga-project/saga-python/issues/105. Can you please investigate wether this is a problem with the SLURM adaptor itself. You can close the ticket if that is not the case and we can then continue debugging on the bigjob side.

ashleyz commented 11 years ago

Hi Ole,

I have seen this ticket and responded -- hopefully this will be patchable after getting some input on my response.

ashleyz commented 11 years ago

Fixed.