Closed Abdullah-Ghani closed 4 years ago
What is the error?
I suspect one of the main issues to be that Amarel requires users to be on the rutgers network or use a VPN to access the cluster.
(ve) abdullahg@js-156-107:~/ve/share/radical.pilot/examples$ python 00_getting_started.py
================================================================================
Getting Started (RP version 1.1.0)
================================================================================
new session: [rp.session.js-156-107.jetstream-cloud.org.abdullahg.018312.0000] \
database : [mongodb://rct:rct_test@two.radical-project.org/rct_test] ok
create pilot manager ok
create unit manager ok
--------------------------------------------------------------------------------
submit pilots
submit 1 pilot(s)
caught Exception: Resource domain 'rutgers' is unknown.
--------------
RADICAL Utils -- Stacktrace [11791] [MainThread]
abdulla+ 11791 11202 4 10:52 pts/18 00:00:00 \_ python 00_getting_started.py
Traceback (most recent call last):
File "00_getting_started.py", line 62, in <module>
pilot = pmgr.submit_pilots(pdesc)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/pilot_manager.py", line 551, in submit_pilots
pilot = ComputePilot(pmgr=self, descr=pd)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/compute_pilot.py", line 99, in __init__
= self._session._get_jsurl (pilot)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/session.py", line 927, in _get_jsurl
rcfg = self.get_resource_config(resrc, schema)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/session.py", line 648, in get_resource_config
raise RuntimeError("Resource domain '%s' is unknown." % domain)
RuntimeError: Resource domain 'rutgers' is unknown.
--------------
--------------------------------------------------------------------------------
finalize
closing session rp.session.js-156-107.jetstream-cloud.org.abdullahg.018312.0000
\
close unit manager ok
close pilot manager \
wait for 0 pilot(s)
0 ok
ok
+ rp.session.js-156-107.jetstream-cloud.org.abdullahg.018312.0000 (json)
session lifetime: 21.4s ok
Traceback (most recent call last):
File "00_getting_started.py", line 62, in <module>
pilot = pmgr.submit_pilots(pdesc)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/pilot_manager.py", line 551, in submit_pilots
pilot = ComputePilot(pmgr=self, descr=pd)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/compute_pilot.py", line 99, in __init__
= self._session._get_jsurl (pilot)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/session.py", line 927, in _get_jsurl
rcfg = self.get_resource_config(resrc, schema)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/session.py", line 648, in get_resource_config
raise RuntimeError("Resource domain '%s' is unknown." % domain)
RuntimeError: Resource domain 'rutgers' is unknown.
You can check your resource config with this command:
python -c 'import radical.pilot as rp; s=rp.Session(); print(s.get_resource_config("rutgers.amarel"))'
This will tell you that your resource config misses a comma at end of line 30 ("python_dist"
). You actually should also have seen an ERROR or WARNING message in the log files?
At this point RP really just complains about the resource not being known, it does not yet attempt to connect to Amarel, so thee VPN is not a problem (yet).
BTW, just checking the json syntax might be quicker with
python -c 'import radical.utils as ru; ru.read_json("/home/merzky/.radical/pilot/configs/resource_rutgers.json")'
but using the RP session for a check also confirms that RP finds the file in the expected location.
my apologies for that. after making sure it is able to read it accordingly, now its giving the could now resolve hostname error which I expected.
================================================================================
Getting Started (RP version 1.1.0)
================================================================================
new session: [rp.session.js-156-107.jetstream-cloud.org.abdullahg.018312.0006] \
database : [mongodb://rct:rct_test@two.radical-project.org/rct_test] ok
create pilot manager ok
create unit manager ok
--------------------------------------------------------------------------------
submit pilots
submit 1 pilot(s)
caught Exception: read from process failed '[Errno 5] Input/output error' : (ssh: Could not resolve hostname amarel.rutgers.edu: Name or service not known
) ((ssh: Could not resolve hostname amarel.rutgers.edu: Name or service not known
)) (/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_exceptions.py +40 (translate_exception) : if 'could not resolve hostname' in lmsg: e = se.BadParameter (cmsg))
--------------
RADICAL Utils -- Stacktrace [8722] [MainThread]
abdulla+ 8722 7146 4 16:10 pts/7 00:00:00 \_ python 00_getting_started.py
abdulla+ 9102 8722 0 16:10 ? 00:00:00 \_ [ssh] <defunct>
Traceback (most recent call last):
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_process.py", line 608, in read
buf = os.read (f, readsize)
OSError: [Errno 5] Input/output error
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_process.py", line 804, in find
data += self.read (timeout=_POLLDELAY)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_process.py", line 688, in read
% (e, self.tail))
radical.saga.exceptions.NoSuccess: read from process failed '[Errno 5] Input/output error' : (ssh: Could not resolve hostname amarel.rutgers.edu: Name or service not known
) (/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_process.py +688 (read) : % (e, self.tail)))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "00_getting_started.py", line 63, in <module>
pilot = pmgr.submit_pilots(pdesc)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/pilot_manager.py", line 551, in submit_pilots
pilot = ComputePilot(pmgr=self, descr=pd)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/compute_pilot.py", line 100, in __init__
self._resource_sandbox = self._session._get_resource_sandbox(pilot)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/session.py", line 772, in _get_resource_sandbox
shell = self.get_js_shell(resource, schema)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/session.py", line 818, in get_js_shell
shell = rsup.PTYShell(js_url, self)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_shell.py", line 245, in __init__
interactive=self.interactive)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_shell_factory.py", line 207, in initialize
self._initialize_pty(info['pty'], info)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_shell_factory.py", line 428, in _initialize_pty
raise ptye.translate_exception (e)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_shell_factory.py", line 274, in _initialize_pty
n, match = pty_shell.find (prompt_patterns, delay)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_process.py", line 807, in find
raise ptye.translate_exception (e, "(%s)" % data)
radical.saga.exceptions.BadParameter: read from process failed '[Errno 5] Input/output error' : (ssh: Could not resolve hostname amarel.rutgers.edu: Name or service not known
) ((ssh: Could not resolve hostname amarel.rutgers.edu: Name or service not known
)) (/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_exceptions.py +40 (translate_exception) : if 'could not resolve hostname' in lmsg: e = se.BadParameter (cmsg))
--------------
--------------------------------------------------------------------------------
finalize
closing session rp.session.js-156-107.jetstream-cloud.org.abdullahg.018312.0006
\
close unit manager ok
close pilot manager \
wait for 0 pilot(s)
0 ok
ok
+ rp.session.js-156-107.jetstream-cloud.org.abdullahg.018312.0006 (json)
session lifetime: 20.1s ok
Traceback (most recent call last):
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_process.py", line 608, in read
buf = os.read (f, readsize)
OSError: [Errno 5] Input/output error
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_process.py", line 804, in find
data += self.read (timeout=_POLLDELAY)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_process.py", line 688, in read
% (e, self.tail))
radical.saga.exceptions.NoSuccess: read from process failed '[Errno 5] Input/output error' : (ssh: Could not resolve hostname amarel.rutgers.edu: Name or service not known
) (/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_process.py +688 (read) : % (e, self.tail)))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "00_getting_started.py", line 63, in <module>
pilot = pmgr.submit_pilots(pdesc)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/pilot_manager.py", line 551, in submit_pilots
pilot = ComputePilot(pmgr=self, descr=pd)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/compute_pilot.py", line 100, in __init__
self._resource_sandbox = self._session._get_resource_sandbox(pilot)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/session.py", line 772, in _get_resource_sandbox
shell = self.get_js_shell(resource, schema)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/pilot/session.py", line 818, in get_js_shell
shell = rsup.PTYShell(js_url, self)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_shell.py", line 245, in __init__
interactive=self.interactive)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_shell_factory.py", line 207, in initialize
self._initialize_pty(info['pty'], info)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_shell_factory.py", line 428, in _initialize_pty
raise ptye.translate_exception (e)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_shell_factory.py", line 274, in _initialize_pty
n, match = pty_shell.find (prompt_patterns, delay)
File "/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_process.py", line 807, in find
raise ptye.translate_exception (e, "(%s)" % data)
radical.saga.exceptions.BadParameter: read from process failed '[Errno 5] Input/output error' : (ssh: Could not resolve hostname amarel.rutgers.edu: Name or service not known
) ((ssh: Could not resolve hostname amarel.rutgers.edu: Name or service not known
)) (/home/abdullahg/ve/local/lib/python3.7/site-packages/radical/saga/utils/pty_exceptions.py +40 (translate_exception) : if 'could not resolve hostname' in lmsg: e = se.BadParameter (cmsg))
my apologies for that.
No need to apologize - that is why we use issues :-)
Could not resolve hostname amarel.rutgers.edu
I am not sure how I can help with that. What is the normal way to access Amarel?
ssh but the machine ssh-ing needs to be on the rutgers network or be using a VPN.
Are you on the Rutgers network or are you using a VPN? :-)
ran it from my local machine on the Rutgers Network with passwordless SSH access. Process terminated by itself after long wait time.
$ python 00_getting_started.py
================================================================================
Getting Started (RP version 1.1.1)
================================================================================
new session: [rp.session.nbp-160-2.nbp.ruw.rutgers.edu.abdullahghani.018316.0001]
\
database : [mongodb://rct:rct_test@two.radical-project.org/rct_test] ok
create pilot manager ok
create unit manager ok
--------------------------------------------------------------------------------
submit pilots
submit 1 pilot(s)
[rutgers.amarel:2]
ok
--------------------------------------------------------------------------------
submit 1024 units
create: ########################################################################
submit: ########################################################################
wait : Terminated: 15
$ grep ERROR *.*
pmgr.0000.log:1582566477.828 : pmgr.0000 : 80767 : 123145553555456 : ERROR : [Callback]: pilot 'pilot.0000' failed - exit
pmgr.0000.log:1582566477.829 : pmgr.0000 : 80767 : 123145553555456 : ERROR : listener died
pmgr_launching.0000.log:1582566477.827 : pmgr_launching.0000 : 80814 : 123145406685184 : ERROR : bulk launch failed
$ radical-stack
python : 3.7.6
pythonpath :
virtualenv : /Users/abdullahghani/venv
radical.entk : 1.0.0
radical.pilot : 1.1.1
radical.saga : 1.1.2
radical.utils : 1.1.1
You found ERROR : bulk launch failed
which is likely the culprit - you should look into that file and see if there are more details to the ERROR log which help understanding the problem.
yes.
$ cat pmgr_launching.0000.log
1582566477.827 : pmgr_launching.0000 : 80814 : 123145406685184 : ERROR : bulk launch failed
Traceback (most recent call last):
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/pilot/pmgr/launching/default.py", line 553, in work
self._start_pilot_bulk(resource, schema, pilots)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/pilot/pmgr/launching/default.py", line 680, in _start_pilot_bulk
info = self._prepare_pilot(resource, rcfg, pilot, expand)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/pilot/pmgr/launching/default.py", line 994, in _prepare_pilot
raise RuntimeError("'global_virtenv' is deprecated (%s)" % resource)
RuntimeError: 'global_virtenv' is deprecated (rutgers.amarel)
RuntimeError: 'global_virtenv' is deprecated
You found the error :-) The documentation you used seems out of sync with the current resource configs. You may want to copy one of the existing resource configs (e.g., xsede.stampede2
) instead, and change them towards the Amarel setup. I'll open a separate ticket for the documentation fix.
this is my new edited config file
# filename: rutgers.json
{
"amarel":
{
"description" : "The Amarel HPC cluster at Rutgers.",
"notes" : "Access from registered IP address.",
"schemas" : ["ssh","local"],
"ssh" :
{
"job_manager_endpoint" : "slurm+ssh://aag193@amarel.rutgers.edu/",
"filesystem_endpoint" : "sftp://aag193@amarel.rutgers.edu/home/aag193/"
},
"local" :
{
"job_manager_endpoint" : "slurm://amarel.rutgers.edu:/home/aag193/",
"filesystem_endpoint" : "file://amarel.rutgers.edu:/home/aag193/"
},
"default_queue" : "normal",
"resource_manager" : "SLURM",
"agent_scheduler" : "CONTINUOUS",
"agent_spawner" : "POPEN",
"agent_launch_method" : "SSH",
"task_launch_method" : "SSH",
"mpi_launch_method" : "IBRUN",
"pre_bootstrap_0" :["module load intel/18.0.0",
"module load python3/3.7.0"
],
"default_remote_workdir" : "$WORK",
"valid_roots" : ["/scratch", "$SCRATCH", "/work", "$WORK"],
"rp_version" : "local",
"virtenv_mode" : "create",
"python_dist" : "default",
"export_to_cu" : ["LMOD_CMD",
"LMOD_SYSTEM_DEFAULT_MODULES",
"LD_LIBRARY_PATH"],
"cu_pre_exec" : ["module restore"]
}
}
Running the pilot example script gives me the following error:
$ cat pmgr_launching.0000.log
1582661910.540 : pmgr_launching.0000 : 98194 : 123145414258688 : ERROR : bulk launch failed
Traceback (most recent call last):
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/pilot/pmgr/launching/default.py", line 553, in work
self._start_pilot_bulk(resource, schema, pilots)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/pilot/pmgr/launching/default.py", line 755, in _start_pilot_bulk
fs.copy(tar_url, tar_rem, flags=rsfs.CREATE_PARENTS)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/saga/namespace/directory.py", line 354, in copy
if url_2: return self._adaptor.copy(url_1, url_2, flags, ttype=ttype)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/saga/adaptors/cpi/decorators.py", line 62, in wrap_function
return sync_function (self, *args, **kwargs)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/saga/adaptors/shell/shell_file.py", line 539, in copy
self._create_parent(cwdurl, tgt)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/saga/adaptors/shell/shell_file.py", line 242, in _create_parent
% (dirname, ret, out))
radical.saga.exceptions.NoSuccess: failed at mkdir '/radical.pilot.sandbox/rp.session.nbp-161-35.nbp.ruw.rutgers.edu.abdullahghani.018317.0005/': (1) (mkdir: cannot create directory ‘/radical.pilot.sandbox’: Permission denied
I added the file endpoint to point to my partition in the home directory but that didn't work. How do I change it.
Seems like "default_remote_workdir" : "$WORK"
is not valid - I would assume that $WORK
is not set by amarel, so this expands to ""
, and RP tries to create the sandbox base in '%s/radical.pilot.sandbox' % cfg.default_remote_workdir
This is my edited config file
# filename: rutgers.json
{
"amarel":
{
"description" : "The Amarel HPC cluster at Rutgers.",
"notes" : "Access from registered IP address.",
"schemas" : ["ssh","local"],
"ssh" :
{
"job_manager_endpoint" : "slurm+ssh://aag193@amarel.rutgers.edu/",
"filesystem_endpoint" : "sftp://aag193@amarel.rutgers.edu/home/aag193/"
},
"local" :
{
"job_manager_endpoint" : "slurm://amarel.rutgers.edu:/home/aag193/",
"filesystem_endpoint" : "file://amarel.rutgers.edu:/home/aag193/"
},
"default_queue" : "main",
"resource_manager" : "SLURM",
"agent_scheduler" : "CONTINUOUS",
"agent_spawner" : "POPEN",
"agent_launch_method" : "SSH",
"task_launch_method" : "SSH",
"mpi_launch_method" : "srun",
"pre_bootstrap_0" :["module load intel/17.0.4",
"module load python3/3.7.0"
],
"default_remote_workdir" : "/home/aag193",
"valid_roots" : ["/scratch", "$SCRATCH", "/home", "$HOME"],
"rp_version" : "local",
"virtenv_mode" : "create",
"python_dist" : "default",
"export_to_cu" : ["LMOD_CMD",
"LMOD_SYSTEM_DEFAULT_MODULES",
"LD_LIBRARY_PATH"],
"cu_pre_exec" : ["module restore"]
}
}
Running into some minor errors when running the script. Not sure what they mean.
$ grep ERROR *.*
grep: pilot.0000: Is a directory
pmgr.0000.log:1582738077.054 : pmgr.0000 : 8468 : 123145539309568 : ERROR : [Callback]: pilot 'pilot.0000' failed - exit
pmgr.0000.log:1582738077.054 : pmgr.0000 : 8468 : 123145539309568 : ERROR : listener died
rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0002.log:1582738227.192 : rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0002 : 8468 : 4637433280 : ERROR : failed to fet profile for pilot.0000
$ cat pmgr.0000.log
1582738077.054 : pmgr.0000 : 8468 : 123145539309568 : ERROR : [Callback]: pilot 'pilot.0000' failed - exit
1582738077.054 : pmgr.0000 : 8468 : 123145539309568 : ERROR : listener died
Traceback (most recent call last):
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/utils/zmq/pubsub.py", line 277, in _listener
cb(t, m)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/pilot/pilot_manager.py", line 312, in _state_sub_cb
if not self._update_pilot(thing, publish=False):
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/pilot/pilot_manager.py", line 353, in _update_pilot
self._pilots[pid]._update(pilot_dict)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/pilot/compute_pilot.py", line 205, in _update
else : cb(self, self.state)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/pilot/compute_pilot.py", line 148, in _default_state_cb
ru.cancel_main_thread('int')
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/utils/threads.py", line 136, in cancel_main_thread
if signame: signum = get_signal_by_name(signame)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/utils/threads.py", line 228, in get_signal_by_name
'cld' : signal.SIGCLD,
AttributeError: module 'signal' has no attribute 'SIGCLD'
(venv) nbp-160-2:rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0002 abdullahghani$ cat rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0002.log
1582738227.192 : rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0002 : 8468 : 4637433280 : ERROR : failed to fet profile for pilot.0000
Traceback (most recent call last):
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/pilot/utils/session.py", line 172, in fetch_profiles
profiles = sandbox.list('*.prof')
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/saga/namespace/directory.py", line 243, in list
return self._adaptor.list (pattern, flags, ttype=ttype)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/saga/adaptors/cpi/decorators.py", line 62, in wrap_function
return sync_function (self, *args, **kwargs)
File "/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/saga/adaptors/shell/shell_file.py", line 467, in list
% (ret, out))
radical.saga.exceptions.NoSuccess: failed to list(): (2)(/bin/ls: cannot access *.prof: No such file or directory
) (/Users/abdullahghani/venv/lib/python3.7/site-packages/radical/saga/adaptors/shell/shell_file.py +467 (list) : % (ret, out)))
AttributeError: module 'signal' has no attribute 'SIGCLD'
Alas, the minor errors are not minor :-P First, it seems Python deprecated a signal name we are still using (passively). I pushed an RU branch hotfix/sigcld_deprecation
which gets rid of this error. Your test will still die (but with more stile), because it seems like your pilot got into FAILED
state. So either you have an error in the launcher again, or you need to check the pilot sandbox you now use on Amarel to see why it failed.
Hi @mtitov : I added you to this ticket so you can tag along if you want. Abdullah goes through the procedure of getting the stack to work on a new resource, Amarel, and it clearly shows some of the painpoints and deficiencies we have in that context.
mistake on my end. the modules being loaded were different iterations. The script now runs on my side without errors.
This is the output on amarel/radical pilot sandbox
some modules/installations failed.
[aag193@amarel1 pilot.0000]$ cat bootstrap_0.out
bootstrap_0 stderr redirected to stdout
---------------------------------------------------------------------
bootstrap_0 running on host: slepner034.amarel.rutgers.edu.
bootstrap_0 started as : '/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/bootstrap_0.sh -d radical.utils-1.1.1.tar.gz:radical.saga-1.1.2.tar.gz:radical.pilot-1.1.1.tar.gz -p pilot.0000 -s rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003 -m create -r local -b default -g default -v /home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1 -y 60 -e module load intel/17.0.4 -e module load python/3.5.2'
Environment of bootstrap_0 process:
BASH_ENV=/opt/sw/admin/lmod/lmod/init/bash
BASH_FUNC_ml()=() { eval $($LMOD_DIR/ml_cmd "$@")
BASH_FUNC_module()=() { eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
ENVIRONMENT=BATCH
HISTCONTROL=ignoredups
HISTSIZE=1000
HOME=/home/aag193
HOSTNAME=slepner034.amarel.rutgers.edu
LANG=en_US.UTF-8
LESSOPEN=||/usr/bin/lesspipe.sh %s
LMOD_CMD=/opt/sw/admin/lmod/lmod/libexec/lmod
LMOD_DIR=/opt/sw/admin/lmod/lmod/libexec
LMOD_PKG=/opt/sw/admin/lmod/lmod
LMOD_SETTARG_FULL_SUPPORT=no
LMOD_sys=Linux
LMOD_VERSION=8.1.6
LOGNAME=aag193
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
MAIL=/var/spool/mail/aag193
MAKEFLAGS=-j
MANPATH=/opt/sw/admin/lmod/lmod/share/man:/usr/lpp/mmfs/share/man:
MODULEPATH=/opt/sw/modulefiles/Linux:/opt/sw/modulefiles/Core:/opt/sw/admin/lmod/lmod/modulefiles/Core
MODULEPATH_ROOT=/opt/sw/modulefiles
MODULESHOME=/opt/sw/admin/lmod/lmod
PATH=/home/aag193/anaconda3/bin:/home/aag193/anaconda3/bin:/usr/lpp/mmfs/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/aag193/.local/bin:/home/aag193/bin:/home/aag193/.local/bin:/home/aag193/bin
PS1=#
PWD=/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000
PYTHONNOUSERSITE=True
RADICAL_PROFILE=TRUE
RP_BOOTSTRAP_0_REDIR=True
SHELL=/bin/bash
SHLVL=3
SLURM_CHECKPOINT_IMAGE_DIR=/var/lib/slurm/checkpoint
SLURM_CLUSTER_NAME=amarel
SLURM_CPUS_ON_NODE=2
SLURM_CPUS_PER_TASK=1
SLURMD_NODENAME=slepner034
SLURM_GTIDS=0
SLURM_JOB_ACCOUNT=general
SLURM_JOB_CPUS_PER_NODE=2
SLURM_JOB_GID=140535
SLURM_JOB_ID=95175965
SLURM_JOBID=95175965
SLURM_JOB_NAME=pilot.0000
SLURM_JOB_NODELIST=slepner034
SLURM_JOB_NUM_NODES=1
SLURM_JOB_PARTITION=main
SLURM_JOB_QOS=normal
SLURM_JOB_UID=140535
SLURM_JOB_USER=aag193
SLURM_LOCALID=0
SLURM_MEM_PER_CPU=4096
SLURM_NNODES=1
SLURM_NODE_ALIASES=(null)
SLURM_NODEID=0
SLURM_NODELIST=slepner034
SLURM_NPROCS=2
SLURM_NTASKS=2
SLURM_PRIO_PROCESS=0
SLURM_PROCID=0
SLURM_SUBMIT_DIR=/cache/home/aag193
SLURM_SUBMIT_HOST=amarel1.amarel.rutgers.edu
SLURM_TASK_PID=37851
SLURM_TASKS_PER_NODE=2
SLURM_TOPOLOGY_ADDR=fdrc[0-1].edge2.slepner034
SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node
SLURM_WORKING_CLUSTER=amarel:saul:6808:8448
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
SSH_CLIENT=172.31.142.101 63876 22
SSH_CONNECTION=172.31.142.101 63876 172.16.94.35 22
SSH_TTY=/dev/pts/182
TERM=vt100
TMPDIR=/tmp
USER=aag193
_=/usr/bin/env
XDG_DATA_DIRS=/home/aag193/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
XDG_RUNTIME_DIR=/run/user/140535
XDG_SESSION_ID=52411
# -------------------------------------------------------------------
# -------------------------------------------------------------------
#
# Running pre_bootstrap_0 command
# cmd: module load intel/17.0.4
#
#
# SUCCESS
#
# -------------------------------------------------------------------
# -------------------------------------------------------------------
#
# Running pre_bootstrap_0 command
# cmd: module load python/3.5.2
#
#
# SUCCESS
#
# -------------------------------------------------------------------
# -------------------------------------------------------------------
# Touching output tarballs
# -------------------------------------------------------------------
create gtod
++ ./gtod
+ TIME_ZERO=1582740496.708426
+ export TIME_ZERO
+ set +x
+ test -z TRUE
+ PROFILE=bootstrap_0.prof
+ event=bootstrap_0_start
+ msg=
++ ./gtod
+ epoch=1582740496.759094
++ awk 'BEGIN{print(1582740496.759094 - 1582740496.708426)}'
+ now=0.050668
+ test -f bootstrap_0.prof
+ echo '#time,name,uid,state,event,msg'
+ printf '%.4f,%s,%s,%s,%s,%s,%s\n' 0.050668 bootstrap_0_start bootstrap_0 MainThread pilot.0000 PMGR_ACTIVE_PENDING ''
+ tee -a bootstrap_0.prof
0.0507,bootstrap_0_start,bootstrap_0,MainThread,pilot.0000,PMGR_ACTIVE_PENDING,
+ set +x
VIRTENV : /home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1
VIRTENV : /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1 (normalized)
PYTHON: /opt/sw/packages/gcc-4.8/python/3.5.2/bin/python
PIP : /opt/sw/packages/gcc-4.8/python/3.5.2/bin/pip
+ test -z TRUE
+ PROFILE=bootstrap_0.prof
+ event=ve_setup_start
+ msg=
++ ./gtod
+ epoch=1582740496.845515
++ awk 'BEGIN{print(1582740496.845515 - 1582740496.708426)}'
+ now=0.137089
+ test -f bootstrap_0.prof
+ printf '%.4f,%s,%s,%s,%s,%s,%s\n' 0.137089 ve_setup_start bootstrap_0 MainThread pilot.0000 PMGR_ACTIVE_PENDING ''
+ tee -a bootstrap_0.prof
0.1371,ve_setup_start,bootstrap_0,MainThread,pilot.0000,PMGR_ACTIVE_PENDING,
+ set +x
virtenv_create : TRUE
virtenv_update : FALSE
rp install sources: radical.utils-1.1.1/ radical.saga-1.1.2/ radical.pilot-1.1.1/
rp install target : SANDBOX
rp install lock : FALSE
rp lock for ve create
obtained lock /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1.lock
+ test -z TRUE
+ PROFILE=bootstrap_0.prof
+ event=ve_create_start
+ msg=
++ ./gtod
+ epoch=1582740498.491163
++ awk 'BEGIN{print(1582740498.491163 - 1582740496.708426)}'
+ now=1.78274
+ test -f bootstrap_0.prof
+ printf '%.4f,%s,%s,%s,%s,%s,%s\n' 1.78274 ve_create_start bootstrap_0 MainThread pilot.0000 PMGR_ACTIVE_PENDING ''
+ tee -a bootstrap_0.prof
1.7827,ve_create_start,bootstrap_0,MainThread,pilot.0000,PMGR_ACTIVE_PENDING,
+ set +x
# -------------------------------------------------------------------
#
# Download virtualenv tgz
# cmd: curl -1 -k -L -O 'https://files.pythonhosted.org/packages/66/f0/6867af06d2e2f511e4e1d7094ff663acdebc4f15d4a0cb0fed1007395124/virtualenv-16.7.5.tar.gz'
#
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4992k 100 4992k 0 0 10.0M 0 --:--:-- --:--:-- --:--:-- 10.0M
#
# SUCCESS
#
# -------------------------------------------------------------------
# -------------------------------------------------------------------
#
# unpacking virtualenv tgz
# cmd: tar zxmf 'virtualenv-16.7.5.tar.gz'
#
#
# SUCCESS
#
# -------------------------------------------------------------------
# -------------------------------------------------------------------
#
# Create virtualenv
# cmd: /opt/sw/packages/gcc-4.8/python/3.5.2/bin/python virtualenv-16.7.5/virtualenv.py -p python3 /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1
#
Already using interpreter /opt/sw/packages/gcc-4.8/python/3.5.2/bin/python3
Using base prefix '/opt/sw/packages/gcc-4.8/python/3.5.2'
New python executable in /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/python3
Also creating executable in /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /opt/sw/packages/gcc-4.8/python/3.5.2/bin/python3
#
# SUCCESS
#
# -------------------------------------------------------------------
+ test -z TRUE
+ PROFILE=bootstrap_0.prof
+ event=ve_activate_start
+ msg=
++ ./gtod
+ epoch=1582740508.094744
++ awk 'BEGIN{print(1582740508.094744 - 1582740496.708426)}'
+ now=11.3863
+ test -f bootstrap_0.prof
+ printf '%.4f,%s,%s,%s,%s,%s,%s\n' 11.3863 ve_activate_start bootstrap_0 MainThread pilot.0000 PMGR_ACTIVE_PENDING ''
+ tee -a bootstrap_0.prof
11.3863,ve_activate_start,bootstrap_0,MainThread,pilot.0000,PMGR_ACTIVE_PENDING,
+ set +x
PYTHON: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/python
PIP : /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip
PYTHON INTERPRETER: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/python
PYTHON_VERSION : 3.5
VE_MOD_PREFIX : /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/lib/python3.5/site-packages
PIP installer : /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip
PIP version : pip 20.0.2 from /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/lib/python3.5/site-packages/pip (python 3.5)
/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/bootstrap_0.sh: line 878: export: `/cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin:/cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin:/opt/sw/packages/gcc-4.8/python/3.5.2/bin:/opt/sw/packages/intel/17.0.4/compilers_and_libraries/linux/bin/intel64:/opt/sw/packages/intel/17.0.4/compilers_and_libraries/linux/mpi/intel64/bin:/opt/sw/packages/intel/17.0.4/debugger_2017/gdb/intel64_mic/bin:/home/aag193/anaconda3/bin:/usr/lpp/mmfs/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/aag193/.local/bin:/home/aag193/bin': not a valid identifier
activated virtenv
VIRTENV : /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1
VE_PYTHONPATH:
VE_MOD_PREFIX: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/lib/python3.5/site-packages
RP_MOD_PREFIX: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/rp_install/lib/python3.5/site-packages
RP_PATH : /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin
+ test -z TRUE
+ PROFILE=bootstrap_0.prof
+ event=ve_activate_stop
+ msg=
++ ./gtod
+ epoch=1582740508.599676
++ awk 'BEGIN{print(1582740508.599676 - 1582740496.708426)}'
+ now=11.8912
+ test -f bootstrap_0.prof
+ printf '%.4f,%s,%s,%s,%s,%s,%s\n' 11.8912 ve_activate_stop bootstrap_0 MainThread pilot.0000 PMGR_ACTIVE_PENDING ''
+ tee -a bootstrap_0.prof
11.8912,ve_activate_stop,bootstrap_0,MainThread,pilot.0000,PMGR_ACTIVE_PENDING,
+ set +x
PYTHON: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/python
PIP : /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip
# -------------------------------------------------------------------
#
# install pymongo
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip --no-cache-dir install --no-build-isolation pymongo
#
Collecting pymongo
Downloading pymongo-3.10.1-cp35-cp35m-manylinux2014_x86_64.whl (459 kB)
Installing collected packages: pymongo
Successfully installed pymongo-3.10.1
#
# SUCCESS
#
# -------------------------------------------------------------------
# -------------------------------------------------------------------
#
# install colorama
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip --no-cache-dir install --no-build-isolation colorama
#
Collecting colorama
Downloading colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Installing collected packages: colorama
Successfully installed colorama-0.4.3
#
# SUCCESS
#
# -------------------------------------------------------------------
# -------------------------------------------------------------------
#
# install python-hostlist
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip --no-cache-dir install --no-build-isolation python-hostlist
#
Collecting python-hostlist
Downloading python-hostlist-1.20.tar.gz (35 kB)
Building wheels for collected packages: python-hostlist
Building wheel for python-hostlist (setup.py): started
Building wheel for python-hostlist (setup.py): finished with status 'done'
Created wheel for python-hostlist: filename=python_hostlist-1.20-py3-none-any.whl size=38932 sha256=1ecaa08dddd6a69d6154408f2d5da4b84cbb947f8ba3bcf36e3d472fc855789f
Stored in directory: /tmp/pip-ephem-wheel-cache-r_8obzq6/wheels/aa/d8/53/ace748a3ae3fd460e6aaca884296ac3312d767f4d424a56ece
Successfully built python-hostlist
Installing collected packages: python-hostlist
Successfully installed python-hostlist-1.20
#
# SUCCESS
#
# -------------------------------------------------------------------
# -------------------------------------------------------------------
#
# install ntplib
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip --no-cache-dir install --no-build-isolation ntplib
#
Collecting ntplib
Downloading ntplib-0.3.3.tar.gz (6.8 kB)
Building wheels for collected packages: ntplib
Building wheel for ntplib (setup.py): started
Building wheel for ntplib (setup.py): finished with status 'done'
Created wheel for ntplib: filename=ntplib-0.3.3-py3-none-any.whl size=5909 sha256=4bfc3b58aea1c2f19245663807be6f56e77677a57d0385399821a588a3a2d512
Stored in directory: /tmp/pip-ephem-wheel-cache-775bwt7s/wheels/d7/72/2e/5d5aa67dd62f46f1d017c1f285fcf3dda579f3aefd37108d6e
Successfully built ntplib
Installing collected packages: ntplib
Successfully installed ntplib-0.3.3
#
# SUCCESS
#
# -------------------------------------------------------------------
# -------------------------------------------------------------------
#
# install pyzmq
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip --no-cache-dir install --no-build-isolation pyzmq
#
Collecting pyzmq
Downloading pyzmq-19.0.0-cp35-cp35m-manylinux1_x86_64.whl (1.1 MB)
Installing collected packages: pyzmq
Successfully installed pyzmq-19.0.0
#
# SUCCESS
#
# -------------------------------------------------------------------
# -------------------------------------------------------------------
#
# install netifaces
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip --no-cache-dir install --no-build-isolation netifaces
#
Collecting netifaces
Downloading netifaces-0.10.9-cp35-cp35m-manylinux1_x86_64.whl (32 kB)
Installing collected packages: netifaces
Successfully installed netifaces-0.10.9
#
# SUCCESS
#
# -------------------------------------------------------------------
# -------------------------------------------------------------------
#
# install setproctitle
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip --no-cache-dir install --no-build-isolation setproctitle
#
Collecting setproctitle
Downloading setproctitle-1.1.10.tar.gz (24 kB)
Building wheels for collected packages: setproctitle
Building wheel for setproctitle (setup.py): started
Building wheel for setproctitle (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-5wo5b6s9/setproctitle/setup.py'"'"'; __file__='"'"'/tmp/pip-install-5wo5b6s9/setproctitle/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-mc2xbenb
cwd: /tmp/pip-install-5wo5b6s9/setproctitle/
Complete output (10 lines):
running bdist_wheel
running build
running build_ext
building 'setproctitle' extension
creating build
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/src
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DHAVE_SYS_PRCTL_H=1 -DSPT_VERSION=1.1.10 -I/opt/sw/packages/gcc-4.8/python/3.5.2/include/python3.5m -c src/setproctitle.c -o build/temp.linux-x86_64-3.5/src/setproctitle.o
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for setproctitle
Running setup.py clean for setproctitle
Failed to build setproctitle
Installing collected packages: setproctitle
Running setup.py install for setproctitle: started
Running setup.py install for setproctitle: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-5wo5b6s9/setproctitle/setup.py'"'"'; __file__='"'"'/tmp/pip-install-5wo5b6s9/setproctitle/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-sembtny_/install-record.txt --single-version-externally-managed --compile --install-headers /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/include/site/python3.5/setproctitle
cwd: /tmp/pip-install-5wo5b6s9/setproctitle/
Complete output (10 lines):
running install
running build
running build_ext
building 'setproctitle' extension
creating build
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/src
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DSPT_VERSION=1.1.10 -DHAVE_SYS_PRCTL_H=1 -I/opt/sw/packages/gcc-4.8/python/3.5.2/include/python3.5m -c src/setproctitle.c -o build/temp.linux-x86_64-3.5/src/setproctitle.o
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-5wo5b6s9/setproctitle/setup.py'"'"'; __file__='"'"'/tmp/pip-install-5wo5b6s9/setproctitle/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-sembtny_/install-record.txt --single-version-externally-managed --compile --install-headers /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/include/site/python3.5/setproctitle Check the logs for full command output.
#
# ERROR
# no fallback command available
#
# -------------------------------------------------------------------
Couldn't install setproctitle! Lets see how far we get ...
# -------------------------------------------------------------------
#
# install msgpack
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip --no-cache-dir install --no-build-isolation msgpack
#
Collecting msgpack
Downloading msgpack-1.0.0-cp35-cp35m-manylinux1_x86_64.whl (270 kB)
Installing collected packages: msgpack
Successfully installed msgpack-1.0.0
#
# SUCCESS
#
# -------------------------------------------------------------------
# -------------------------------------------------------------------
#
# install future
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip --no-cache-dir install --no-build-isolation future
#
Collecting future
Downloading future-0.18.2.tar.gz (829 kB)
Building wheels for collected packages: future
Building wheel for future (setup.py): started
Building wheel for future (setup.py): finished with status 'done'
Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491058 sha256=569ddd1263b98a1195ade42307033c5968f997554a4cd0be2d56e2029d5bc006
Stored in directory: /tmp/pip-ephem-wheel-cache-w0n6hm0d/wheels/c4/f0/ae/d4689c4532d1f111462ed6a884a7767d502e511ee65f0d8e1b
Successfully built future
Installing collected packages: future
Successfully installed future-0.18.2
#
# SUCCESS
#
# -------------------------------------------------------------------
# -------------------------------------------------------------------
#
# install regex
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip --no-cache-dir install --no-build-isolation regex
#
Collecting regex
Downloading regex-2020.2.20.tar.gz (681 kB)
Building wheels for collected packages: regex
Building wheel for regex (setup.py): started
Building wheel for regex (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-h4u34czw/regex/setup.py'"'"'; __file__='"'"'/tmp/pip-install-h4u34czw/regex/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-v95siaj6
cwd: /tmp/pip-install-h4u34czw/regex/
Complete output (17 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.5
creating build/lib.linux-x86_64-3.5/regex
copying regex_3/__init__.py -> build/lib.linux-x86_64-3.5/regex
copying regex_3/regex.py -> build/lib.linux-x86_64-3.5/regex
copying regex_3/_regex_core.py -> build/lib.linux-x86_64-3.5/regex
copying regex_3/test_regex.py -> build/lib.linux-x86_64-3.5/regex
running build_ext
building 'regex._regex' extension
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/regex_3
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/sw/packages/gcc-4.8/python/3.5.2/include/python3.5m -c regex_3/_regex.c -o build/temp.linux-x86_64-3.5/regex_3/_regex.o
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for regex
Running setup.py clean for regex
Failed to build regex
Installing collected packages: regex
Running setup.py install for regex: started
Running setup.py install for regex: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-h4u34czw/regex/setup.py'"'"'; __file__='"'"'/tmp/pip-install-h4u34czw/regex/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-sw34taeq/install-record.txt --single-version-externally-managed --compile --install-headers /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/include/site/python3.5/regex
cwd: /tmp/pip-install-h4u34czw/regex/
Complete output (17 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.5
creating build/lib.linux-x86_64-3.5/regex
copying regex_3/__init__.py -> build/lib.linux-x86_64-3.5/regex
copying regex_3/regex.py -> build/lib.linux-x86_64-3.5/regex
copying regex_3/_regex_core.py -> build/lib.linux-x86_64-3.5/regex
copying regex_3/test_regex.py -> build/lib.linux-x86_64-3.5/regex
running build_ext
building 'regex._regex' extension
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/regex_3
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/sw/packages/gcc-4.8/python/3.5.2/include/python3.5m -c regex_3/_regex.c -o build/temp.linux-x86_64-3.5/regex_3/_regex.o
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-h4u34czw/regex/setup.py'"'"'; __file__='"'"'/tmp/pip-install-h4u34czw/regex/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-sw34taeq/install-record.txt --single-version-externally-managed --compile --install-headers /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/include/site/python3.5/regex Check the logs for full command output.
#
# ERROR
# no fallback command available
#
# -------------------------------------------------------------------
Couldn't install regex! Lets see how far we get ...
# -------------------------------------------------------------------
#
# install munch
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip --no-cache-dir install --no-build-isolation munch
#
Collecting munch
Downloading munch-2.5.0-py2.py3-none-any.whl (10 kB)
Collecting six
Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: six, munch
Successfully installed munch-2.5.0 six-1.14.0
#
# SUCCESS
#
# -------------------------------------------------------------------
+ test -z TRUE
+ PROFILE=bootstrap_0.prof
+ event=ve_create_stop
+ msg=
++ ./gtod
+ epoch=1582740529.117960
++ awk 'BEGIN{print(1582740529.117960 - 1582740496.708426)}'
+ now=32.4095
+ test -f bootstrap_0.prof
+ printf '%.4f,%s,%s,%s,%s,%s,%s\n' 32.4095 ve_create_stop bootstrap_0 MainThread pilot.0000 PMGR_ACTIVE_PENDING ''
+ tee -a bootstrap_0.prof
32.4095,ve_create_stop,bootstrap_0,MainThread,pilot.0000,PMGR_ACTIVE_PENDING,
+ set +x
removed ‘/cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1.lock’
+ test -z TRUE
+ PROFILE=bootstrap_0.prof
+ event=ve_activate_start
+ msg=
++ ./gtod
+ epoch=1582740529.172846
++ awk 'BEGIN{print(1582740529.172846 - 1582740496.708426)}'
+ now=32.4644
+ test -f bootstrap_0.prof
+ printf '%.4f,%s,%s,%s,%s,%s,%s\n' 32.4644 ve_activate_start bootstrap_0 MainThread pilot.0000 PMGR_ACTIVE_PENDING ''
+ tee -a bootstrap_0.prof
32.4644,ve_activate_start,bootstrap_0,MainThread,pilot.0000,PMGR_ACTIVE_PENDING,
+ set +x
do not update virtenv /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1
+ test -z TRUE
+ PROFILE=bootstrap_0.prof
+ event=rp_install_start
+ msg=
++ ./gtod
+ epoch=1582740529.211221
++ awk 'BEGIN{print(1582740529.211221 - 1582740496.708426)}'
+ now=32.5028
+ test -f bootstrap_0.prof
+ printf '%.4f,%s,%s,%s,%s,%s,%s\n' 32.5028 rp_install_start bootstrap_0 MainThread pilot.0000 PMGR_ACTIVE_PENDING ''
+ tee -a bootstrap_0.prof
32.5028,rp_install_start,bootstrap_0,MainThread,pilot.0000,PMGR_ACTIVE_PENDING,
+ set +x
Using RADICAL-Pilot install sources ' radical.utils-1.1.1/ radical.saga-1.1.2/ radical.pilot-1.1.1/'
VE_MOD_PREFIX: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/lib/python3.5/site-packages
VIRTENV : /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1
SANDBOX : /cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000
VE_LOC_PREFIX:
using local install tree
PYTHONPATH: /cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/lib/python3.5/site-packages::
rp_install: /cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/lib/python3.5/site-packages
radicalmod: /cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/lib/python3.5/site-packages/radical/
# -------------------------------------------------------------------
#
# update radical.utils-1.1.1/ via pip
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip install --src '/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/src' --build '/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/build' --install-option='--prefix=/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install' --no-deps --no-cache-dir --no-build-isolation radical.utils-1.1.1/
#
/cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/lib/python3.5/site-packages/pip/_internal/commands/install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
cmdoptions.check_install_build_global(options)
DEPRECATION: Location-changing options found in --install-option: ['--prefix'] from command line. This configuration may cause unexpected behavior and is unsupported. pip 20.2 will remove support for this functionality. A possible replacement is using pip-level options like --user, --prefix, --root, and --target. You can find discussion regarding this at https://github.com/pypa/pip/issues/7309.
Processing ./radical.utils-1.1.1
Skipping wheel build for radical.utils, due to binaries being disabled for it.
Installing collected packages: radical.utils
Running setup.py install for radical.utils: started
Running setup.py install for radical.utils: finished with status 'done'
Successfully installed radical.utils
#
# SUCCESS
#
# -------------------------------------------------------------------
purge install source at radical.utils-1.1.1/
# -------------------------------------------------------------------
#
# update radical.saga-1.1.2/ via pip
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip install --src '/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/src' --build '/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/build' --install-option='--prefix=/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install' --no-deps --no-cache-dir --no-build-isolation radical.saga-1.1.2/
#
/cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/lib/python3.5/site-packages/pip/_internal/commands/install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
cmdoptions.check_install_build_global(options)
DEPRECATION: Location-changing options found in --install-option: ['--prefix'] from command line. This configuration may cause unexpected behavior and is unsupported. pip 20.2 will remove support for this functionality. A possible replacement is using pip-level options like --user, --prefix, --root, and --target. You can find discussion regarding this at https://github.com/pypa/pip/issues/7309.
Processing ./radical.saga-1.1.2
Skipping wheel build for radical.saga, due to binaries being disabled for it.
Installing collected packages: radical.saga
Running setup.py install for radical.saga: started
Running setup.py install for radical.saga: finished with status 'done'
Successfully installed radical.saga
#
# SUCCESS
#
# -------------------------------------------------------------------
purge install source at radical.saga-1.1.2/
# -------------------------------------------------------------------
#
# update radical.pilot-1.1.1/ via pip
# cmd: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/pip install --src '/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/src' --build '/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/build' --install-option='--prefix=/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install' --no-deps --no-cache-dir --no-build-isolation radical.pilot-1.1.1/
#
/cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/lib/python3.5/site-packages/pip/_internal/commands/install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
cmdoptions.check_install_build_global(options)
DEPRECATION: Location-changing options found in --install-option: ['--prefix'] from command line. This configuration may cause unexpected behavior and is unsupported. pip 20.2 will remove support for this functionality. A possible replacement is using pip-level options like --user, --prefix, --root, and --target. You can find discussion regarding this at https://github.com/pypa/pip/issues/7309.
Processing ./radical.pilot-1.1.1
Skipping wheel build for radical.pilot, due to binaries being disabled for it.
Installing collected packages: radical.pilot
Running setup.py install for radical.pilot: started
Running setup.py install for radical.pilot: finished with status 'done'
Successfully installed radical.pilot
#
# SUCCESS
#
# -------------------------------------------------------------------
purge install source at radical.pilot-1.1.1/
+ test -z TRUE
+ PROFILE=bootstrap_0.prof
+ event=rp_install_stop
+ msg=
++ ./gtod
+ epoch=1582740538.442146
++ awk 'BEGIN{print(1582740538.442146 - 1582740496.708426)}'
+ now=41.7337
+ test -f bootstrap_0.prof
+ printf '%.4f,%s,%s,%s,%s,%s,%s\n' 41.7337 rp_install_stop bootstrap_0 MainThread pilot.0000 PMGR_ACTIVE_PENDING ''
+ tee -a bootstrap_0.prof
41.7337,rp_install_stop,bootstrap_0,MainThread,pilot.0000,PMGR_ACTIVE_PENDING,
+ set +x
+ test -z TRUE
+ PROFILE=bootstrap_0.prof
+ event=ve_setup_stop
+ msg=
++ ./gtod
+ epoch=1582740538.484094
++ awk 'BEGIN{print(1582740538.484094 - 1582740496.708426)}'
+ now=41.7757
+ test -f bootstrap_0.prof
+ printf '%.4f,%s,%s,%s,%s,%s,%s\n' 41.7757 ve_setup_stop bootstrap_0 MainThread pilot.0000 PMGR_ACTIVE_PENDING ''
+ tee -a bootstrap_0.prof
41.7757,ve_setup_stop,bootstrap_0,MainThread,pilot.0000,PMGR_ACTIVE_PENDING,
+ set +x
+ test -z TRUE
+ PROFILE=bootstrap_0.prof
+ event=ve_activate_start
+ msg=
++ ./gtod
+ epoch=1582740538.522479
++ awk 'BEGIN{print(1582740538.522479 - 1582740496.708426)}'
+ now=41.8141
+ test -f bootstrap_0.prof
+ printf '%.4f,%s,%s,%s,%s,%s,%s\n' 41.8141 ve_activate_start bootstrap_0 MainThread pilot.0000 PMGR_ACTIVE_PENDING ''
+ tee -a bootstrap_0.prof
41.8141,ve_activate_start,bootstrap_0,MainThread,pilot.0000,PMGR_ACTIVE_PENDING,
+ set +x
verify python viability: /cache/home/aag193/radical.pilot.sandbox/ve.rutgers.amarel.1.1.1/bin/python ... ok
verify module viability: radical.pilot ...Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/lib/python3.5/site-packages/radical/pilot/__init__.py", line 10, in <module>
import radical.utils as _ru
File "/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/lib/python3.5/site-packages/radical/utils/__init__.py", line 14, in <module>
from .plugin_manager import PluginManager
File "/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/lib/python3.5/site-packages/radical/utils/plugin_manager.py", line 15, in <module>
from .logger import Logger
File "/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/lib/python3.5/site-packages/radical/utils/logger.py", line 44, in <module>
from .misc import get_env_ns as ru_get_env_ns
File "/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/lib/python3.5/site-packages/radical/utils/misc.py", line 16, in <module>
from .ru_regex import ReString
File "/cache/home/aag193/radical.pilot.sandbox/rp.session.nbp-142-101.nbp.ruw.rutgers.edu.abdullahghani.018318.0003/pilot.0000/rp_install/lib/python3.5/site-packages/radical/utils/ru_regex.py", line 7, in <module>
import regex
ImportError: No module named 'regex'
failed
python installation cannot load module radical.pilot - abort
AttributeError: module 'signal' has no attribute 'SIGCLD'
Alas, the minor errors are not minor :-P First, it seems Python deprecated a signal name we are still using (passively). I pushed an RU branch
hotfix/sigcld_deprecation
which gets rid of this error. Your test will still die (but with more stile), because it seems like your pilot got intoFAILED
state. So either you have an error in the launcher again, or you need to check the pilot sandbox you now use on Amarel to see why it failed.
@andre-merzky I also went through other signals just in case, and see that SIGPOLL, SIGPWR, SIGRTMAX, SIGRTMIN are also deprecated (pls confirm that)
>>> dir(signal.Signals)
['SIGABRT', 'SIGALRM', 'SIGBUS', 'SIGCHLD', 'SIGCONT', 'SIGEMT', 'SIGFPE', 'SIGHUP', 'SIGILL', 'SIGINFO', 'SIGINT', 'SIGIO', 'SIGKILL', 'SIGPIPE', 'SIGPROF', 'SIGQUIT', 'SIGSEGV', 'SIGSTOP', 'SIGSYS', 'SIGTERM', 'SIGTRAP', 'SIGTSTP', 'SIGTTIN', 'SIGTTOU', 'SIGURG', 'SIGUSR1', 'SIGUSR2', 'SIGVTALRM', 'SIGWINCH', 'SIGXCPU', 'SIGXFSZ', '__class__', '__doc__', '__members__', '__module__']
specifically setproctitle
andregex
@mtitov : thanks for checking! But, WTH Python? SIGCLD
is not standardized, but SIGPOLL
for example is POSIX.2001, as are the SIGRTMIN/MAX
?! Pfft... Oh well, either way, would you mind removing those then, too?
@Abdullah-Ghani : if you don't mind, shorten the pasted parts to make the errors clearer - I would not have seen the regex
and setproctitle
errors... But to those, the message includes:
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
which is likely the exact problem: you current configuration does not have a functional compiler. You likely need a module load gnu
or something like that to activate a compiler chain acceptable to your Python module / setup.
@mtitov : thanks for checking! But, WTH Python?
SIGCLD
is not standardized, butSIGPOLL
for example is POSIX.2001, as are theSIGRTMIN/MAX
?! Pfft... Oh well, either way, would you mind removing those then, too?
@andre-merzky python 3.7.6 and yeah, will make an update for these signals as well
made great progress with @mtitov help. This is the edited config file.
# filename: rutgers.json
{
"amarel":
{
"description" : "The Amarel HPC cluster at Rutgers.",
"notes" : "Access from registered IP address.",
"schemas" : ["ssh","local"],
"ssh" :
{
"job_manager_endpoint" : "slurm+ssh://aag193@amarel.rutgers.edu/",
"filesystem_endpoint" : "sftp://aag193@amarel.rutgers.edu/home/aag193/"
},
"local" :
{
"job_manager_endpoint" : "slurm://amarel.rutgers.edu:/home/aag193/",
"filesystem_endpoint" : "file://amarel.rutgers.edu:/home/aag193/"
},
"default_queue" : "main",
"resource_manager" : "SLURM",
"agent_scheduler" : "CONTINUOUS",
"agent_spawner" : "POPEN",
"agent_launch_method" : "SSH",
"task_launch_method" : "SSH",
"mpi_launch_method" : "SRUN",
"pre_bootstrap_0" :["module load gcc/5.4",
"module load python/3.5.2",
"module load intel/17.0.4"
],
"default_remote_workdir" : "/home/aag193",
"valid_roots" : ["/scratch", "$SCRATCH", "/home", "$HOME"],
"rp_version" : "local",
"virtenv_mode" : "update",
"virtenv_dist" : "default",
"python_dist" : "default",
"export_to_cu" : ["LMOD_CMD",
"LMOD_SYSTEM_DEFAULT_MODULES",
"LD_LIBRARY_PATH"],
"cu_pre_exec" : ["module restore"]
}
}
Scripts runs completely with no problem.
However getting some permission problems on executing units.
aag193@amarel1 pilot.0000]$ cd unit.000000/
aag193@amarel1 unit.000000]$ ls
STDERR STDOUT unit.000000.sh
[aag193@amarel1 unit.000000]$ cat STDERR
Warning: Permanently added 'slepner012,192.168.8.12' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
[aag193@amarel1 unit.000000]$ pwd
/home/aag193/radical.pilot.sandbox/rp.session.vpn-client-172-16-9-130.rutgers.edu.abdullahghani.018320.0002/pilot.0000/unit.000000
Can you please try to setup ssh so that login to localhost
(i.e., the login node) works w/o password? You are setting agent_launch_method
and task_launch_method
to ssh, and I assume that this stumbles over the default configuration on amarel. On Slurm, you can also try to use SRUN
for both settings.
That works. Thank you
@Abdullah-Ghani @andre-merzky should we just provide a summary for this issue? (since it was resolved and this ticket can be closed after the summary?)
@Abdullah-Ghani : would you mind providing that summary (short)? Thanks!
Yes. Ofcourse :)
*If you are situated off campus you would need to install the pulse secure/Cisco AnyConnect VPN as Amarel only allows connection from RU-Wireless network.
Make sure you have passwordless SSH access set up to connect to amarel.
Place the config file I have attached below in the following directory. ~/.radical/pilot/configs/resource_rutgers.json
(If any of those directories do not exist, make sure you create them)
Change the resource description in the EnTK/Pilot script as such
# Apply the resource configuration provided by the user
res_dict = {
'resource': 'rutgers.amarel',
'walltime': 10,
'cpus': 2,
'schema': 'local'}
# Assign resource request description to the Application Manager
amgr.resource_desc = res_dict
*You can change the allocations to suit your needs.
Run the script. That should be it.
# filename: rutgers.json
{
"amarel":
{
"description" : "The Amarel HPC cluster at Rutgers.",
"notes" : "Access from registered IP address.",
"schemas" : ["ssh","local"],
"ssh" :
{
"job_manager_endpoint" : "slurm+ssh://amarel.rutgers.edu/",
"filesystem_endpoint" : "sftp://amarel.rutgers.edu/"
},
"local" :
{
"job_manager_endpoint" : "slurm://amarel.rutgers.edu:/home/",
"filesystem_endpoint" : "file://amarel.rutgers.edu:/home/"
},
"default_queue" : "main",
"resource_manager" : "SLURM",
"agent_scheduler" : "CONTINUOUS",
"agent_spawner" : "POPEN",
"agent_launch_method" : "SRUN",
"task_launch_method" : "SRUN",
"mpi_launch_method" : "SRUN",
"pre_bootstrap_0" :["export MODULEPATH=$MODULEPATH:/projects/community/modulefiles","module load gcc/5.4",
"module load py-data-science-stack/5.1.0-kp807",
"module load intel/17.0.4"
],
"default_remote_workdir" : "$HOME",
"valid_roots" : ["/scratch", "$SCRATCH", "/home", "$HOME"],
"rp_version" : "local",
"virtenv_mode" : "create",
"virtenv_dist" : "default",
"python_dist" : "default",
"export_to_cu" : ["LMOD_CMD",
"LMOD_SYSTEM_DEFAULT_MODULES",
"LD_LIBRARY_PATH"],
"cu_pre_exec" : []
}
}
I am trying to use radical pilot to run experiments on the Rutgers Amarel cluster. I used the documentation https://radicalpilot.readthedocs.io/en/devel/machconf.html#writing-a-custom-resource-configuration-file to create my own config file.
I saved it in
[aag193@amarel2 ~]$ vi .radical/pilot/configs/resource_rutgers.json
this is the file
but when I run the script on jet stream with the following resource allocation, it gives an error.
my resource allocation in my script