search

radical-collaboration / extasy-bpti

0 stars 1 forks source link

Bluewaters ve_bw failure to build correctly #4

Open FranklinBetten opened 6 years ago

FranklinBetten commented 6 years ago

I am trying to run some ExTASY jobs on Bluewaters. These script works on Supermic and Comet.

JETSTREAM - Remote client - Software stack

(extasy-tools) hal9000@js-17-62:~/Documents/jha/bw_sync_01/bw_p14b02_left_d3_k12_1000_k34_1000_1ns_sims$ radical-stack 
python            : 2.7.12
virtualenv        : /home/hal9000/Documents/jha/extasy-tools
radical.utils     : 0.45
saga-python       : 0.45.1
radical.pilot     : 0.45.3

(extasy-tools) hal9000@js-17-62:~/Documents/jha/bw_sync_01/bw_p14b02_left_d3_k12_1000_k34_1000_1ns_sims$ ensemblemd-version 
0.4.6

BLUEWATERS - job details

radical.sandbox/rp.session.jobname/bootstrap_1.err

python 2.7.11
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named saga

RAN SOME TEST ON BLUEWATERS TO TEST OUT VE OPTIONS and ISSUES

VE TESTS ON BW

module switch PrgEnv-cray PrgEnv-gnu
module load bwpy
cd /tmp/

Test 1

virtualenv ve
source ve/bin/activate
(ve) bettenco@h2ologin4:/tmp> pip install saga-python

Building wheels for collected packages: saga-python, radical.utils, parse, future, netifaces, setproctitle
  Running setup.py bdist_wheel for saga-python ... done
  Stored in directory: /u/sciteam/bettenco/.cache/pip/wheels/cf/d5/83/ea24d52ef5c1bef07a0b2676dc3e911e88b905de1a82705563
  Running setup.py bdist_wheel for radical.utils ... done
  Stored in directory: /u/sciteam/bettenco/.cache/pip/wheels/03/5c/91/7f651eb722ff85ede654671c2169a3e4aa93363e36f509dbcd
  Running setup.py bdist_wheel for parse ... done
  Stored in directory: /u/sciteam/bettenco/.cache/pip/wheels/b0/3b/37/4ab694a0d331a3cc487923ff9c3645d0f103e9cd4762065f77
  Running setup.py bdist_wheel for future ... done
  Stored in directory: /u/sciteam/bettenco/.cache/pip/wheels/c2/50/7c/0d83b4baac4f63ff7a765bd16390d2ab43c93587fac9d6017a
  Running setup.py bdist_wheel for netifaces ... done
  Stored in directory: /u/sciteam/bettenco/.cache/pip/wheels/42/b4/d9/cd4ae93303efc2ba3441a21e2f2d74a49e07a5be9012d01520
  Running setup.py bdist_wheel for setproctitle ... done
  Stored in directory: /u/sciteam/bettenco/.cache/pip/wheels/5b/e6/ac/0a9f3ba8962551d270aa651226c7ccb561b3c3f94063cb2f27
Successfully built saga-python radical.utils parse future netifaces setproctitle
Installing collected packages: certifi, chardet, idna, urllib3, requests, apache-libcloud, future, colorama, netifaces, setproctitle, radical.utils, parse, saga-python
Successfully installed apache-libcloud-2.2.1 certifi-2017.11.5 chardet-3.0.4 colorama-0.3.9 future-0.16.0 idna-2.6 netifaces-0.10.4 parse-1.8.2 radical.utils-0.47 requests-2.18.4 saga-python-0.47 setproctitle-1.1.10 urllib3-1.22

Test 2

(ve2) bettenco@h2ologin4:/tmp> pip install -Iv saga-python==0.45.1

  Stored in directory: /u/sciteam/bettenco/.cache/pip/wheels/bf/e0/32/90bd785705007ccd2f04d81d6698f19b7337ebb6e16d247e7c
  Removing source in /tmp/pip-build-r6aNUK/saga-python
Successfully built saga-python
Installing collected packages: certifi, chardet, idna, urllib3, requests, apache-libcloud, netifaces, setproctitle, future, colorama, radical.utils, saga-python

  changing mode of /tmp/ve2/bin/chardetect to 755
  changing mode of /tmp/ve2/bin/pasteurize to 755
  changing mode of /tmp/ve2/bin/futurize to 755

Successfully installed apache-libcloud-2.2.1 certifi-2017.11.5 chardet-3.0.4 colorama-0.3.9 future-0.16.0 idna-2.6 netifaces-0.10.4 radical.utils-0.47 requests-2.18.4 saga-python-0.45.1 setproctitle-1.1.10 urllib3-1.22
Cleaning up...
(ve2) bettenco@h2ologin4:/tmp> radical-stack 

  python               : 2.7.11
  pythonpath           : /opt/xalt/0.7.6/sles11.3/libexec
  virtualenv           : /tmp/ve2

  radical.utils        : 0.47

(ve2) bettenco@h2ologin4:/tmp> sagapython-version 
Traceback (most recent call last):
  File "/tmp/ve2/bin/sagapython-version", line 3, in <module>
    import saga
  File "/tmp/ve2/lib/python2.7/site-packages/saga/__init__.py", line 46, in <module>
    version, version_detail, version_branch, sdist_name, sdist_path = ru.get_version ([root, pwd])
ValueError: too many values to unpack

Test 3

(ve3) bettenco@h2ologin4:/tmp> pip install -Iv radical.pilot==0.45.3 saga-python==0.45.1 radical.utils==0.45

  Stored in directory: /u/sciteam/bettenco/.cache/pip/wheels/51/8d/ed/21145cb835db97c9e1307b007e3be899d3e79d0651ba638dd4
  Removing source in /tmp/pip-build-LjWDCI/ntplib
Successfully built radical.pilot radical.utils pymongo python-hostlist ntplib
Installing collected packages: certifi, chardet, idna, urllib3, requests, apache-libcloud, colorama, netifaces, radical.utils, saga-python, pymongo, python-hostlist, setproctitle, ntplib, pyzmq, radical.pilot

changing mode of /tmp/ve3/bin/chardetect to 755

  Compiling /tmp/pip-build-LjWDCI/radical.pilot/radical.pilot-0.45.3.data/data/share/radical.pilot/examples/docs/chained_tasks.py ...
    File "/tmp/pip-build-LjWDCI/radical.pilot/radical.pilot-0.45.3.data/data/share/radical.pilot/examples/docs/chained_tasks.py", line 29
      RESOURCE_LABEL=
                    ^
  SyntaxError: invalid syntax

  Compiling /tmp/pip-build-LjWDCI/radical.pilot/radical.pilot-0.45.3.data/data/share/radical.pilot/examples/docs/coupled_tasks.py ...
    File "/tmp/pip-build-LjWDCI/radical.pilot/radical.pilot-0.45.3.data/data/share/radical.pilot/examples/docs/coupled_tasks.py", line 28
      RESOURCE_LABEL =
                     ^
  SyntaxError: invalid syntax

Successfully installed apache-libcloud-2.2.1 certifi-2017.11.5 chardet-3.0.4 colorama-0.3.9 idna-2.6 netifaces-0.10.4 ntplib-0.3.3 pymongo-2.8 python-hostlist-1.17 pyzmq-16.0.3 radical.pilot-0.45.3 radical.utils-0.45 requests-2.18.4 saga-python-0.45.1 setproctitle-1.1.10 urllib3-1.22
Cleaning up...
(ve3) bettenco@h2ologin4:/tmp> sagapython-version 
0.45.1
(ve3) bettenco@h2ologin4:/tmp> radical-stack 
python            : 2.7.11
virtualenv        : /tmp/ve3
radical.utils     : 0.45
saga-python       : 0.45.1
radical.pilot     : 0.45.3
(ve3) bettenco@h2ologin4:/tmp>

It looks like when rp tries to install the ve on bluewaters to run jobs it does the same thing as shown in the ve2 test. It appears that when installing saga-python for ve_bw it only specifies saga-python==45.1 but does not specify radical.pilot==0.45.3 and instead tries to install radical.utils 0.47 which causes an error.

andre-merzky commented 6 years ago

radical-cybertools/radical.pilot/issues/1546 seems solved by now - Franklin, are you ok with that, or are there still open issues?