IBM / pytorch-large-model-support

Large Model Support in PyTorch
Apache License 2.0
131 stars 19 forks source link

Error when using torch.cuda.set_enabled_lms(True) #2

Closed NoAchache closed 4 years ago

NoAchache commented 4 years ago

I get the following error when running torch.cuda.set_enabled_lms(True): AttributeError: module 'torch.cuda' has no attribute 'set_enabled_lms'

Steps to reproduce the error:

As specified in the WML CE install documentation:

When running torch.cuda.set_enabled_lms(True), my conda env uses Python 3.7, pytorch 1.3.1 and powerai 1.7.0

System: Ubuntu 20.04

jayfurmanek commented 4 years ago

Can you post the output of conda list and conda info ?

thanks!

NoAchache commented 4 years ago

Sure!

Conda list:

# packages in environment at /home/noe/anaconda3:
#
# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0                    py37_0  
_libgcc_mutex             0.1                        main  
_pytorch_select           0.2                       gpu_0  
alabaster                 0.7.12                   py37_0  
anaconda                  2020.02                  py37_0  
anaconda-client           1.7.2                    py37_0  
anaconda-navigator        1.9.12                   py37_0  
anaconda-project          0.8.4                      py_0  
argh                      0.26.2                   py37_0  
asn1crypto                1.3.0                    py37_0  
astroid                   2.3.3                    py37_0  
astropy                   4.0              py37h7b6447c_0  
atomicwrites              1.3.0                    py37_1  
attrs                     19.3.0                     py_0  
autopep8                  1.4.4                      py_0  
babel                     2.8.0                      py_0  
backcall                  0.1.0                    py37_0  
backports                 1.0                        py_2  
backports.functools_lru_cache 1.6.1                      py_0  
backports.shutil_get_terminal_size 1.0.0                    py37_2  
backports.tempfile        1.0                        py_1  
backports.weakref         1.0.post1                  py_1  
beautifulsoup4            4.8.2                    py37_0  
bitarray                  1.2.1            py37h7b6447c_0  
bkcharts                  0.2                      py37_0  
blas                      1.0                         mkl  
bleach                    3.1.0                    py37_0  
blosc                     1.16.3               hd408876_0  
bokeh                     1.4.0                    py37_0  
boto                      2.49.0                   py37_0  
bottleneck                1.3.2            py37heb32a55_0  
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2020.1.1                      0  
cairo                     1.14.12              h8948797_3  
certifi                   2019.11.28               py37_0  
cffi                      1.14.0           py37h2e261b9_0  
chardet                   3.0.4                 py37_1003  
click                     7.0                      py37_0  
cloudpickle               1.3.0                      py_0  
clyent                    1.2.2                    py37_1  
colorama                  0.4.3                      py_0  
conda                     4.8.3                    py37_0  
conda-build               3.18.11                  py37_0  
conda-env                 2.6.0                         1  
conda-package-handling    1.6.1            py37h7b6447c_0  
conda-verify              3.4.2                      py_1  
contextlib2               0.6.0.post1                py_0  
cryptography              2.8              py37h1ba5d50_0  
cudatoolkit               10.0.130                      0  
cudnn                     7.6.5                cuda10.0_0  
curl                      7.68.0               hbc83047_0  
cycler                    0.10.0                   py37_0  
cython                    0.29.15          py37he6710b0_0  
cytoolz                   0.10.1           py37h7b6447c_0  
dask                      2.11.0                     py_0  
dask-core                 2.11.0                     py_0  
dbus                      1.13.12              h746ee38_0  
decorator                 4.4.1                      py_0  
defusedxml                0.6.0                      py_0  
diff-match-patch          20181111                   py_0  
distributed               2.11.0                   py37_0  
docutils                  0.16                     py37_0  
entrypoints               0.3                      py37_0  
et_xmlfile                1.0.1                    py37_0  
expat                     2.2.6                he6710b0_0  
fastcache                 1.1.0            py37h7b6447c_0  
filelock                  3.0.12                     py_0  
flake8                    3.7.9                    py37_0  
flask                     1.1.1                      py_0  
fontconfig                2.13.0               h9420a91_0  
freetype                  2.9.1                h8a8886c_1  
fribidi                   1.0.5                h7b6447c_0  
fsspec                    0.6.2                      py_0  
future                    0.18.2                   py37_0  
get_terminal_size         1.0.0                haa9412d_0  
gevent                    1.4.0            py37h7b6447c_0  
glib                      2.63.1               h5a9c865_0  
glob2                     0.7                        py_0  
gmp                       6.1.2                h6c8ec71_1  
gmpy2                     2.0.8            py37h10f8cd9_2  
graphite2                 1.3.13               h23475e2_0  
greenlet                  0.4.15           py37h7b6447c_0  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
h5py                      2.10.0           py37h7918eee_0  
harfbuzz                  1.8.8                hffaf4a1_0  
hdf5                      1.10.4               hb1b8bf9_0  
heapdict                  1.0.1                      py_0  
html5lib                  1.0.1                    py37_0  
hypothesis                5.5.4                      py_0  
icu                       58.2                 h9c2bf20_1  
idna                      2.8                      py37_0  
imageio                   2.6.1                    py37_0  
imagesize                 1.2.0                      py_0  
importlib_metadata        1.5.0                    py37_0  
intel-openmp              2020.0                      166  
intervaltree              3.0.2                      py_0  
ipykernel                 5.1.4            py37h39e3cac_0  
ipython                   7.12.0           py37h5ca1d4c_0  
ipython_genutils          0.2.0                    py37_0  
ipywidgets                7.5.1                      py_0  
isort                     4.3.21                   py37_0  
itsdangerous              1.1.0                    py37_0  
jbig                      2.1                  hdba287a_0  
jdcal                     1.4.1                      py_0  
jedi                      0.14.1                   py37_0  
jeepney                   0.4.2                      py_0  
jinja2                    2.11.1                     py_0  
joblib                    0.14.1                     py_0  
jpeg                      9b                   h024ee3a_2  
json5                     0.9.1                      py_0  
jsonschema                3.2.0                    py37_0  
jupyter                   1.0.0                    py37_7  
jupyter_client            5.3.4                    py37_0  
jupyter_console           6.1.0                      py_0  
jupyter_core              4.6.1                    py37_0  
jupyterlab                1.2.6              pyhf63ae98_0  
jupyterlab_server         1.0.6                      py_0  
keyring                   21.1.0                   py37_0  
kiwisolver                1.1.0            py37he6710b0_0  
krb5                      1.17.1               h173b8e3_0  
lazy-object-proxy         1.4.3            py37h7b6447c_0  
ld_impl_linux-64          2.33.1               h53a641e_7  
libarchive                3.3.3                h5d8350f_5  
libcurl                   7.68.0               h20c2e04_0  
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
liblief                   0.9.0                h7725739_2  
libpng                    1.6.37               hbc83047_0  
libsodium                 1.0.16               h1bed415_0  
libspatialindex           1.9.3                he6710b0_0  
libssh2                   1.8.2                h1ba5d50_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
libtiff                   4.1.0                h2733197_0  
libtool                   2.4.6                h7b6447c_5  
libuuid                   1.0.3                h1bed415_2  
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.9                hea5a465_1  
libxslt                   1.1.33               h7d1a2b0_0  
llvmlite                  0.31.0           py37hd408876_0  
locket                    0.2.0                    py37_1  
lxml                      4.5.0            py37hefd8a0e_0  
lz4-c                     1.8.1.2              h14c3975_0  
lzo                       2.10                 h49e0be7_2  
markupsafe                1.1.1            py37h7b6447c_0  
matplotlib                3.1.3                    py37_0  
matplotlib-base           3.1.3            py37hef1b27d_0  
mccabe                    0.6.1                    py37_1  
mistune                   0.8.4            py37h7b6447c_0  
mkl                       2020.0                      166  
mkl-service               2.3.0            py37he904b0f_0  
mkl_fft                   1.0.15           py37ha843d7b_0  
mkl_random                1.1.0            py37hd6b4f25_0  
mock                      4.0.1                      py_0  
more-itertools            8.2.0                      py_0  
mpc                       1.1.0                h10f8cd9_1  
mpfr                      4.0.1                hdf1c602_3  
mpmath                    1.1.0                    py37_0  
msgpack-python            0.6.1            py37hfd86e86_1  
multipledispatch          0.6.0                    py37_0  
navigator-updater         0.2.1                    py37_0  
nbconvert                 5.6.1                    py37_0  
nbformat                  5.0.4                      py_0  
ncurses                   6.2                  he6710b0_0  
networkx                  2.4                        py_0  
ninja                     1.9.0            py37hfd86e86_0  
nltk                      3.4.5                    py37_0  
nose                      1.3.7                    py37_2  
notebook                  6.0.3                    py37_0  
numba                     0.48.0           py37h0573a6f_0  
numexpr                   2.7.1            py37h423224d_0  
numpy                     1.18.1           py37h4f9e942_0  
numpy-base                1.18.1           py37hde5b4d6_1  
numpydoc                  0.9.2                      py_0  
olefile                   0.46                     py37_0  
openpyxl                  3.0.3                      py_0  
openssl                   1.1.1d               h7b6447c_4  
packaging                 20.1                       py_0  
pandas                    1.0.1            py37h0573a6f_0  
pandoc                    2.2.3.2                       0  
pandocfilters             1.4.2                    py37_1  
pango                     1.42.4               h049681c_0  
parso                     0.5.2                      py_0  
partd                     1.1.0                      py_0  
patchelf                  0.10                 he6710b0_0  
path                      13.1.0                   py37_0  
path.py                   12.4.0                        0  
pathlib2                  2.3.5                    py37_0  
pathtools                 0.1.2                      py_1  
patsy                     0.5.1                    py37_0  
pcre                      8.43                 he6710b0_0  
pep8                      1.7.1                    py37_0  
pexpect                   4.8.0                    py37_0  
pickleshare               0.7.5                    py37_0  
pillow                    7.0.0            py37hb39fc2d_0  
pip                       20.0.2                   py37_1  
pixman                    0.38.0               h7b6447c_0  
pkginfo                   1.5.0.1                  py37_0  
pluggy                    0.13.1                   py37_0  
ply                       3.11                     py37_0  
powerai-license           1.7.0              778.g91a0fd9    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
powerai-release           1.7.0              627.g1c389a2    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
powerai-tools             1.7.0              623.g843ad38    https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
prometheus_client         0.7.1                      py_0  
prompt_toolkit            3.0.3                      py_0  
psutil                    5.6.7            py37h7b6447c_0  
ptyprocess                0.6.0                    py37_0  
py                        1.8.1                      py_0  
py-lief                   0.9.0            py37h7725739_2  
pycodestyle               2.5.0                    py37_0  
pycosat                   0.6.3            py37h7b6447c_0  
pycparser                 2.19                     py37_0  
pycrypto                  2.6.1            py37h14c3975_9  
pycurl                    7.43.0.5         py37h1ba5d50_0  
pydocstyle                4.0.1                      py_0  
pyflakes                  2.1.1                    py37_0  
pygments                  2.5.2                      py_0  
pylint                    2.4.4                    py37_0  
pyodbc                    4.0.30           py37he6710b0_0  
pyopenssl                 19.1.0                   py37_0  
pyparsing                 2.4.6                      py_0  
pyqt                      5.9.2            py37h05f1152_2  
pyrsistent                0.15.7           py37h7b6447c_0  
pysocks                   1.7.1                    py37_0  
pytables                  3.6.1            py37h71ec239_0  
pytest                    5.3.5                    py37_0  
pytest-arraydiff          0.3              py37h39e3cac_0  
pytest-astropy            0.8.0                      py_0  
pytest-astropy-header     0.1.2                      py_0  
pytest-doctestplus        0.5.0                      py_0  
pytest-openfiles          0.4.0                      py_0  
pytest-remotedata         0.3.2                    py37_0  
python                    3.7.6                h0371630_2  
python-dateutil           2.8.1                      py_0  
python-jsonrpc-server     0.3.4                      py_0  
python-language-server    0.31.7                   py37_0  
python-libarchive-c       2.8                     py37_13  
pytorch                   1.3.1           cuda100py37h53c1284_0  
pytz                      2019.3                     py_0  
pywavelets                1.1.1            py37h7b6447c_0  
pyxdg                     0.26                       py_0  
pyyaml                    5.3              py37h7b6447c_0  
pyzmq                     18.1.1           py37he6710b0_0  
qdarkstyle                2.8                        py_0  
qt                        5.9.7                h5867ecd_1  
qtawesome                 0.6.1                      py_0  
qtconsole                 4.6.0                      py_1  
qtpy                      1.9.0                      py_0  
readline                  7.0                  h7b6447c_5  
requests                  2.22.0                   py37_1  
ripgrep                   11.0.2               he32d670_0  
rope                      0.16.0                     py_0  
rtree                     0.9.3                    py37_0  
ruamel_yaml               0.15.87          py37h7b6447c_0  
scikit-image              0.16.2           py37h0573a6f_0  
scikit-learn              0.22.1           py37hd81dba3_0  
scipy                     1.4.1            py37h0b6359f_0  
seaborn                   0.10.0                     py_0  
secretstorage             3.1.2                    py37_0  
send2trash                1.5.0                    py37_0  
setuptools                45.2.0                   py37_0  
simplegeneric             0.8.1                    py37_2  
singledispatch            3.4.0.3                  py37_0  
sip                       4.19.8           py37hf484d3e_0  
six                       1.14.0                   py37_0  
snappy                    1.1.7                hbae5bb6_3  
snowballstemmer           2.0.0                      py_0  
sortedcollections         1.1.2                    py37_0  
sortedcontainers          2.1.0                    py37_0  
soupsieve                 1.9.5                    py37_0  
sphinx                    2.4.0                      py_0  
sphinxcontrib             1.0                      py37_1  
sphinxcontrib-applehelp   1.0.1                      py_0  
sphinxcontrib-devhelp     1.0.1                      py_0  
sphinxcontrib-htmlhelp    1.0.2                      py_0  
sphinxcontrib-jsmath      1.0.1                      py_0  
sphinxcontrib-qthelp      1.0.2                      py_0  
sphinxcontrib-serializinghtml 1.1.3                      py_0  
sphinxcontrib-websupport  1.2.0                      py_0  
spyder                    4.0.1                    py37_0  
spyder-kernels            1.8.1                    py37_0  
sqlalchemy                1.3.13           py37h7b6447c_0  
sqlite                    3.31.1               h7b6447c_0  
statsmodels               0.11.0           py37h7b6447c_0  
sympy                     1.5.1                    py37_0  
tbb                       2020.0               hfd86e86_0  
tblib                     1.6.0                      py_0  
terminado                 0.8.3                    py37_0  
testpath                  0.4.4                      py_0  
tk                        8.6.8                hbc83047_0  
toolz                     0.10.0                     py_0  
tornado                   6.0.3            py37h7b6447c_3  
tqdm                      4.42.1                     py_0  
traitlets                 4.3.3                    py37_0  
ujson                     1.35             py37h14c3975_0  
unicodecsv                0.14.1                   py37_0  
unixodbc                  2.3.7                h14c3975_0  
urllib3                   1.25.8                   py37_0  
watchdog                  0.10.2                   py37_0  
wcwidth                   0.1.8                      py_0  
webencodings              0.5.1                    py37_1  
werkzeug                  1.0.0                      py_0  
wheel                     0.34.2                   py37_0  
widgetsnbextension        3.5.1                    py37_0  
wrapt                     1.11.2           py37h7b6447c_0  
wurlitzer                 2.0.0                    py37_0  
xlrd                      1.2.0                    py37_0  
xlsxwriter                1.2.7                      py_0  
xlwt                      1.3.0                    py37_0  
xmltodict                 0.12.0                     py_0  
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7                had09818_2  
yapf                      0.28.0                     py_0  
zeromq                    4.3.1                he6710b0_3  
zict                      1.0.0                      py_0  
zipp                      2.2.0                      py_0  
zlib                      1.2.11               h7b6447c_3  
zstd                      1.3.7                h0b5b093_0  

Conda info:

     active environment : base
    active env location : /home/noe/anaconda3
            shell level : 1
       user config file : /home/noe/.condarc
 populated config files : /home/noe/.condarc
          conda version : 4.8.3
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages : __cuda=10.2
                          __glibc=2.31
       base environment : /home/noe/anaconda3  (writable)
           channel URLs : https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/linux-64
                          https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/noe/anaconda3/pkgs
                          /home/noe/.conda/pkgs
       envs directories : /home/noe/anaconda3/envs
                          /home/noe/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.3 requests/2.22.0 CPython/3.7.6 Linux/5.4.0-37-generic ubuntu/20.04 glibc/2.31
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
jayfurmanek commented 4 years ago

It looks like your pytorch package came from defaults. Try using strict channel priority and then re-installing Pytorch, which can be set in one of two ways:

NoAchache commented 4 years ago

Thank you, it's working