Closed brando90 closed 4 years ago
when through the whole history of releases none worked: https://pypi.org/project/torchmeta/#history
with command `pip install torchmeta=x.x.x'
My versions:
torchvision-base 0.4.2
pytorch 1.3.1 21932.g39c5d28 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pytorch-base 1.3.1 gpu_py37_21932.g39c5d28 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
any version that works with the above should be fine
when through the whole history of releases none worked: https://pypi.org/project/torchmeta/#history
with command `pip install torchmeta=x.x.x'
Isn't torchmeta==1.3.1
compatible with PyTorch 1.3 & Torchvision 0.4? I haven't tried it yet, but the requirements in setup.py
seem to match
https://github.com/tristandeleu/pytorch-meta/blob/5bc8638f1f640724a6f03aafd3b7606595d308d8/setup.py#L33-L34
ok seems you suggest pip install torchmeta==1.3.1
should work. I will try it again in a bit and report what happened.
when through the whole history of releases none worked: https://pypi.org/project/torchmeta/#history with command `pip install torchmeta=x.x.x'
Isn't
torchmeta==1.3.1
compatible with PyTorch 1.3 & Torchvision 0.4? I haven't tried it yet, but the requirements insetup.py
seem to match https://github.com/tristandeleu/pytorch-meta/blob/5bc8638f1f640724a6f03aafd3b7606595d308d8/setup.py#L33-L34
weird, I get this error:
$ pip install torchmeta==1.3.1
Defaulting to user installation because normal site-packages is not writeable
Collecting torchmeta==1.3.1
Using cached torchmeta-1.3.1-py3-none-any.whl (144 kB)
Requirement already satisfied: tqdm>=4.0.0 in /opt/apps/anaconda3/lib/python3.8/site-packages (from torchmeta==1.3.1) (4.47.0)
Collecting h5py~=2.9.0
Using cached h5py-2.9.0.tar.gz (287 kB)
ERROR: Could not find a version that satisfies the requirement torchvision<0.6.0,>=0.4.0 (from torchmeta==1.3.1) (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3)
ERROR: No matching distribution found for torchvision<0.6.0,>=0.4.0 (from torchmeta==1.3.1)
It says that torchmeta wants a version of torchvision less than 0.6.0
but more than 0.4.0
.
After looking more into it seems I have (under conda list
, nothing relevant under pip list
)
torchvision 0.2.2 pypi_0 pypi
will see if I can update it.
asking on forum how to force install the version of torchvision I need here: https://discuss.pytorch.org/t/force-installing-torchvision/97279/2
@tristandeleu what do you think of installing torchmeta and ignoring the dependencies?
e.g.
pip install --no-deps torchmeta
pip install --no-deps torchmeta
ok that didn't work without needed to modify torchmeta (to be expected):
ModuleNotFoundError: No module named 'ordered_set'
perhaps:
conda install torchvision==0.4.2 -c pytorch
or
conda install torchvision -c pytorch
might work...it's running right now.
about to try:
pip install torchmeta==1.3.1
seems torchvision installed...
but that gave an error:
pip install torchmeta==1.3.1
Collecting torchmeta==1.3.1
Using cached https://files.pythonhosted.org/packages/a2/78/2260a3f2a4c4653ba4de5485b32995a5ee73be93b27c29f8dd235459127e/torchmeta-1.3.1-py3-none-any.whl
ERROR: Could not find a version that satisfies the requirement torchvision<0.6.0,>=0.4.0 (from torchmeta==1.3.1) (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3)
ERROR: No matching distribution found for torchvision<0.6.0,>=0.4.0 (from torchmeta==1.3.1)
@tristandeleu what about this instead.
Is there a version of torchmeta that works with: 1) pytorch 1.3.1 + torchvision 0.2.2 2) pytorch 1.1.1 + torchvision 0.2.2
I am still struggling to get torchvision 0.4.2 so meanwhile those might work.
Unfortunately the earliest version of Torchvision required seems to be 0.4 (even for the earliest version of Torchmeta). If you install the dependencies manually (in particular h5py & tqdm), then you might be able to install Torchmeta with pip install --no-deps torchmeta==1.3.1
(what you tried earlier, but using a specific version). The versions of Torchvision will not match the requirements from Torchmeta, so I can't guarantee that the library will work as expected, but this is probably your best option.
Unfortunately the earliest version of Torchvision required seems to be 0.4 (even for the earliest version of Torchmeta). If you install the dependencies manually (in particular h5py & tqdm), then you might be able to install Torchmeta with
pip install --no-deps torchmeta==1.3.1
(what you tried earlier, but using a specific version). The versions of Torchvision will not match the requirements from Torchmeta, so I can't guarantee that the library will work as expected, but this is probably your best option.
I did try that. Torchmeta started complaining about some orderdict not being there and more. I will try that (more deeply) path later (fork torchmeta and fix it) after I get help from the IBM people/sys admins to install the required version of pytorch + torchvision on their fancy ppc64le hardware architecture.
Thanks for the help. I will post the solution here once I solve it.
For all details check (https://github.com/IBM/powerai/issues/268).
Make sure you have the right conda channel prepended:
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/#/
then install the powerai wmlce that you want e.g. 1.7.0 (most recent as of this writing):
conda create -n my_new_env python=3.7 powerai=1.7.0
conda activate my_new_env
though I am still unable to install torchmeta. Now there is some h5py installation failure on my ppc64le architecture:
...
building 'h5py.defs' extension
creating build/temp.linux-ppc64le-3.7
creating build/temp.linux-ppc64le-3.7/tmp
creating build/temp.linux-ppc64le-3.7/tmp/pip-install-xw5w7nuz
creating build/temp.linux-ppc64le-3.7/tmp/pip-install-xw5w7nuz/h5py
creating build/temp.linux-ppc64le-3.7/tmp/pip-install-xw5w7nuz/h5py/h5py
gcc -pthread -B /home/miranda9/.conda/envs/my_new_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DH5_USE_16_API -I./h5py -I/tmp/pip-install-xw5w7nuz/h5py/lzf -I/opt/local/include -I/usr/local/include -I/home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include -I/home/miranda9/.conda/envs/my_new_env/include/python3.7m -c /tmp/pip-install-xw5w7nuz/h5py/h5py/defs.c -o build/temp.linux-ppc64le-3.7/tmp/pip-install-xw5w7nuz/h5py/h5py/defs.o
In file included from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830:0,
from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/pip-install-xw5w7nuz/h5py/h5py/api_compat.h:26,
from /tmp/pip-install-xw5w7nuz/h5py/h5py/defs.c:654:
/home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it with " \
^
In file included from /tmp/pip-install-xw5w7nuz/h5py/h5py/defs.c:654:0:
/tmp/pip-install-xw5w7nuz/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
#include "hdf5.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Rolling back uninstall of h5py
Moving to /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/h5py
from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/~5py
Moving to /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/h5py-2.8.0-py3.7.egg-info
from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/~5py-2.8.0-py3.7.egg-info
ERROR: Command errored out with exit status 1: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xw5w7nuz/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xw5w7nuz/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-k4yqbpkb/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/.conda/envs/my_new_env/include/python3.7m/h5py Check the logs for full command output.
@tristandeleu I am getting some issues with h5py that torchmeta requires. Is this really required? do you see a way around this?
No unfortunately, h5py is a hard requirement because all the dataloaders are based on HDF5. You could probably still use some features of Torchmeta (such as the MetaModules and utility functions), but you would be missing out on all the dataloaders (which is one of the main features of the library).
@tristandeleu
I'm not sure why pip doesn't let me install torchmeta. The setup.py
says that any version except 2.9.0
:
install_requires=[
'torch>=1.3.0,<1.5.0',
'torchvision>=0.4.0,<0.6.0',
'numpy>=1.14.0',
'Pillow>=5.0.0,<7.0.0',
'h5py~=2.9.0',
'tqdm>=4.0.0',
'requests' # Required by Torchvision
]
so I am not sure why it's not installing it.
I will try to install it with the no dependencies flag and see what happens since just from reading your list it should be fine.
pip install --no-deps torchmeta==1.3.1
It seems I was wrong h5py
really isn't there. That seems to be my only issue now:
$ conda list
# packages in environment at /home/miranda9/.conda/envs/dawei:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_pytorch_select 2.0 gpu_21940.g1a3a219 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
absl-py 0.8.1 py37_0
apex 0.1.0_1.7.0 py37_662.gb1ba068 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
atomicwrites 1.4.0 py_0
attrs 20.2.0 py_0
blas 1.0 openblas
brotlipy 0.7.0 py37h7b6447c_1000
bzip2 1.0.8 h7b6447c_0
c-ares 1.16.1 h7b6447c_0
ca-certificates 2020.7.22 0
cairo 1.14.12 h8948797_3
certifi 2020.6.20 py37_0
cffi 1.12.3 py37h2e261b9_0
chardet 3.0.4 py37_1003
click 7.0 py37_0
cloudpickle 1.6.0 py_0
coverage 4.5.4 py37h7b6447c_0
cryptography 3.1 py37h1ba5d50_0
cudatoolkit 10.2.89 680.g0f7a43a https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
cudnn 7.6.5_10.2 650.g338a052 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
cxxfilt 0.2.0 py_622.gbc2955e https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
cycler 0.10.0 py37_0
cytoolz 0.10.1 py37h7b6447c_0
dask-core 2.27.0 py_0
ddl 1.5.1 py37_1364.ged89432 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
decorator 4.4.2 py_0
ffmpeg 4.2.2 h20bf706_0
fontconfig 2.13.0 h9420a91_0
freeglut 3.0.0 hf484d3e_5
freetype 2.9.1 h8a8886c_0
future 0.17.1 py37_0
giflib 5.1.4 h14c3975_1
glib 2.63.1 h5a9c865_0
gmp 6.1.2 h7f7056e_2
gnutls 3.6.5 h71b1129_1002
graphite2 1.3.14 h23475e2_0
graphsurgeon 0.4.1 py37_697.g9922bde https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
grpcio 1.16.1 py37hf8bcb03_1
harfbuzz 1.8.8 hffaf4a1_0
hdf5 1.10.2 hba1933b_1
hypothesis 3.59.1 py37h39e3cac_0
icu 58.2 he6710b0_3
idna 2.8 py37_0
imageio 2.9.0 py_0
importlib-metadata 1.7.0 py37_0
importlib_metadata 1.7.0 0
jasper 2.0.14 h07fcdf6_1
jpeg 9b hcb7ba68_2
kiwisolver 1.2.0 py37hfd86e86_0
lame 3.100 h7b6447c_0
ld_impl_linux-ppc64le 2.33.1 h0f24833_7
leveldb 1.20 hf484d3e_1
libedit 3.1.20191231 h14c3975_1
libffi 3.2.1 hf62a594_5
libgcc-ng 8.2.0 h822a55f_1
libgfortran-ng 7.3.0 h822a55f_1
libglu 9.0.0 hf484d3e_1
libopenblas 0.3.10 h5a2b251_0
libopencv 3.4.8 py37_784.g5a642ca https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
libopus 1.3.1 h7b6447c_0
libpng 1.6.36 hbc83047_0
libprotobuf 3.8.0 634.g08dc819 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
libstdcxx-ng 8.2.0 h822a55f_1
libtiff 4.1.0 h2733197_1
libuuid 1.0.3 h1bed415_2
libvpx 1.7.0 hf484d3e_0
libwebp 1.0.1 h8e7db2f_0
libxcb 1.14 h7b6447c_0
libxml2 2.9.10 he19cac6_1
llvmlite 0.31.0 py37hd408876_0
lmdb 0.9.22 hf484d3e_1
lz4-c 1.9.2 he6710b0_1
markdown 3.1.1 py37_0
matplotlib 3.0.3 py37h5429711_0
mock 3.0.5 py37_0
more-itertools 8.5.0 py_0
nccl 2.5.6 645.g51c2e94 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
ncurses 6.2 he6710b0_1
nettle 3.4.1 hbb512f6_0
networkx 2.3 py_0
ninja 1.9.0 py37hfd86e86_0
numactl 2.0.12 628.gb5e1afd https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
numba 0.47.0 py37h962f231_0
numpy 1.17.4 py37hd5be1e1_0
numpy-base 1.17.4 py37h2f8d375_0
olefile 0.46 py37_0
onnx 1.6.0 py37_671.g75d3229 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
opencv 3.4.8 py37_784.g5a642ca https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
openh264 2.1.0 hd408876_0
openssl 1.1.1h h7b6447c_0
pciutils 3.6.2 627.g804ec60 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pcre 8.44 he6710b0_0
pillow 7.1.2 py37haac5956_0
pip 20.2.2 py37_0
pixman 0.40.0 h7b6447c_0
pluggy 0.13.1 py37_0
powerai-license 1.7.0 778.g91a0fd9 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
powerai-release 1.7.0 627.g1c389a2 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
powerai-tools 1.7.0 623.g843ad38 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
protobuf 3.8.0 py37_642.gdc7b773 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
psutil 5.6.7 py37h7b6447c_0
py 1.9.0 py_0
py-opencv 3.4.8 py37_784.g5a642ca https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pycparser 2.20 py_2
pyopenssl 19.1.0 py_1
pyparsing 2.4.7 py_0
pysocks 1.7.1 py37_1
pytest 4.4.2 py37_0
python 3.7.7 ha29dc6b_0_cpython
python-dateutil 2.8.1 py_0
python-lmdb 0.98 py37he6710b0_0
pytorch 1.3.1 21940.g1a3a219 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pytorch-base 1.3.1 gpu_py37_21940.g1a3a219 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
pytz 2020.1 py_0
pywavelets 1.1.1 py37h7b6447c_0
pyyaml 5.3.1 py37h7b6447c_1
readline 8.0 h7b6447c_0
requests 2.22.0 py37_1
scikit-image 0.15.0 py37he6710b0_0
scipy 1.3.1 py37he2b7bc3_0
setuptools 49.6.0 py37_0
six 1.13.0 py37_0
snappy 1.1.8 he6710b0_0
spectrum-mpi 10.03 678.ga72dafb https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
sqlite 3.33.0 hbc83047_0
tabulate 0.8.2 py37_0
tbb 2020.3 hfd86e86_0
tensorboard 2.1.0 py37_3dc74fe_3959.g2372771 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tensorrt 7.0.0.11 py37_697.g9922bde https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tk 8.6.10 hbc83047_0
toolz 0.10.0 py_0
torchmeta 1.3.1 pypi_0 pypi
torchtext 0.4.0 py37_633.g16a90d7 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
torchvision 0.4.2 661.g37d5d4d https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
torchvision-base 0.4.2 gpu_py37_661.g37d5d4d https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
tornado 6.0.4 py37h7b6447c_1
tqdm 4.36.1 py_0
typing 3.6.4 py37_0
uff 0.6.5 py37_697.g9922bde https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
urllib3 1.25.10 py_0
werkzeug 0.16.0 py_0
wheel 0.35.1 py_0
x264 1!157.20191217 h7b6447c_0
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0
zipp 3.1.0 py_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.5 h9ceee32_0
@tristandeleu
Sorry I wish I could use the most recent version...
I am having this error with my code:
super().__init__(index=index, num_classes=None) # Regression task
TypeError: __init__() got an unexpected keyword argument 'index'
is it ok if I remove that index that the newer version requires?
old:
class Task(Dataset):
"""Base class for a classification task.
Parameters
----------
num_classes : int
Number of classes for the classification task.
"""
def __init__(self, num_classes, transform=None, target_transform=None):
super(Task, self).__init__(transform=transform,
target_transform=target_transform)
self.num_classes = num_classes
vs new:
class Task(Dataset):
"""Base class for a classification task.
Parameters
----------
num_classes : int
Number of classes for the classification task.
"""
def __init__(self, index, num_classes,
transform=None, target_transform=None):
super(Task, self).__init__(index, transform=transform,
target_transform=target_transform)
self.num_classes = num_classes
It should be ok to remove that, the only thing you'd be losing is some determinism (if you see the same task twice, then the datasets could contain different samples). Depending on the version you are rolling with, you might need to change the __hash__
method of Dataset
and SubsetTask
(you probably just generate a random number for the hash).
def __hash__(self):
return random.randrange(1 << 32)
to complement the close.
I eventually did get it to work by cloning the conda version of the appropriate binaries and doing everything from scratch. I don't think the commands I ran will be useful since they depend on my HPC's conda stuff but will paste them anyway:
module load wmlce/1.7.0-py3.7
module load wmlce/1.7.0-py3.7
create copy of wmlce env that HAL has
conda create --name=automl-meta-learning_wmlce-v1.7.0-py3.7 --clone=wmlce-v1.7.0-py3.7
conda activate automl-meta-learning_wmlce-v1.7.0-py3.7
conda remove --name xx --all
prepend the 1.7.0 code for wmlce
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/#/
pip install torchmeta==1.3.1
pip install --no-deps torchmeta==1.3.1
there is a way to get them using conda and the IBM channel but I forget how to do it now. Going through the other links above referenced here should eventually get them.
I got the following error:
unfortunately, the cluster I use has only 1.3.1 and I believe it's due to hardware stuff. They don't use x86 basically.
what are solutions for me?
SO: https://stackoverflow.com/questions/64049603/how-does-one-install-torchmeta-for-a-ppc64le-architecture-in-pytorch SO: https://stackoverflow.com/questions/64033543/force-installing-torchvision-0-4-2-when-i-am-forced-to-use-pytorch-1-3-1-due-to ibm issue: https://github.com/IBM/powerai/issues/268