search

NVlabs / few-shot-vid2vid

Pytorch implementation for few-shot photorealistic video-to-video translation.
Other
1.79k stars 276 forks source link

TypeError: 'numpy.float64' object cannot be interpreted as an integer #31

Closed nalzok closed 3 years ago

nalzok commented 4 years ago

I am on Ubuntu 18.04. It seems that there are some issues with DataLoader?

(few-shot-new) ~/few-shot-vid2vid$ bash ./scripts/pose/train_g1.sh
CustomDatasetDataLoader
2 sequences
dataset [PoseDataset] was created
5000
Resuming from epoch 1 at iteration 0
create web directory ./checkpoints/pose/web...
---------- Networks initialized -------------
---------- Optimizers initialized -------------
./checkpoints/pose/latest_net_G.pth not exists yet!
./checkpoints/pose/latest_net_D.pth not exists yet!
./checkpoints/pose/latest_net_Df.pth not exists yet!
model [Vid2VidModel] was created
Traceback (most recent call last):
  File "train.py", line 73, in <module>
    train()
  File "train.py", line 40, in train
    for idx, data in enumerate(dataset, start=trainer.epoch_iter):
  File "/home/chuan/anaconda3/envs/few-shot-new/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/chuan/anaconda3/envs/few-shot-new/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "/home/chuan/anaconda3/envs/few-shot-new/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/home/chuan/anaconda3/envs/few-shot-new/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/chuan/anaconda3/envs/few-shot-new/lib/python3.7/site-packages/numpy/core/function_base.py", line 117, in linspace
    num = operator.index(num)
TypeError: 'numpy.float64' object cannot be interpreted as an integer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/chuan/anaconda3/envs/few-shot-new/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/chuan/anaconda3/envs/few-shot-new/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/chuan/anaconda3/envs/few-shot-new/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/chuan/few-shot-vid2vid/data/fewshot_pose_dataset.py", line 91, in __getitem__
    idx, ref_size, img_params, self.ref_crop_coords[i])
  File "/home/chuan/few-shot-vid2vid/data/fewshot_pose_dataset.py", line 129, in get_images
    O, op, crop_coords, face_pts = self.get_image(op_path, size, params, crop_coords, input_type='openpose', ref_face_pts=ref_face_pts)
  File "/home/chuan/few-shot-vid2vid/data/fewshot_pose_dataset.py", line 149, in get_image
    opt.basic_point_only, opt.remove_face_labels, ppl_idx, ref_face_pts)
  File "/home/chuan/few-shot-vid2vid/data/keypoint2img.py", line 48, in read_keypoints
    pose_img = connect_keypoints(opt, pts, edge_lists, size, random_drop_prob, distort_points, basic_point_only, remove_face_labels)
  File "/home/chuan/few-shot-vid2vid/data/keypoint2img.py", line 92, in connect_keypoints
    curve_x, curve_y = interpPoints(x, y)
  File "/home/chuan/few-shot-vid2vid/data/keypoint2img.py", line 309, in interpPoints
    curve_x = np.linspace(x[0], x[-1], (x[-1]-x[0]))
  File "<__array_function__ internals>", line 6, in linspace
  File "/home/chuan/anaconda3/envs/few-shot-new/lib/python3.7/site-packages/numpy/core/function_base.py", line 121, in linspace
    .format(type(num)))
TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.

./scripts/pose/train_g1.sh: line 10: 20085 Segmentation fault      (core dumped) python train.py --name pose --dataset_mode fewshot_pose --adaptive_spade --warp_ref --spade_combine --remove_face_labels --add_face_D --niter_single 100 --niter 200 --batchSize 2 --continue_train
FredMusoro commented 4 years ago

Hey there, please detail the setup you're using in terms of hardware. That error occurs when when the program tries to access restricted memory, the machine will send a kill signal. What worked for me was reducing the batch size to 2. However I'm using 2 RTX 2080 Ti GPUs.

nalzok commented 4 years ago

Hi @FredMusoro! The good news is, we have just resolved the issue; the bad news is, we have no idea how. Basically, I cloned the conda environment created by another colleague, and it just works.

Our server uses a cluster of 4 GeForce RTX 2080 Ti GPUs.

I am also reproducing the output of conda list for future reference

# packages in environment at /home/chuan/anaconda3/envs/few-shot:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
absl-py                   0.7.1                    pypi_0    pypi
astroid                   2.2.5                    py37_0    conda-forge
attrs                     19.1.0                     py_0    conda-forge
audioread                 2.1.6                    py37_0    conda-forge
autopep8                  1.4.4                    pypi_0    pypi
backcall                  0.1.0                      py_0    conda-forge
bleach                    3.1.0                      py_0    conda-forge
boto                      2.49.0                   pypi_0    pypi
boto3                     1.9.228                  pypi_0    pypi
botocore                  1.12.228                 pypi_0    pypi
bzip2                     1.0.8                h516909a_0    conda-forge
ca-certificates           2019.11.27                    0
certifi                   2019.11.28               py37_0
cffi                      1.12.3           py37h8022711_0    conda-forge
chardet                   3.0.4                    pypi_0    pypi
cudatoolkit               10.0.130                      0
cycler                    0.10.0                     py_1    conda-forge
dbus                      1.13.6               he372182_0    conda-forge
decorator                 4.4.0                      py_0    conda-forge
defusedxml                0.5.0                      py_1    conda-forge
docutils                  0.15.2                   pypi_0    pypi
dominate                  2.4.0                    pypi_0    pypi
entrypoints               0.3                   py37_1000    conda-forge
expat                     2.2.5             he1b5a44_1003    conda-forge
ffmpeg                    1.4                      pypi_0    pypi
fontconfig                2.13.1            h86ecdb6_1001    conda-forge
freetype                  2.10.0               he983fc9_0    conda-forge
future                    0.17.1                   pypi_0    pypi
gensim                    3.8.0                    pypi_0    pypi
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
glib                      2.58.3            h6f030ca_1002    conda-forge
gmp                       6.1.2             hf484d3e_1000    conda-forge
gnutls                    3.6.5             hd3a4fd2_1002    conda-forge
grpcio                    1.22.0                   pypi_0    pypi
gst-plugins-base          1.14.5               h0935bb2_0    conda-forge
gstreamer                 1.14.5               h36ae1b5_0    conda-forge
h5py                      2.9.0           nompi_py37h513d04c_1104    conda-forge
hdf5                      1.10.5          nompi_h3c11f04_1100    conda-forge
icu                       64.2                 he1b5a44_0    conda-forge
idna                      2.8                      pypi_0    pypi
intel-openmp              2019.4                      243
ipykernel                 5.1.1            py37h5ca1d4c_0    conda-forge
ipython                   7.7.0            py37h5ca1d4c_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
isort                     4.3.21                   py37_0    conda-forge
jedi                      0.14.1                   py37_0    conda-forge
jinja2                    2.10.1                     py_0    conda-forge
jmespath                  0.9.4                    pypi_0    pypi
joblib                    0.14.1                     py_0
jpeg                      9c                h14c3975_1001    conda-forge
json5                     0.8.5                      py_0    conda-forge
jsonschema                3.0.2                    py37_0    conda-forge
jupyter_client            5.3.1                      py_0    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
jupyterlab                1.0.4                    py37_0    conda-forge
jupyterlab_server         1.0.0                      py_1    conda-forge
kiwisolver                1.1.0            py37hc9558a2_0    conda-forge
krippendorff              0.3.0                    pypi_0    pypi
lame                      3.100             h14c3975_1001    conda-forge
lazy-object-proxy         1.4.1            py37h516909a_0    conda-forge
libblas                   3.8.0               11_openblas    conda-forge
libcblas                  3.8.0               11_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libflac                   1.3.1             hf484d3e_1002    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0
libgfortran-ng            7.3.0                hdf63c60_0
libiconv                  1.15              h516909a_1005    conda-forge
liblapack                 3.8.0               11_openblas    conda-forge
libogg                    1.3.2             h14c3975_1001    conda-forge
libopenblas               0.3.6                h6e990d7_6    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
librosa                   0.7.0                      py_0    conda-forge
libsndfile                1.0.28            hf484d3e_1000    conda-forge
libsodium                 1.0.17               h516909a_0    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0
libtiff                   4.0.10            h57b8799_1003    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libvorbis                 1.3.5             h14c3975_1001    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxml2                   2.9.9                hee79883_2    conda-forge
llvmlite                  0.29.0           py37hfd453ef_1    conda-forge
lz4-c                     1.8.3             he1b5a44_1001    conda-forge
mad                       0.15.1b              he1b5a44_0    conda-forge
markdown                  3.1.1                    pypi_0    pypi
markupsafe                1.1.1            py37h14c3975_0    conda-forge
matplotlib                3.1.1                    py37_1    conda-forge
matplotlib-base           3.1.1            py37he7580a8_1    conda-forge
mccabe                    0.6.1                      py_1    conda-forge
mistune                   0.8.4           py37h14c3975_1000    conda-forge
mkl                       2019.4                      243
nbconvert                 5.5.0                      py_0    conda-forge
nbformat                  4.4.0                      py_1    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
nettle                    3.4.1             h1bed415_1002    conda-forge
ninja                     1.9.0                h6bb024c_0    conda-forge
nltk                      3.4.5                    pypi_0    pypi
notebook                  6.0.0                    py37_0    conda-forge
numba                     0.45.1           py37hb3f55d8_0    conda-forge
numpy                     1.17.0           py37h95a1406_0    conda-forge
olefile                   0.46                       py_0    conda-forge
opencv-python             4.1.0.25                 pypi_0    pypi
openh264                  1.8.0             hdbcaa40_1000    conda-forge
openssl                   1.1.1d               h7b6447c_3
pandas                    0.25.0           py37hb3f55d8_0    conda-forge
pandoc                    2.7.3                         0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.5.1                      py_0    conda-forge
patsy                     0.5.1                      py_0    conda-forge
pcre                      8.41              hf484d3e_1003    conda-forge
pexpect                   4.7.0                    py37_0    conda-forge
pickleshare               0.7.5                 py37_1000    conda-forge
pillow                    6.1.0            py37h6b7be26_1    conda-forge
pip                       19.2.1                   py37_0    conda-forge
prometheus_client         0.7.1                      py_0    conda-forge
prompt_toolkit            2.0.9                      py_0    conda-forge
protobuf                  3.9.1                    pypi_0    pypi
pthread-stubs             0.4               h14c3975_1001    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
pycodestyle               2.5.0                    pypi_0    pypi
pycparser                 2.19                     py37_1    conda-forge
pygments                  2.4.2                      py_0    conda-forge
pylint                    2.3.1                    py37_0    conda-forge
pyparsing                 2.4.2                      py_0    conda-forge
pyqt                      5.9.2            py37hcca6a23_2    conda-forge
pyrsistent                0.15.4           py37h516909a_0    conda-forge
pysoundfile               0.10.2                  py_1001    conda-forge
python                    3.7.3                h33d41f4_1    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
pytorch                   1.2.0           py3.7_cuda10.0.130_cudnn7.6.2_0    pytorch
pytorch-pretrained-bert   0.6.2                    pypi_0    pypi
pytz                      2019.2                     py_0    conda-forge
pyzmq                     18.0.2           py37h1768529_2    conda-forge
qt                        5.9.7                h0c104cb_3    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
regex                     2019.8.19                pypi_0    pypi
requests                  2.22.0                   pypi_0    pypi
resampy                   0.2.1                      py_1    conda-forge
s3transfer                0.2.1                    pypi_0    pypi
scikit-learn              0.21.3           py37hcdab131_0    conda-forge
scipy                     1.3.1            py37h921218d_0    conda-forge
seaborn                   0.9.0                      py_1    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                41.0.1                   py37_0    conda-forge
sip                       4.19.8          py37hf484d3e_1000    conda-forge
six                       1.12.0                py37_1000    conda-forge
slurmpy                   0.0.7                    pypi_0    pypi
smart-open                1.8.4                    pypi_0    pypi
sox                       14.4.2            h27f72bc_1013    conda-forge
sqlite                    3.29.0               hcee41ef_0    conda-forge
statsmodels               0.10.1           py37hc1659b7_0    conda-forge
tb-nightly                1.15.0a20190806          pypi_0    pypi
terminado                 0.8.2                    py37_0    conda-forge
testpath                  0.4.2                   py_1001    conda-forge
tk                        8.6.9             hed695b0_1002    conda-forge
torchaudio                0.3.0                    pypi_0    pypi
torchvision               0.4.0                py37_cu100    pytorch
tornado                   6.0.3            py37h516909a_0    conda-forge
tqdm                      4.40.0                     py_0
traitlets                 4.3.2                 py37_1000    conda-forge
urllib3                   1.25.3                   pypi_0    pypi
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
werkzeug                  0.15.5                   pypi_0    pypi
wheel                     0.33.4                   py37_0    conda-forge
wordsegment               1.3.1                    pypi_0    pypi
wrapt                     1.11.2           py37h516909a_0    conda-forge
x264                      1!152.20180806       h14c3975_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zeromq                    4.3.2                he1b5a44_2    conda-forge
zlib                      1.2.11            h516909a_1005    conda-forge
zstd                      1.4.0                h3b9ef0a_0    conda-forge
IpsumDominum commented 4 years ago

Original Traceback (most recent call last): File "/home/chuan/anaconda3/envs/few-shot-new/lib/python3.7/site-packages/numpy/core/function_base.py", line 117, in linspace num = operator.index(num) Go to your numpy and change the line to:

num = operator.index(int(num))

jjandnn commented 4 years ago

conda install numpy=1.17.5 --yes

kabbi159 commented 4 years ago

I also have this issue when using Colab with numpy 1.18.1, but after downgrading numpy to 1.17 works.

haimin777 commented 4 years ago

+1 numpy 17 vs numpy 18

Bowenwu1 commented 3 years ago

Solved by pip install numpy==1.17.5