video_gen dataloader: 0%| | 0/1 [17:01<?, ?it/s]
ERROR - main.py - 2022-09-03 17:05:24,564 - Traceback (most recent call last):
File "main.py", line 404, in main
runner.train()
File "/cluster/work/mcvd-pytorch/runners/ncsn_runner.py", line 497, in train
vid_metrics = self.video_gen(scorenet=test_scorenet, ckpt=step, train=True)
File "/cluster/home/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in de
corate_context
return func(*args, kwargs)
File "/cluster/work/mcvd-pytorch/runners/ncsn_runner.py", line 1940, in video_gen
real_embeddings.append(get_fvd_feats(real_fvd, i3d=i3d, device=self.config.device))
File "/cluster/work/mcvd-pytorch/models/fvd/fvd.py", line 55, in get_fvd_feats
embeddings = get_feats(videos, i3d, device, bs)
File "/cluster/work/mcvd-pytorch/models/fvd/fvd.py", line 48, in get_feats
feats = np.vstack([feats, detector(torch.stack([preprocess_single(video) for video in videos[ibs:(i+1)
bs]]).to(device), detector_kwargs).detach().cpu().numpy()])
File "/cluster/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _c
all_impl
result = self.forward(*input, kwargs)
File "/cluster/home/.local/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 16
5, in forward
return self.module(*inputs[0], *kwargs[0])
File "/cluster/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _c
all_impl
result = self.forward(input, kwargs)
RuntimeError: nvrtc: error: failed to open libnvrtc-builtins.so.11.1.
Make sure that libnvrtc-builtins.so.11.1 is installed correctly.
nvrtc compilation failed:
define NAN __int_as_float(0x7fffffff)
define POS_INFINITY __int_as_float(0x7f800000)
define NEG_INFINITY __int_as_float(0xff800000)
template device T maximum(T a, T b) {
return isnan(a) ? a : (a > b ? a : b);
}
Dataset length: 60000
Dataset length: 256
Setting up Perceptual loss...
Downloading: "https://download.pytorch.org/models/alexnet-owt-4df8aa71.pth" to /cluster/home/tangha/.cache/t orch/hub/checkpoints/alexnet-owt-4df8aa71.pth
100%|█████████████████████████████████████████████████████████████████████| 233M/233M [00:02<00:00, 106MB/s$ Loading model from: /cluster/work/mcvd-pytorch/models/weights/v0.1/alex.pth
...[net-lin [alex]] initialized
...Done
video_gen dataloader: 0%| | 0/1 [00:00<?, ?it/s]I NFO - ncsn_runner.py - 2022-09-03 16:48:22,970 - (1) Video Pred
INFO - ncsn_runner.py - 2022-09-03 16:48:22,971 - PREDICTING 20 frames, using a 5 frame model conditioned on 5 frames, subsample=1000, preds_per_test=1
Generating video frames: 100%|███████████████████████████████████████████████| 4/4 [16:49<00:00, 252.40s/it] INFO - ncsn_runner.py - 2022-09-03 17:05:21,209 - fvd1 True, fvd2 False, fvd3 False[16:49<00:00, 252.36s/it]
video_gen dataloader: 0%| | 0/1 [17:01<?, ?it/s] ERROR - main.py - 2022-09-03 17:05:24,564 - Traceback (most recent call last):
File "main.py", line 404, in main
runner.train()
File "/cluster/work/mcvd-pytorch/runners/ncsn_runner.py", line 497, in train
vid_metrics = self.video_gen(scorenet=test_scorenet, ckpt=step, train=True)
File "/cluster/home/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in de corate_context
return func(*args, kwargs)
File "/cluster/work/mcvd-pytorch/runners/ncsn_runner.py", line 1940, in video_gen
real_embeddings.append(get_fvd_feats(real_fvd, i3d=i3d, device=self.config.device))
File "/cluster/work/mcvd-pytorch/models/fvd/fvd.py", line 55, in get_fvd_feats
embeddings = get_feats(videos, i3d, device, bs)
File "/cluster/work/mcvd-pytorch/models/fvd/fvd.py", line 48, in get_feats
feats = np.vstack([feats, detector(torch.stack([preprocess_single(video) for video in videos[ibs:(i+1) bs]]).to(device), detector_kwargs).detach().cpu().numpy()])
File "/cluster/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _c all_impl
result = self.forward(*input, kwargs)
File "/cluster/home/.local/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 16 5, in forward
return self.module(*inputs[0], *kwargs[0])
File "/cluster/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _c all_impl
result = self.forward(input, kwargs)
RuntimeError: nvrtc: error: failed to open libnvrtc-builtins.so.11.1.
Make sure that libnvrtc-builtins.so.11.1 is installed correctly.
nvrtc compilation failed:
define NAN __int_as_float(0x7fffffff)
define POS_INFINITY __int_as_float(0x7f800000)
define NEG_INFINITY __int_as_float(0xff800000)
template
device T maximum(T a, T b) {
return isnan(a) ? a : (a > b ? a : b);
}