Open Maichelanger opened 4 years ago
I'm not one of the project's programmers/maintainers, but I'll just chime in & say this is the spot where gcs_utils.py
is trying to call wget
to download the models; do you have wget
on your computer? If not, you should try installing it (from here, maybe?) and then give this another shot.
I'm not one of the project's programmers/maintainers, but I'll just chime in & say this is the spot where
gcs_utils.py
is trying to callwget
to download the models; do you havewget
on your computer? If not, you should try installing it (from here, maybe?) and then give this another shot.
Yes, that was the problem. Now, with wget properly installed, everything works fine.
Except for this:
Running this cell:
zs = upsample(zs, labels, sampling_kwargs, [*upsamplers, top_prior], hps)
The console comes up with an error:
Sampling level 1
Sampling 8192 tokens for [0,8192]. Conditioning on 0 tokens
Ancestral sampling 3 samples with temp=0.99, top_k=0, top_p=0.0
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-11-8b3a8268585a> in <module>
----> 1 zs = upsample(zs, labels, sampling_kwargs, [*upsamplers, top_prior], hps)
~\anaconda3\lib\site-packages\jukebox\sample.py in upsample(zs, labels, sampling_kwargs, priors, hps)
137 def upsample(zs, labels, sampling_kwargs, priors, hps):
138 sample_levels = list(range(len(priors) - 1))
--> 139 zs = _sample(zs, labels, sampling_kwargs, priors, sample_levels, hps)
140 return zs
141
~\anaconda3\lib\site-packages\jukebox\sample.py in _sample(zs, labels, sampling_kwargs, priors, sample_levels, hps)
100 total_length = hps.sample_length//prior.raw_to_tokens
101 hop_length = int(hps.hop_fraction[level]*prior.n_ctx)
--> 102 zs = sample_level(zs, labels[level], sampling_kwargs[level], level, prior, total_length, hop_length, hps)
103
104 prior.cpu()
~\anaconda3\lib\site-packages\jukebox\sample.py in sample_level(zs, labels, sampling_kwargs, level, prior, total_length, hop_length, hps)
83 if total_length >= prior.n_ctx:
84 for start in get_starts(total_length, prior.n_ctx, hop_length):
---> 85 zs = sample_single_window(zs, labels, sampling_kwargs, level, prior, start, hps)
86 else:
87 zs = sample_partial_window(zs, labels, sampling_kwargs, level, prior, total_length, hps)
~\anaconda3\lib\site-packages\jukebox\sample.py in sample_single_window(zs, labels, sampling_kwargs, level, prior, start, hps)
67 z_samples = []
68 for z_i, z_conds_i, y_i in zip(z_list, z_conds_list, y_list):
---> 69 z_samples_i = prior.sample(n_samples=z_i.shape[0], z=z_i, z_conds=z_conds_i, y=y_i, **sampling_kwargs)
70 z_samples.append(z_samples_i)
71 z = t.cat(z_samples, dim=0)
~\anaconda3\lib\site-packages\jukebox\prior\prior.py in sample(self, n_samples, z, z_conds, y, fp16, temp, top_k, top_p, chunk_size, sample_tokens)
259 with t.no_grad():
260 # Currently x_cond only uses immediately above layer
--> 261 x_cond, y_cond, prime = self.get_cond(z_conds, y)
262 if self.single_enc_dec:
263 # assert chunk_size % self.prime_loss_dims == 0. TODO: Check if needed
~\anaconda3\lib\site-packages\jukebox\prior\prior.py in get_cond(self, z_conds, y)
240 y, prime = None, None
241 y_cond, y_pos = self.y_emb(y) if self.y_cond else (None, None)
--> 242 x_cond = self.x_emb(z_conds) if self.x_cond else y_pos
243 return x_cond, y_cond, prime
244
~\anaconda3\lib\site-packages\jukebox\prior\prior.py in x_emb(self, z_conds)
208 x_cond = None
209 for z_cond, conditioner_block in reversed(list(zip(z_conds, self.conditioner_blocks))):
--> 210 x_cond = conditioner_block(z_cond, x_cond)
211 return x_cond
212
~\anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)
~\anaconda3\lib\site-packages\jukebox\prior\conditioners.py in forward(self, x, x_cond)
43 # Run conditioner
44 x = self.preprocess(x)
---> 45 x = self.cond(x)
46 x = self.postprocess(x)
47 x = self.ln(x)
~\anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)
~\anaconda3\lib\site-packages\jukebox\vqvae\encdec.py in forward(self, x)
44
45 def forward(self, x):
---> 46 return self.model(x)
47
48 class Encoder(nn.Module):
~\anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)
~\anaconda3\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
98 def forward(self, input):
99 for module in self:
--> 100 input = module(input)
101 return input
102
~\anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)
~\anaconda3\lib\site-packages\torch\nn\modules\conv.py in forward(self, input)
210 _single(0), self.dilation, self.groups)
211 return F.conv1d(input, self.weight, self.bias, self.stride,
--> 212 self.padding, self.dilation, self.groups)
213
214
RuntimeError: cuDNN error: CUDNN_STATUS_ALLOC_FAILED
From what I could find, it seems that my GPU doesn't have enough VRAM. Am I right? Then... Wouldn't there be another way to run the upsampling on my computer without relying on Google CoLab's machines?
To check this hypothesis, you could try changing the model from 5b_lyrics
to 1b_lyrics
, which requires less GPU memory, and see if you're able to upsample.
Unfortunately... I already changed the model to 1b and lowered the max batch size to 3 before sampling. It's obvious that my card is not enough for AI (An RTX 2060 with 6GB of VRAM), but I thought that maybe there would be some workaround...
Any other idea?
BTW, thanks for your answers.
The docs say the 1b_lyrics
model takes 3.8 GB, with another few hundred megabytes on top for other parts of the model, so maybe it would be worth trying a batch size of just 1? If that doesn't work, then yeah, it might simply not be possible with a 6GB GPU.
No problem---happy to help, even it's only slightly.
Nope... Same Issue. I think the other things that load into the VRAM takes much more space than a few hundred MB.
Well, It was nice to try at least.
Before saying anything, I must say that I'm not a programmer and I only have little idea about Phyton yet (almost no idea), but I want to experiment with programs like this one. The problem is, due to my lack of knowledge, my problem-solving skills are very limited. And although I have found some solutions on my own, others scapes from my abilities, like this one:
With the "notebook" already being run in my local machine, I download all the things I need for Jukebox. Then, the problem appears when I try to start the model in this cell:
And the response in the console seems to say that the system was unable to find some files:
I can assume that the problem is not that much complicated, but again, I have little idea of Phyton.
I thank beforehand any of the help that you could bring me.