rscammell
commented
1 year ago I was able to solve this issue by changing the PointCloudSamplerInstance to a modified version of the one suggested by @easonnie in Issue #7:
sampler = PointCloudSampler(
device=device,
models=[base_model, upsampler_model, upsampler_diffusion],
diffusions=[base_diffusion, upsampler_diffusion],
num_points=[1024, 4096 - 1024, 4096 * 4 - 4096],
aux_channels=['R', 'G', 'B'],
guidance_scale=[3.0, 3.0, 3.0],
use_karras = (True, True, True),
karras_steps = (64, 64, 64),
sigma_min = (1e-3, 1e-3, 1e-3),
sigma_max = (120, 160, 160),
s_churn = (3, 0, 0),
)However, it still seems that the text2pointcloud.ipynb example code has a bug, inasmuch as a sampler doesn't seem to be instantiated elsewhere, so I am leaving this issue open for now.
I am working on getting the text2pointcloud.ipynb example code running in Blender 3.4's built-in Python environment (Windows 11, Python 3.10.8 (main, Oct 18 2022, 21:01:35) [MSC v.1928 64 bit (AMD64)] on win32, numpy 1.24.1, torch 1.13.1+cpu)
In doing so, I have encountered the error below when producing a sample from the model:
creating base model... MODEL_CONFIGS[base_name] dict{'cond_drop_prob': 0.1, 'heads': 8, 'init_scale': 0.25, 'input_channels': 6, 'layers': 12, 'n_ctx': 1024, 'name': 'CLIPImagePointDiffusionTransformer', 'output_channels': 12, 'time_token_cond': True, 'token_cond': True, 'width': 512, 'cache_dir': 'C:\Users\MyUser\Blender-Project'} creating upsample model... downloading base checkpoint... downloading upsampler checkpoint... creating base model... creating upsample model... downloading base checkpoint... downloading upsampler checkpoint... Producing a sample from the model... 0it [00:01, ?it/s] Error: Python: Traceback (most recent call last): File "C:\Users\MyUser\Blender-Project\pgu.blend\pgu.py", line 72, in
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\tqdm-4.64.1-py3.10.egg\tqdm\std.py", line 1195, in iter
for obj in iterable:
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\point_e-0.0.0-py3.10.egg\point_e\diffusion\sampler.py", line 163, in sample_batch_progressive
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\point_e-0.0.0-py3.10.egg\point_e\diffusion\k_diffusion.py", line 181, in karras_sample_progressive
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\torch-1.13.1-py3.10-win-amd64.egg\torch\autograd\grad_mode.py", line 43, in generator_context
response = gen.send(None)
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\point_e-0.0.0-py3.10.egg\point_e\diffusion\k_diffusion.py", line 265, in sample_heun
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\point_e-0.0.0-py3.10.egg\point_e\diffusion\k_diffusion.py", line 173, in guided_denoiser
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\point_e-0.0.0-py3.10.egg\point_e\diffusion\k_diffusion.py", line 160, in denoiser
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\point_e-0.0.0-py3.10.egg\point_e\diffusion\k_diffusion.py", line 105, in denoise
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\point_e-0.0.0-py3.10.egg\point_e\diffusion\gaussian_diffusion.py", line 285, in p_mean_variance
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\torch-1.13.1-py3.10-win-amd64.egg\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, *kwargs)
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\point_e-0.0.0-py3.10.egg\point_e\models\transformer.py", line 287, in forward
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\point_e-0.0.0-py3.10.egg\point_e\models\transformer.py", line 208, in _forward_with_cond
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\torch-1.13.1-py3.10-win-amd64.egg\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(input, kwargs)
File "C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages\torch-1.13.1-py3.10-win-amd64.egg\torch\nn\modules\linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [2, 4] but got: [2, 6].**
My setup is admittedly non-standard, but I am hoping that someone can help. To get to this point, I also made a couple of changes to the example code, including passing a custom 'cache_dir' to the model, since Blender defaults to using C:\Windows\system32 otherwise:
MODEL_CONFIGS[base_name]['cache_dir'] = cache_dir MODEL_CONFIGS['upsample']['cache_dir'] = cache_dir
I also encountered a problem in which the PointCloudSampler instance called 'sampler' wasn't instantiated, so I am creating an instance of this class as follows:
sampler = PointCloudSampler( device=device, models=[base_model, upsampler_model], diffusions=[base_diffusion, upsampler_diffusion], num_points=[1024, 2048], aux_channels=["normals"] )
this is called before the "for x in tqdm(sampler.sample_batch_progressive(batch_size=1, model_kwargs=dict(texts=[prompt]))): samples = x"
invocation.
Finally, I have applied the float to int TypeError fix described in Issue 6 (in point_e\diffusion\k_diffusion.py), but this does not solve the problem.