eckertzhang
commented
3 months ago I guess this is caused by the difference in the ‘diffusers’ versions.
Open Thesaltedfish-single opened 3 months ago
eckertzhang
commented
3 months ago I guess this is caused by the difference in the ‘diffusers’ versions.
admg67
commented
3 months ago Good job! I had a similar problem. Can you please provide the download address for the diffusion model?
torch.set_float32_matmul_precision('medium' | 'high') which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
[INFO] single image dataset: load normal from ECON model
[INFO] Lightning automatically upgraded your loaded checkpoint from v1.7.4 to v2.0.3. To apply the upgrade to your files permanently, run python -m pytorch_lightning.utilities.upgrade_checkpoint --file ../data/econ_weights/ckpt/normal.ckpt
[INFO] LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
[INFO]
| Name | Type | Params12.6 M Trainable params
164 M Non-trainable params
176 M Total params
707.813 Total estimated model params size (MB)
[INFO] Validation results will be saved to ./Results/humanref/a_woman_wearing_a_plaid_coat_and_black_jeans,_high_quality/save
Initializing SDF to a(n) mesh:: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:19<00:00, 50.09it/s]
initialization loss: 1.219243889877264e-12
Epoch 0: : 601it [04:10, 2.39it/s]╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /hy-tmp/HumanRef/run.py:213 in `optimizer.zero_grad │ │ │ │ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/pytorch_lightning/core/optimizer │ │ .py:155 in step │ │ │ │ 152 │ │ │ raise MisconfigurationException("Whenoptimizer.step(closure)` is called, t │
│ 153 │ │ │
│ 154 │ │ assert self._strategy is not None │
│ ❱ 155 │ │ step_output = self._strategy.optimizer_step(self._optimizer, closure, kwargs) │
│ 156 │ │ │
│ 157 │ │ self._on_after_step() │
│ 158 │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/pytorch_lightning/strategies/str │
│ ategy.py:225 in optimizer_step │
│ │
│ 222 │ │ model = model or self.lightning_module │
│ 223 │ │ # TODO(fabric): remove assertion once strategy's optimizer_step typing is fixed │
│ 224 │ │ assert isinstance(model, pl.LightningModule) │
│ ❱ 225 │ │ return self.precision_plugin.optimizer_step(optimizer, model=model, closure=clos │
│ 226 │ │
│ 227 │ def _setup_model_and_optimizers(self, model: Module, optimizers: List[Optimizer]) -> │
│ 228 │ │ """Setup a model and multiple optimizers together. │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/pytorch_lightning/plugins/precis │
│ ion/precision_plugin.py:114 in optimizer_step │
│ │
│ 111 │ ) -> Any: │
│ 112 │ │ """Hook to run the optimizer step.""" │
│ 113 │ │ closure = partial(self._wrap_closure, model, optimizer, closure) │
│ ❱ 114 │ │ return optimizer.step(closure=closure, kwargs) │
│ 115 │ │
│ 116 │ def _clip_gradients( │
│ 117 │ │ self, │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/torch/optim/optimizer.py:140 in │
│ wrapper │
│ │
│ 137 │ │ │ │ obj, _ = args │
│ 138 │ │ │ │ profile_name = "Optimizer.step#{}.step".format(obj.class.name) │
│ 139 │ │ │ │ with torch.autograd.profiler.record_function(profile_name): │
│ ❱ 140 │ │ │ │ │ out = func(*args, *kwargs) │
│ 141 │ │ │ │ │ obj._optimizer_step_code() │
│ 142 │ │ │ │ │ return out │
│ 143 │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/torch/optim/optimizer.py:23 in │
│ _use_grad │
│ │
│ 20 │ │ prev_grad = torch.is_grad_enabled() │
│ 21 │ │ try: │
│ 22 │ │ │ torch.set_grad_enabled(self.defaults['differentiable']) │
│ ❱ 23 │ │ │ ret = func(self, args, kwargs) │
│ 24 │ │ finally: │
│ 25 │ │ │ torch.set_grad_enabled(prev_grad) │
│ 26 │ │ return ret │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/torch/optim/adam.py:183 in step │
│ │
│ 180 │ │ loss = None │
│ 181 │ │ if closure is not None: │
│ 182 │ │ │ with torch.enable_grad(): │
│ ❱ 183 │ │ │ │ loss = closure() │
│ 184 │ │ │
│ 185 │ │ for group in self.param_groups: │
│ 186 │ │ │ params_with_grad = [] │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/pytorch_lightning/plugins/precis │
│ ion/precision_plugin.py:101 in _wrap_closure │
│ │
│ 98 │ │ The closure (generally) runs backward so this allows inspecting gradients in │
│ 99 │ │ consistent with the PrecisionPlugin subclasses that cannot pass `optimizer. │ │ 100 │ │ """ │ │ ❱ 101 │ │ closure_result = closure() │ │ 102 │ │ self._after_closure(model, optimizer) │ │ 103 │ │ return closure_result │ │ 104 │ │ │ │ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/pytorch_lightning/loops/optimiza │ │ tion/automatic.py:140 in __call__ │ │ │ │ 137 │ │ return step_output │ │ 138 │ │ │ 139 │ def __call__(self, *args: Any, **kwargs: Any) -> Optional[Tensor]: │ │ ❱ 140 │ │ self._result = self.closure(*args, **kwargs) │ │ 141 │ │ return self._result.loss │ │ 142 │ │ 143 │ │ │ │ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/pytorch_lightning/loops/optimiza │ │ tion/automatic.py:126 in closure │ │ │ │ 123 │ │ self._zero_grad_fn = zero_grad_fn │ │ 124 │ │ │ 125 │ def closure(self, *args: Any, **kwargs: Any) -> ClosureResult: │ │ ❱ 126 │ │ step_output = self._step_fn() │ │ 127 │ │ │ │ 128 │ │ if step_output.closure_loss is None: │ │ 129 │ │ │ self.warning_cache.warn("training_stepreturnedNone`. If this was on pur │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/pytorch_lightning/loops/optimiza │
│ tion/automatic.py:307 in _training_step │
│ │
│ 304 │ │ trainer = self.trainer │
│ 305 │ │ │
│ 306 │ │ # manually capture logged metrics │
│ ❱ 307 │ │ training_step_output = call._call_strategy_hook(trainer, "training_step", kwarg │
│ 308 │ │ self.trainer.strategy.post_training_step() │
│ 309 │ │ │
│ 310 │ │ return self.output_result_cls.from_training_step_output(training_step_output, tr │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/pytorch_lightning/trainer/call.p │
│ y:287 in _call_strategy_hook │
│ │
│ 284 │ │ return None │
│ 285 │ │
│ 286 │ with trainer.profiler.profile(f"[Strategy]{trainer.strategy.class.name}.{hoo │
│ ❱ 287 │ │ output = fn(args, kwargs) │
│ 288 │ │
│ 289 │ # restore current_fx when nested context │
│ 290 │ pl_module._current_fx_name = prev_fx_name │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/pytorch_lightning/strategies/str │
│ ategy.py:367 in training_step │
│ │
│ 364 │ │ """ │
│ 365 │ │ with self.precision_plugin.train_step_context(): │
│ 366 │ │ │ assert isinstance(self.model, TrainingStep) │
│ ❱ 367 │ │ │ return self.model.training_step(args, kwargs) │
│ 368 │ │
│ 369 │ def post_training_step(self) -> None: │
│ 370 │ │ pass │
│ │
│ /hy-tmp/HumanRef/threestudio/systems/humanref_sdf.py:391 in training_step │
│ │
│ 388 │ │ │ │ mask_image_ini = torch.cat([1. - pred_mask.detach(), pred_mask.detach()] │
│ 389 │ │ │ else: │
│ 390 │ │ │ │ mask_image_ref, mask_image_ini = None, None │
│ ❱ 391 │ │ │ guidance_out = self.guidance( │
│ 392 │ │ │ │ rgb=pred_rgb, │
│ 393 │ │ │ │ ref_image=rgb_ref, │
│ 394 │ │ │ │ prompt_utils=self.prompt_utils, batch, │
│ │
│ /hy-tmp/HumanRef/threestudio/models/guidance/reference_sds.py:886 in call │
│ │
│ 883 │ │ ) │
│ 884 │ │ │
│ 885 │ │ if with_clip_loss: │
│ ❱ 886 │ │ │ grad, guidance_eval_utils = self.compute_grad_sds_clip( │
│ 887 │ │ │ │ latents, ref_image_latents, t, promptutils, elevation, azimuth, camera │
│ 888 │ │ │ │ clip_model=clip_model, rendered_region=rendered_region, with_clip_text_l │
│ 889 │ │ │ │ mask_image_ref=mask_image_ref, mask_image_ini=mask_image_ini, │
│ │
│ /hy-tmp/HumanRef/threestudio/models/guidance/reference_sds.py:759 in compute_grad_sds_clip │
│ │
│ 756 │ │ │ ref_xt = torch.cat([ref_xt] 2) │
│ 757 │ │ │ ref_xt = self.scheduler.scale_model_input(ref_xt, t) │
│ 758 │ │ │ MODE = "write" │
│ ❱ 759 │ │ │ self.unet( │
│ 760 │ │ │ │ │ ref_xt.to(self.weights_dtype), │
│ 761 │ │ │ │ │ torch.cat([t] 2).to(self.weights_dtype), │
│ 762 │ │ │ │ │ encoder_hidden_states=text_embeddings.to(self.weights_dtype), │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/torch/nn/modules/module.py:1194 │
│ in _call_impl │
│ │
│ 1191 │ │ # this function, and just call forward. │
│ 1192 │ │ if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks o │
│ 1193 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1194 │ │ │ return forward_call(input, kwargs) │
│ 1195 │ │ # Do not call functions when jit is used │
│ 1196 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1197 │ │ if self._backward_hooks or _global_backward_hooks: │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/diffusers/models/unet_2d_conditi │
│ on.py:899 in forward │
│ │
│ 896 │ │ │ image_embeds = added_cond_kwargs.get("image_embeds") │
│ 897 │ │ │ encoder_hidden_states = self.encoder_hid_proj(image_embeds) │
│ 898 │ │ # 2. pre-process │
│ ❱ 899 │ │ sample = self.conv_in(sample) │
│ 900 │ │ │
│ 901 │ │ # 3. down │
│ 902 │ │ down_block_res_samples = (sample,) │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/torch/nn/modules/module.py:1194 │
│ in _call_impl │
│ │
│ 1191 │ │ # this function, and just call forward. │
│ 1192 │ │ if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks o │
│ 1193 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1194 │ │ │ return forward_call(*input, **kwargs) │
│ 1195 │ │ # Do not call functions when jit is used │
│ 1196 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1197 │ │ if self._backward_hooks or _global_backward_hooks: │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/torch/nn/modules/conv.py:463 in │
│ forward │
│ │
│ 460 │ │ │ │ │ │ self.padding, self.dilation, self.groups) │
│ 461 │ │
│ 462 │ def forward(self, input: Tensor) -> Tensor: │
│ ❱ 463 │ │ return self._conv_forward(input, self.weight, self.bias) │
│ 464 │
│ 465 class Conv3d(_ConvNd): │
│ 466 │ doc = r"""Applies a 3D convolution over an input signal composed of several inpu │
│ │
│ /usr/local/miniconda3/envs/HumanRef/lib/python3.8/site-packages/torch/nn/modules/conv.py:459 in │
│ _conv_forward │
│ │
│ 456 │ │ │ return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=sel │
│ 457 │ │ │ │ │ │ │ weight, bias, self.stride, │
│ 458 │ │ │ │ │ │ │ _pair(0), self.dilation, self.groups) │
│ ❱ 459 │ │ return F.conv2d(input, weight, bias, self.stride, │
│ 460 │ │ │ │ │ │ self.padding, self.dilation, self.groups) │
│ 461 │ │
│ 462 │ def forward(self, input: Tensor) -> Tensor: │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Given groups=1, weight of size [320, 8, 3, 3], expected input[2, 4, 64, 64] to have 8 channels, but got 4 channels instead
Thesaltedfish-single
commented
3 months ago I guess this is caused by the difference in the ‘diffusers’ versions.
Thank you for your reply! Because stablediffusion1-5 on Huggingface has been taken down, I have chosen to https://modelscope.cn/models/AI-ModelScope/stable-diffusion-v1-5 I downloaded the diffusion model. Is there a difference between the two
Sean858
commented
2 months ago I had the same issue, any help? I am using the sd-legacy/stable-diffusion-v1-5.
Excellent work！
I have completed Preprocessing: Estimate the SMPL-X body mesh using ECON and segment the input image into RGBA format And use Python run. py -- config config/humanref. yaml -- train -- GPU 0 image_cath='/ data/Results_ECON/image_000355/econ/imgs_crop/image_000355_0_rgba.png' But during the running process, it prompted me that the total number of attention tensor is 20480, which cannot be organized into the space of [2,8, -1,4096]
This happened during https://github.com/eckertzhang/HumanRef/blob/ccc0ba7b95b68b90851d5163b80edafe44379de0/threestudio/models/guidance/reference_sds.py#L200