[Bug]: ControlNet Travel script broken? "NameError: name 'self' is not defined"

[marcsyp]

Hi there! Love the prompt travel script, thank you for this!

I am trying to use ControlNet Travel for the first time, and was super excited to get started, but it appears to be broken. Perhaps I am not using it correctly, but I can't find any issue or discussion related to this script, and there's not much in the readme outside of the cool example you've provided, so I'm not sure what to do... I would really love to get something going soon and maybe there's an easy fix!

In any case, here are steps to reproduce:

1. Use txt2img to set up a controlnet that you like, add an image to a controlnet module and play until you're satisfied.
2. Remove the image from the ControlNet module in preparation for batch processing.
3. Select ControlNet Travel script and add the directory path where you are storing your images.
4. Hit Generate, you will get "NameError: name 'self' is not defined"
5. Add your starting image back in to the ControlNet module to see if that's the problem.
6. Hit Generate, you will get "NameError: name 'self' is not defined"

Full error in terminal:

Error completing request Arguments: ('task(jdnn5h3kpd9x6h8)', 'a portrait of a CEO for a large company, 8k, cinematic, dramatic, late evening light', 'lens flare', [], 20, 0, False, False, 1, 1, 7, 980000670.0, -1.0, 0, 512, 512, True, 512, 512, False, 0.42, 2, 'Latent', 12, 0, 0, [], 9, True, 'keyword prompt', 'keyword1, keyword2', 'None', 'textual inversion first', 'None', 0.7, 'None', True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', False, 'x264', 'mci', 10, 0, False, True, True, True, 'intermediate', 'animation', <scripts.external_code.ControlNetUnit object at 0x00000136AA898700>, <scripts.external_code.ControlNetUnit object at 0x00000136AD6B0D90>, <scripts.external_code.ControlNetUnit object at 0x00000136AD6E18A0>, False, '1:1,1:2,1:2', '0:0,0:0,0:1', '0.2,0.8,0.8', 20, 0.2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0, '', 'None', 30, 4, 0, 0, False, 'None', '
', 'None', 30, 4, 0, 0, 4, 0.4, True, 32, None, False, None, False, None, False, 50, False, 4.0, '', 10.0, 'Linear', 3, False, 30.0, True, False, False, 0, 0.0, 'Lanczos', 1, True, 0, 0, 0.001, 75, 0.0, False, True, 10.0, 30.0, True, 0.0, 'Lanczos', 1, 0, 0, 75, 0.0001, 0.0, False, True, '', False, False, False, False, 'Auto', 0.5, 1, 'linear (weight sum)', '5', 'C:\CEOs', 'Lanczos', 2, 0, 0, 'mp4', 3.0, 0, '', True, False, 'linear', 'lerp', 'token', 'random', '30', 'fixed', 1, '8', None, 'Lanczos', 2, 0, 0, 'mp4', 10.0, 0, '', True, False, False) {} Traceback (most recent call last): File "C:\SDW\stable-diffusion-webui\modules\call_queue.py", line 56, in f res = list(func(*args, kwargs)) File "C:\SDW\stable-diffusion-webui\modules\call_queue.py", line 37, in f res = func(*args, *kwargs) File "C:\SDW\stable-diffusion-webui\modules\txt2img.py", line 53, in txt2img processed = modules.scripts.scripts_txt2img.run(p, args) File "C:\SDW\stable-diffusion-webui\modules\scripts.py", line 407, in run processed = script.run(p, script_args) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 681, in run gen_image() File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 664, in gen_image proc = process_images(p) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 240, in process_images res = process_images_inner(p) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 402, in process_images_inner samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts) File "C:\SDW\stable-diffusion-webui\modules\processing.py", line 869, in sample samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x)) File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 358, in sample samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={ File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 234, in launch_sampling return func() File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 358, in samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={ File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context return func(args, kwargs) File "C:\SDW\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral denoised = model(x, sigmas[i] * s_in, extra_args) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, *kwargs) File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 126, in forward x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict([cond_in], image_cond_in)) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(input, kwargs) File "C:\SDW\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), kwargs) File "C:\SDW\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps return self.inner_model.apply_model(*args, *kwargs) File "C:\SDW\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in setattr(resolved_obj, func_path[-1], lambda args, kwargs: self(*args, kwargs)) File "C:\SDW\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in call return self.__orig_func(args, kwargs) File "C:\SDW\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model x_recon = self.model(x_noisy, t, cond) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(input, kwargs) File "C:\SDW\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward out = self.diffusion_model(x, t, context=cc) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, kwargs) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 312, in unet.forward = lambda *args, *kwargs: forward2(unethook, args, kwargs) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 215, in forward2 return forward(self, *args, **kwargs) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 195, in forward h = cfg_based_adder(h, control_in, require_inpaint_hijack) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 48, in cfg_based_adder if base.shape[0] % 2 == 0 and (self.guess_mode or shared.opts.data.get("control_net_cfg_based_guidance", False)): NameError: name 'self' is not defined

[marcsyp]

Note: after disabling all other extensions except prompt travel + controlNet, checking boxes and playing around, I got a different error, trying to use HED (but no modules appear to work). Terminal message:

n_stages: 4 n_frames: 19 steps: [5, 5, 5] seed: 3269063086 subseed: -1 subseed_strength: 0 Loading model from cache: control_sd15_hed [fef5e48e] Loading preprocessor: hed 0%| | 0/20 [00:00<?, ?it/s] Error completing request Arguments: ('task(3gvmgyd5mudw30w)', 'a portrait of a CEO for a large company, 8k, cinematic, dramatic, late evening light', 'lens flare', [], 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 512, 512, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, ['Model hash: 9aba26abdf'], 5, <scripts.external_code.ControlNetUnit object at 0x00000136A2C35240>, <scripts.external_code.ControlNetUnit object at 0x00000136AA707070>, <scripts.external_code.ControlNetUnit object at 0x00000136A2C35A20>, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0, None, False, None, False, None, False, 50, 'linear (weight sum)', '5', 'C:\CEOs', 'Lanczos', 2, 0, 0, 'mp4', 10.0, 0, '', True, False, 'linear', 'lerp', 'token', 'random', '30', 'fixed', 1, '8', None, 'Lanczos', 2, 0, 0, 'mp4', 10.0, 0, '', True, False, False) {} Traceback (most recent call last): File "C:\SDW\stable-diffusion-webui\modules\call_queue.py", line 56, in f res = list(func(*args, kwargs)) File "C:\SDW\stable-diffusion-webui\modules\call_queue.py", line 37, in f res = func(*args, *kwargs) File "C:\SDW\stable-diffusion-webui\modules\txt2img.py", line 53, in txt2img processed = modules.scripts.scripts_txt2img.run(p, args) File "C:\SDW\stable-diffusion-webui\modules\scripts.py", line 407, in run processed = script.run(p, script_args) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 681, in run gen_image() File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 664, in gen_image proc = process_images(p) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 240, in process_images res = process_images_inner(p) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 402, in process_images_inner samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts) File "C:\SDW\stable-diffusion-webui\modules\processing.py", line 869, in sample samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x)) File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 358, in sample samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={ File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 234, in launch_sampling return func() File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 358, in samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={ File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context return func(args, kwargs) File "C:\SDW\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral denoised = model(x, sigmas[i] * s_in, extra_args) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, *kwargs) File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 126, in forward x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict([cond_in], image_cond_in)) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(input, kwargs) File "C:\SDW\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), kwargs) File "C:\SDW\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps return self.inner_model.apply_model(*args, *kwargs) File "C:\SDW\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in setattr(resolved_obj, func_path[-1], lambda args, kwargs: self(*args, kwargs)) File "C:\SDW\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in call return self.__orig_func(args, kwargs) File "C:\SDW\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model x_recon = self.model(x_noisy, t, cond) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(input, kwargs) File "C:\SDW\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward out = self.diffusion_model(x, t, context=cc) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, kwargs) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 312, in unet.forward = lambda *args, *kwargs: forward2(unethook, args, kwargs) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 215, in forward2 return forward(self, *args, kwargs) File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 147, in forward control = param.control_model(x=x_in, hint=param.hint_cond, timesteps=timesteps, context=context) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, *kwargs) File "C:\SDW\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 115, in forward return self.control_model(args, kwargs) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, kwargs) File "C:\SDW\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 368, in forward emb = self.time_embed(t_emb) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, *kwargs) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\container.py", line 204, in forward input = module(input) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(input, kwargs) File "C:\SDW\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 307, in lora_Linear_forward return torch.nn.Linear_forward_before_lora(self, input) File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_addmm)

[marcsyp]

automatic1111 commit: 22bcc7be

prompt travel: d3c89446 (Mon Apr 10 10:37:55 2023)

[marcsyp]

FYI -- I have a hunch that the tensors error above is a result of a bug in ControlNet after turning on and off Low Vram mode. Fresh loads always give the the Self not defined error.

[Kahsolt]

Thanks for your report~ I will check them one by one.

That ControlNet Travel script is so far an experiment, not fully tested, not stable. I wrote it one week ago to quickly verify the idea's viability. From yesterday on I resumed to reformat it, trying to find a more neat way for implementation.

You really feedback with plenty useful debug info, and understand to ask questions in a smart way, nice~ 🤤

[marcsyp]

No worries -- thank you for your contributions to the community! I am very excited about controlnet travel as an idea, I think it could have some really cool use cases that are not possible with other methods. Looking forward to your updates! I'm happy to be a tester. :)

[Kahsolt]

😆 I just got it refactored, add trivial options, and works fine on my machine now. Give it a try, I'm looking forward to your feedbacks~