codeautopilot[bot]
commented
6 months ago Potential solution
The plan to solve the bug involves ensuring that the file paths for the LoRA model weights are correct and that the model is loaded successfully before attempting to generate the video. Additionally, we need to add error handling to provide informative messages if the model or weights cannot be found or loaded. This will help the user to quickly identify and fix the issue.
What is causing this bug?
The bug is likely caused by one or more of the following issues:
- Incorrect file paths for the LoRA model weights, leading to a "No such file or directory" error.
- Failure to load the LoRA model correctly, resulting in a
Noneobject and subsequently causing the "'NoneType' object has no attribute 'items'" error when attempting to access the model's items. - Potential issues with the LoRA injection and realization process, which could leave the model in an incomplete or incorrect state.
Code
To address the potential causes, the following code snippets and checks can be implemented:
- Verify file paths and existence of the LoRA weights:
import os
lora_weights_path = 'path/to/lora/weights' # Replace with the correct path if not os.path.exists(lora_weights_path): raise FileNotFoundError(f"LoRA weights file not found at {lora_weights_path}")
2. Add error handling after model loading:
```python
model = load_model_function(...) # Replace with the actual function to load the model
if model is None:
raise ValueError("Failed to load the LoRA model. Please check the model path and weights.")- Ensure LoRA injection and realization are correctly implemented:
# After injecting LoRA layers if not are_lora_layers_injected(model): # Replace with the actual check raise ValueError("LoRA layers were not injected correctly.")
After realizing LoRA layers
if not is_lora_realization_successful(model): # Replace with the actual check raise ValueError("LoRA layers were not realized correctly.")
# How to replicate the bug
To replicate the bug, follow these steps:
1. Go to the web UI and open the text2video generation feature.
2. Load the trained LoRA text-to-video model using the option provided at the bottom.
3. Enter a prompt and attempt to generate a video.
4. The error should appear in the terminal.
# Task
To resolve the issue, the following tasks should be performed:
1. Verify and correct the file paths for the LoRA model weights.
2. Implement error handling to check for successful model loading and provide informative error messages.
3. Review the LoRA injection and realization logic to ensure it is applied correctly.
4. Add unit tests to validate the functionality of LoRA-related operations.
[Click here to create a Pull Request with the proposed solution](https://api.codeautopilot.com/create-pr?task=23927&hash=a6479eda6331b3833e7f64db2e22b876db66f7935752bd91fa121bf38e025924)
**Files used for this task:**
<details>
<summary>Changes on utils/lora.py</summary>
Upon reviewing the `utils/lora.py` file, it is evident that the code is designed to handle the injection, manipulation, and saving of LoRA (Low-Rank Adaptation) layers into neural network models. The file contains a variety of classes and functions to support these operations.
The user's issue is related to loading a fine-tuned/trained LoRA model and generating a video. The error messages mentioned are "No such file or directory" and "'NoneType' object has no attribute 'items'".
Based on the provided code and the error messages, here are some potential causes and recommendations for the issue:
1. **File Not Found Error**: The error "No such file or directory" suggests that the code is trying to access a file that does not exist at the specified path. This could be due to an incorrect file path or the file not being present where the code expects it to be.
- **Recommendation**: Ensure that the file paths specified in the code for loading the LoRA weights are correct and that the files actually exist at those locations. This includes checking the paths provided to functions like `torch.load()` which are used to load the LoRA weights.
2. **'NoneType' object Error**: The error "'NoneType' object has no attribute 'items'" indicates that a variable expected to be a dictionary or an object with an `items()` method is actually `None`. This could happen if the model or weights are not loaded correctly, resulting in `None` being passed to a function or method that expects a dictionary.
- **Recommendation**: Add error handling around the code that loads the LoRA weights to ensure that the loading process is successful. If the weights cannot be loaded, the code should provide a clear error message indicating the problem. This can be done by checking if the result of `torch.load()` or similar functions is `None` and handling that case appropriately.
3. **Safe Loading of Weights**: The code uses a function `safe_open` from the `safetensors` library to load weights safely. If the weights file is not in the expected format or is corrupted, it could lead to the "No such file or directory" error.
- **Recommendation**: Verify that the weights file is in the correct format expected by the `safe_open` function. If the file is not a valid `safetensors` file, it will not be loaded correctly.
4. **LoRA Injection and Realization**: The code contains methods for injecting LoRA layers (`inject_trainable_lora`, `inject_trainable_lora_extended`) and realizing them as part of the model (`collapse_lora`). If these processes are not executed correctly, it could lead to the model being in an incomplete or incorrect state.
- **Recommendation**: Review the logic related to LoRA injection and realization to ensure that it is being applied correctly. Ensure that the LoRA layers are properly initialized and that their weights are being set as expected.
In summary, the user should verify the file paths for the LoRA weights, add error handling to provide informative messages if the weights file cannot be found or loaded, and ensure that the LoRA injection and realization processes are correctly implemented. Additionally, it would be helpful to have unit tests to validate the functionality of the LoRA-related operations to catch any issues early in the development process.
</details>
<details>
<summary>Changes on inference.py</summary>
After reviewing the `inference.py` file, here are some observations and recommendations:
1. **Model Loading Path**: The `initialize_pipeline` function loads the model using `UNet3DConditionModel.from_pretrained(model, subfolder='unet')`. It is important to ensure that the `model` variable passed to this function contains the correct path to the model directory. If the path is incorrect, the model will not load, which could lead to the 'NoneType' object has no attribute 'items' error.
2. **Error Handling**: There is no explicit error handling for the case when the model fails to load. It would be beneficial to add checks after the model loading lines to ensure that the model is not `None`. If the model is `None`, an informative error message should be displayed to the user.
3. **LoRA Weights**: The `inject_inferable_lora` function is called with `pipeline`, `lora_path`, and `lora_rank` as arguments. It is crucial to verify that the `lora_path` is correct and points to the location where the LoRA weights are stored. If the path is incorrect, the function will not be able to load the LoRA weights, which could be the cause of the error.
4. **File Paths in Arguments**: The script accepts file paths as arguments (e.g., `--model`, `--init-video`). Users should be careful to provide the correct paths when running the script. It might be helpful to add checks to verify that the provided file paths exist before attempting to use them.
5. **Model Loading Checks**: Before calling functions that use the model (e.g., `vid2vid`, `pipeline`), there should be checks to ensure that the model and its components (e.g., `scheduler`, `tokenizer`, `text_encoder`, `vae`, `unet`) are loaded correctly and are not `None`.
6. **Output Directory**: The script creates an output directory if it does not exist. However, there is no check to see if the directory creation was successful. Adding a check after `os.makedirs(output_dir, exist_ok=True)` could prevent issues related to directory access or creation.
7. **Inference Function**: The `inference` function is the main entry point for generating videos. It is important to ensure that all the necessary components are loaded correctly before proceeding with video generation. If any component fails to load, the function should return an error message.
In summary, the main focus should be on verifying file paths, adding error handling to check for successful model loading, and ensuring that the LoRA weights are correctly loaded. These steps will help prevent the 'No such file or directory' and 'NoneType' object has no attribute 'items' errors.
</details>
<footer>
_Disclaimer: This comment was entirely generated using AI. Be aware that the information provided may be incorrect._
Current plan usage: 0.00%
**Have feedback or need help?**
[Discord](https://discord.gg/r72ykfvyx7)
[Documentation](https://docs.codeautopilot.com/)
[support@codeautopilot.com](mailto:support@codeautopilot.com)
</footer>
What happened? I am trying to generate video after loading a finetuned/ trained lora model, although the model gets loaded, it doesnt generate video it says the above error, first the error was the file not found, where the weights of the model are located, if i give a direct path in the code then it says 'NoneType' object has no attribute 'items' Error.
Steps to reproduce the problem Go to webui and open text2video generation load the trained lora text to video model in the option given at the bottom give a prompt and generate a video after you generate this error should come in the terminal Screenshot 2024-02-14 144855
file where the webui lora model located along with the weights 111
21
What should have happened? Normally it should work witout giving any errors
WebUI and Deforum extension Commit IDs webui commit id - txt2vid commit id -
Torch version '2.2.0'
What GPU were you using for launching? NVIDIA RTX A4000 16GB
On which platform are you launching the webui backend with the extension? Local PC setup (Windows)
Settings version: [v1.7.0] • python: 3.10.6 • torch: 2.0.1+cu118 • xformers: N/A • gradio: 3.41.2 • checkpoint: [6ce0161689]
Console logs C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui>webui-user venv "C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui\venv\Scripts\Python.exe" Python 3.10.6 | packaged by conda-forge | (main, Oct 24 2022, 16:02:16) [MSC v.1916 64 bit (AMD64)] Version: v1.7.0 Commit hash: cf2772fab0af5573da775e7437e6acdca424f26e Launching Web UI with arguments: no module 'xformers'. Processing without... no module 'xformers'. Processing without... No module 'xformers'. Proceeding without it. Style database not found: C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui\styles.csv Loading weights [6ce0161689] from C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui\models\Stable-diffusion\v1-5-pruned-emaonly.safetensors C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui\extensions\sd-webui-text2video\scripts\text2vid.py:48: GradioDeprecationWarning: The style method is deprecated. Please set these arguments in the constructor instead. with gr.Row(elem_id='t2v-core').style(equal_height=False, variant='compact'): Running on local URL: http://127.0.0.1:7860/ Creating model from config: C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui\configs\v1-inference.yaml
To create a public link, set share=True in launch(). Startup time: 24.4s (prepare environment: 5.4s, import torch: 6.7s, import gradio: 2.8s, setup paths: 2.6s, initialize shared: 0.6s, other imports: 1.9s, setup codeformer: 0.5s, load scripts: 1.9s, create ui: 0.7s, gradio launch: 1.2s). Applying attention optimization: Doggettx... done. Model loaded in 18.4s (load weights from disk: 1.1s, load config: 0.2s, create model: 0.5s, apply weights to model: 10.2s, apply half(): 2.9s, calculate empty prompt: 3.3s). text2video — The model selected is: (ModelScope-like) text2video extension for auto1111 webui Git commit: 989f5cfe Starting text2video Pipeline setup config namespace(framework='pytorch', task='text-to-video-synthesis', model={'type': 'latent-text-to-video-synthesis', 'model_args': {'ckpt_clip': 'open_clip_pytorch_model.bin', 'ckpt_unet': 'text2video_pytorch_model.pth', 'ckpt_autoencoder': 'VQGAN_autoencoder.pth', 'max_frames': 16, 'tiny_gpu': 1}, 'model_cfg': {'unet_in_dim': 4, 'unet_dim': 320, 'unet_y_dim': 768, 'unet_context_dim': 1024, 'unet_out_dim': 4, 'unet_dim_mult': [1, 2, 4, 4], 'unet_num_heads': 8, 'unet_head_dim': 64, 'unet_res_blocks': 2, 'unet_attn_scales': [1, 0.5, 0.25], 'unet_dropout': 0.1, 'temporal_attention': 'True', 'num_timesteps': 1000, 'mean_type': 'eps', 'var_type': 'fixed_small', 'loss_type': 'mse'}}, pipeline={'type': 'latent-text-to-video-synthesis'}) Error verifying pickled file from C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui\models\Lora\lora\full_weights The file may be malicious, so the program is not going to read it. *** You can skip this check with --disable-safe-unpickle commandline argument.
Traceback (most recent call last): File "C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui\modules\safe.py", line 137, in load_with_extra check_pt(filename, extra_handler) File "C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui\modules\safe.py", line 83, in check_pt with zipfile.ZipFile(filename) as z: File "C:\Users\INP_Rohit.conda\envs\ttv\lib\zipfile.py", line 1249, in init self.fp = io.open(file, filemode) FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui\models\Lora\lora\full_weights\' Traceback (most recent call last): File "C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui/extensions/sd-webui-text2video/scripts\t2v_helpers\render.py", line 30, in run vids_pack = process_modelscope(args_dict, args) File "C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui/extensions/sd-webui-text2video/scripts\modelscope\process_modelscope.py", line 71, in process_modelscope stable_lora_processor.process(pipe, stable_lora_args) File "C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui/extensions/sd-webui-text2video/scripts\stable_lora\scripts\lora_webui.py", line 203, in process self.process_lora(args, undo_merge=False) File "C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui\venv\lib\site-packages\torch\amp\autocast_mode.py", line 14, in decorate_autocast return func(args, kwargs) File "C:\Users\INP_Rohit\Documents\ImageGeneration\stable-diffusion-webui/extensions/sd-webui-text2video/scripts\stable_lora\stable_utils\lora_processor.py", line 217, in process_lora for k, v in lora_model.items(): AttributeError: 'NoneType' object has no attribute 'items' Exception occurred: 'NoneType' object has no attribute 'items'
I have added the link of issue mentioned explicitly with images below:
https://github.com/kabachuha/sd-webui-text2video/issues/241
Can you please help me out with this asap