Loading aborted due to error reloading workflow data

[hobojoker]

After spending tons of hours working on a layout, I when I closed it down and reopened, suddenly I was getting an error. Apparently the error had been there for the last 10 saves but it never notified me!

Unfortunately, the error handling is awful and completely useless to help... so I manually dug through the file looking for differences from the most recent working save, and oldest not working save. I managed to find the error and fix it, but it was unpleasant and time consuming to say the least!

This is the error I was getting:

This was what the error ended up being:

There was an ID mismatch, and an ID was duplicated (in red). Deleting the part in yellow allowed the save to be loaded.

Honestly, while this error handling is terrible and you can easily lose hours of work, I'd gladly take occasional errors like this if I could get some additional customizable keyboard shortcuts!!

[hobojoker]

Alright, I wrote a quick python script to parse the saved workflows and remove any keys where the id does not match, as I looked at my later saves I ended up with A LOT of them! After running the scripts my workflows seem to have no issues at all and are working properly.

If you can't identify where the source of these errors is, at least running this script on your save files will keep you progressing!

import json

def check_id_and_remove(json_data):
    if "layout" in json_data and "allItems" in json_data["layout"]:
        all_items = json_data["layout"]["allItems"]
        keys_to_remove = []
        for key, value in all_items.items():
            if isinstance(value, dict) and "dragItem" in value and "id" in value["dragItem"]:
                if key != value["dragItem"]["id"]:
                    keys_to_remove.append(key)
        for key in keys_to_remove:
            all_items.pop(key, None)

if __name__ == "__main__":
    file_path = input("Enter the path of the JSON file: ")
    try:
        with open(file_path, "r") as json_file:
            data = json.load(json_file)
            check_id_and_remove(data)

        with open(file_path, "w") as json_file:
            json.dump(data, json_file, indent=2)

    except FileNotFoundError:
        print(f"File not found: {file_path}")
    except json.JSONDecodeError:
        print(f"Invalid JSON format in file: {file_path}")

[hobojoker]

Alright, as the save files continue to accumulate errors I have developed this further to clean them up:

Basically what keeps happening is that there are leftover IDs as children/parents and mismatched ID and keys. This basically just goes through the .json, finds all the matching Key/ID pairs, then checks to make sure there are no lonely IDs hanging out anywhere else.

This is super clunky, and purely functional, not pretty.

import json
import shutil
import tkinter as tk
from tkinter import filedialog

def get_mismatched_ids(json_data):
    counter = 0
    mismatched_key = set()
    if "layout" in json_data and "allItems" in json_data["layout"]:
        all_items = json_data["layout"]["allItems"]
        for key, value in all_items.items():
            if isinstance(value, dict) and "dragItem" in value and "id" in value["dragItem"]:
                if key != value["dragItem"]["id"]:
                    mismatched_key.add(key)
                    mismatched_key.add(value["dragItem"]["id"])
                    # this is +2 because there both the ID and key count!
                    counter = counter + 2
    print(f"mismatch: {counter}")
    return mismatched_key

def get_valid_keys(json_data):
    matched_key = set()

    if "layout" in json_data and "allItems" in json_data["layout"]:
        all_items = json_data["layout"]["allItems"]
        for key, value in all_items.items():
            if isinstance(value, dict) and "dragItem" in value and "id" in value["dragItem"]:
                if key == value["dragItem"]["id"]:
                    matched_key.add(key)

    return matched_key

def get_keys_to_remove(json_data):

    mismatched_key = set()
    if "layout" in json_data and "allItems" in json_data["layout"]:
        all_items = json_data["layout"]["allItems"]
        for key, value in all_items.items():
            if isinstance(value, dict) and "dragItem" in value and "id" in value["dragItem"]:
                if key != value["dragItem"]["id"]:
                    mismatched_key.add(key)

    for key in mismatched_key:
        all_items.pop(key, None)

def get_matching_ids(json_data, ids):
    has_valid_pair = set()
    if "layout" in json_data and "allItems" in json_data["layout"]:
        all_items = json_data["layout"]["allItems"]
        for key, value in all_items.items():
            if isinstance(value, dict) and "dragItem" in value and "id" in value["dragItem"]:
                if key == value["dragItem"]["id"] and (key in ids):
                    has_valid_pair.add(key)

    return has_valid_pair

def remove_id_from_json(json_data, ids, data_remove_count):

    for id_to_remove in ids:
        if isinstance(json_data, dict):
            keys_to_remove = [key for key in json_data if key == id_to_remove]
            for key in keys_to_remove:
                json_data.pop(key, None)

                data_remove_count = data_remove_count + 1

            for value in json_data.values():
                remove_id_from_json(value, id_to_remove,data_remove_count)
        elif isinstance(json_data, list):
            json_data[:] = [item for item in json_data if item != id_to_remove]
        elif isinstance(json_data, str):
            try:
                json_data = json.loads(json_data)
                if isinstance(json_data, (dict, list)):
                    remove_id_from_json(json_data, id_to_remove, data_remove_count)
                    return json.dumps(json_data)
            except json.JSONDecodeError:
                pass

    return json_data

def remove_references(json_data, mismatched_ids):
    if "layout" in json_data and "allItems" in json_data["layout"]:
        all_items = json_data["layout"]["allItems"]
        valid_pairs = get_matching_ids(json_data, mismatched_ids)

        get_keys_to_remove(json_data)

        valid = set()

        valid_counter = 0
        for ids in mismatched_ids:
            if ids in valid_pairs:
                valid.add(ids)
                valid_counter = valid_counter + 1

        for value in valid:
            mismatched_ids.remove(value)

    # TODO: Loop through the entire .json looking specifically for the remaining mismatched ids, it doesn't matter if
    #  they are attached to a drag item or anything else. If found then remove the text with the value of the mismatched id.

    print(f"Valid Counter: {valid_counter}")

def clean_children(children, valid_keys):
    return [child for child in children if child in valid_keys]

def clean_data(item, json_data, valid_keys):
    if "parent" in item and item["parent"] not in valid_keys:
        item.pop("parent")
    if "children" in item:
        item["children"] = clean_children(item["children"], valid_keys)
        for child_id in item["children"]:
            child_item = data["layout"]["allItems"].get(child_id)
            if child_item:
                clean_data(child_item, json_data, valid_keys)

if __name__ == "__main__":
    root = tk.Tk()
    root.withdraw()
    file_path = filedialog.askopenfilename(filetypes=[("JSON Files", "*.json")])

    mismatched_ids = set()

    remove_data_counter = 0

    if file_path:
        try:
            # Create a backup of the original file
            backup_file_path = file_path + ".bak"
            shutil.copyfile(file_path, backup_file_path)

            with open(file_path, "r") as json_file:
                data = json.load(json_file)
                mismatched_ids = get_mismatched_ids(data)
                remove_references(data, mismatched_ids)

                data = remove_id_from_json(data, mismatched_ids, remove_data_counter)

            print(f"Removed ids from json: {remove_data_counter}")
            # If you want to save the updated data back to the same file, uncomment the following lines:
            with open(file_path, "w") as json_file:
                json.dump(data, json_file, indent=2)

            # try removing parents and children

            with open(file_path, "r") as json_file:
                json_data = json.load(json_file)
                if not json_data:
                    print("No file selected or file is invalid.")
                    exit()

                # Determine valid keys
                good_values = get_valid_keys(json_data)

                # Clean the "parent" and "children" fields in the JSON data
                for item in json_data["layout"]["allItems"].values():
                    clean_data(item, json_data, good_values)

            # If you want to save the updated data back to the same file, uncomment the following lines:
            with open(file_path, "w") as json_file:
                json.dump(json_data, json_file, indent=2)

        except FileNotFoundError:
            print(f"File not found: {file_path}")
        except json.JSONDecodeError:
            print(f"Invalid JSON format in file: {file_path}")

[featherice]

Alright, as the save files continue to accumulate errors I have developed this further to clean them up:

Basically what keeps happening is that there are leftover IDs as children/parents and mismatched ID and keys. This basically just goes through the .json, finds all the matching Key/ID pairs, then checks to make sure there are no lonely IDs hanging out anywhere else.

Can you elaborate how to run it correctly? I'm nonprogrammer, spent 6 hours trying to build a workflow and files got corrupted by the same issue.

By the way, i'm not sure what is bugged, but i know how to easily reproduce it. Start new workflow, add lora x5 template, delete 1 of lora subgraphs, at this point file is corrupted.

[hobojoker]

Alright, as the save files continue to accumulate errors I have developed this further to clean them up: Basically what keeps happening is that there are leftover IDs as children/parents and mismatched ID and keys. This basically just goes through the .json, finds all the matching Key/ID pairs, then checks to make sure there are no lonely IDs hanging out anywhere else.

Can you elaborate how to run it correctly? I'm nonprogrammer, spent 6 hours trying to build a workflow and files got corrupted by the same issue.

By the way, i'm not sure what is bugged, but i know how to easily reproduce it. Start new workflow, add lora x5 template, delete 1 of lora subgraphs, at this point file is corrupted.

Sorry, I just kind of assumed people using stable diffusion had at least a little bit of experience with code. It's just a basic python program, you already have python installed in order to be able to run stable diffusion, so all you need is a developer environment. You can use visual studio code, or pycharm or something and just copy paste that text in and then run it -- or even more simply you can just copy it to a text file and rename it .py, then execute it from command prompt. It uses just built in functions in python and is super basic - it opens up a file browser, then goes through and collects the trash in the file basically and then resaves it. It also automatically makes a backup of the file.

tldr:

1. Make a new text file
2. copy code to text file
3. rename text file to (whatever name you want).py
4. run command prompt in that window
5. run "python (NameOfFile.py)"
6. select your file

Done!