[Bug] Still losing information

[JingJang]

Related to close issue #11

It still seems to delete information. I attached two files you can test on.

3d Render Style XL Attached File The model has been deleted. I think this is one from 1.7.0 but I am not sure. It should be able to extract all the information as it was created by the addon.

Possible checking method: 1) Check if file exists online. If it does then get information from the web and merge trigger words from old file with new one, as user may manually add words.

2) If file does not exist online, check if "version info" property exist, then you realise the file contains a merger between Model and Model Version JSON data. The data should be updated accordingly.

3) If file does not exist online, and "version info" property does not exist. It means, this is an older version and it is only contains the Model Version JSON data. The JSON data should be kept and updated.

Luffy Attached File Luffy (This one I created manually). This one is not so important, but it would be nice if it can just transfer the trigger words (or as much info as possible).

Thank you

3d_render_style_xl.civitai.info.txt Luffy.info.txt

[zixaphir]

I tried to do some refactoring. Let me know if this is still happening.

[JingJang]

Thank you for the update. I just managed to try it now. And it seems to hash all the files even when the info file already contains the checksum. It's really slow when you have a 100s of gigabytes of data.

I had to remove all the files and just put the wanted test files, just to check if it works. It does not edit or add any detail. Just skips the metadata.

Civitai Helper: Civitai does not have this model
Civitai Helper: 'New metadata id () does not match old metadata id (129610)', aborting

I thought it might just update existing structure and add the last time it checked and version and maybe another property stating Civitai update failed. Something like so.

    "extensions": {
        "sd_civitai_helper": {
            "version": "1.7.5",
            "last_update": 1697223432,
            "update": failed
        }
    }

If the metadata exist and its still on Civitai I get this:

Civitai Helper: Model metadata not needed for Barbie_❴4B0FE67F4F❵.safetensors
Civitai Helper: Downloading model image.

Which perfect but it does not update the extensions property. It's still at 1.7.2, not updated to 1.7.5 version.

    "extensions": {
        "sd_civitai_helper": {
            "version": "1.7.2"
        }
    }

But at least it does not delete the metadata. So that works. But hashing all the files again when you have the hash and file size stored in the info seems reductive.

I also now get this error, which I didn't before.

Civitai Helper: Civitai does not have this model
Traceback (most recent call last):
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1103, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "T:\SD\stable-diffusion-webui\venv\lib\site-packages\gradio\utils.py", line 707, in wrapper
    response = f(*args, **kwargs)
  File "T:\SD\stable-diffusion-webui\extensions\Stable-Diffusion-Webui-Civitai-Helper\scripts\ch_lib\model_action_civitai.py", line 125, in scan_model
    success = scan_single_model(filename, root, model_type, refetch_old, delay)
  File "T:\SD\stable-diffusion-webui\extensions\Stable-Diffusion-Webui-Civitai-Helper\scripts\ch_lib\model_action_civitai.py", line 78, in scan_single_model
    model_info = dummy_model_info(item, sha256_hash, model_type)
  File "T:\SD\stable-diffusion-webui\extensions\Stable-Diffusion-Webui-Civitai-Helper\scripts\ch_lib\model_action_civitai.py", line 177, in dummy_model_info
    file_metadata = sd_models.read_metadata_from_safetensors(path)
  File "T:\SD\stable-diffusion-webui\modules\sd_models.py", line 269, in read_metadata_from_safetensors
    assert metadata_len > 2 and json_start in (b'{"', b"{'"), f"{filename} is not a safetensors file"
AssertionError: T:\SD\stable-diffusion-webui\models\Stable-diffusion\NovelAI.ckpt is not a safetensors file

It's a ckpt file. I have no clue why this occurs.

[zixaphir]

It's really slow when you have a 100s of gigabytes of data.

Probably an oversight on my part. I'll see about working that out.

I thought it might just update existing structure

This is not really in the works. Currently changing the structure of the metadata is done by just re-downloading the metadata and overwriting it. Part of the reason I haven't implemented appending new data to existing model info is because it would require comparing every field and making some assumptions (like whether the user edited it or did the model author change the metadata?), which is still easier than picking an entire branch, moving it to a different part of the model data, and hoping I haven't fucked with anything.

and add the last time it checked and version and maybe another property stating Civitai update failed. Something like so.

This is a work in progress. I've gotten some unused shim code in for this, I just need to finalize it and write in the function calls to use it.

I also now get this error, which I didn't before.

This was an oversight on my part and should be fixed now. I don't know why I just assumed all models would be safetensors.

[zixaphir]

I'm closing this issue since the main bug has been addressed. I will continue further work on updating the existing model metadata, but I do not want to give potential users reading the issue list the impression that they may lose data when that issue has been resolved. If you wish to track development on updating the metadata in-place, please open a new issue as a feature request and I will keep you updated. :)