Additional Interrogator, AlwaysVisible and Experimental (Fun) Tools Update

[SmirkingKitsune]

This update implements a new set of features to improve the interrogations of img2img interrogations. Now additional interrogators from A1111 extensions can be used to interrogate images. Additionally, I added some UI cleanup to keep from over cluttering the UI. This should also resolve issue #9.

New Interaction Suite: Changed in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/1f8d5373503c2c5f77ab476934180e1813b2fe33.

• Added API awareness. Removed in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/1f8d5373503c2c5f77ab476934180e1813b2fe33.
• Added Interaction Hooks for:
  • CLIP Interrogator Ext ('clip-interrogator-ext')
  • WD14 Tagger ('stable-diffusion-webui-wd14-tagger')

UI Overhaul:

• Clutter Cleanup, now unused options and sliders are hidden
• Changed the up_front checkbox to a radio.
• Created dynamically visible options accordion for:
  • CLIP Interrogator Ext ('clip-interrogator-ext')
  • WD14 Tagger ('stable-diffusion-webui-wd14-tagger')
• Users that do not have the API interrogators will not see the option to use them in the model_selector. Changed in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/1f8d5373503c2c5f77ab476934180e1813b2fe33.
  • Made extension always visible on the img2img tab. This will allow for other scripts to be run alongside extension. Added in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/6b1b47f3929bfebeffecf2f81dcf9d494f6e726d Removed pending investigation in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/b7f1440367a332d696dd4e217ab3d54f6e5677cf
  • Added triggers to determine if the script should run. Triggers are if a model is selected or if an image is present. Added in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/6b1b47f3929bfebeffecf2f81dcf9d494f6e726d Removed pending investigation in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/b7f1440367a332d696dd4e217ab3d54f6e5677cf
  • Removed "Deepbooru (Native)" from being the default model, so that script is not active by default. Added in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/6b1b47f3929bfebeffecf2f81dcf9d494f6e726d

Bug Fixes:

• Replaced the backward slashes \ with forward slashes / for better OS compatibility.
• Fixed some typos in the comments
  • Library declarations needed to be organized and cleaned of repetition. Added in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/6b1b47f3929bfebeffecf2f81dcf9d494f6e726d
  • Script will no longer run if there is no image to interrogate, (I think that img2img probably should not run if there is no images anyways...) Added in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/6b1b47f3929bfebeffecf2f81dcf9d494f6e726d Removed pending investigation in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/b7f1440367a332d696dd4e217ab3d54f6e5677cf

Special Notes:

• The get_server_address function has a fallback from an earlier iteration of this update. As far as I can tell through testing, it will not need the fallback. It can be removed, but it is there just in case. Removed in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/1f8d5373503c2c5f77ab476934180e1813b2fe33.
• The WD14 Tagger API often deadlocked when I was testing it, but I think it was my system it did not like. I put a timeout on it to prevent A1111 from becoming deadlocked. However, I am unsure if the timeout time is too long or too short. No longer matters as of https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/1f8d5373503c2c5f77ab476934180e1813b2fe33.
• There is no way to unload the CLIP Interrogator models from img2img-batch-interrogator, it must be unloaded from the CLIP Interrogator tab. Unsure if we should place a label indicating that CLIP API models are not unloaded and need to be unloaded in their tab. Fixed in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/1f8d5373503c2c5f77ab476934180e1813b2fe33. - I was unsure if we should add the option to unload WD14 models after use. Since it would be unloaded then reloaded over and over in a batch job. However, to do that simply run the post_wd_api_unload function in the run. Added this in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/e47443ee98ab852e149c8395f98f3bfc1843927b
• WD14 seems to output a lot of tags with underscores in it, so I implemented an option to remove the underscores when parsing WD14 prompts.

[SmirkingKitsune]

~~BIG IMPORTANT SPECIAL NOTE: Since this uses the API, it might require the --api command argument to be declared at runtime.~~ No longer matters as of https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/1f8d5373503c2c5f77ab476934180e1813b2fe33.

[SmirkingKitsune]

Supplementary note:

• I set the default of the CLIP API Options to be ViT-L-14/openai model and Fast mode. This is slightly different from the defaults in 'clip-interrogator-ext', where they use Best mode. Default of the CLIP API Options to be ViT-L-14/openai model and Best mode. Changed in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/6b1b47f3929bfebeffecf2f81dcf9d494f6e726d
• I set the default of the WD14 API Options to be wd-v1-4-moat-tagger.v2 model with a 0.35 threshold. Personally I would prefer to set the model to be one of WD's v3 models my preference being wd-swinv2-tagger.v3, but I also know that wd-vit-v3 is pretty good. Anyways I went with their defaults, since the MOAT tagger is really good with VRAM.

[SmirkingKitsune]

The purpose of https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/6b1b47f3929bfebeffecf2f81dcf9d494f6e726d is so that this script can be run alongside any other script one might want to use in img2img. By making this a AlwaysVisible element on the img2img tab, the script dropdown can select other scripts. Such as in the case of using outpainting or text matrix, but still wanting to interrogate the imported images. Removed pending bug investigation in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/commit/b7f1440367a332d696dd4e217ab3d54f6e5677cf

[SmirkingKitsune]

There is a severe bug in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/commit/6b1b47f3929bfebeffecf2f81dcf9d494f6e726d

[SmirkingKitsune]

Always Visible is the source of the bug. Always Visible functions was rolled back in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/b7f1440367a332d696dd4e217ab3d54f6e5677cf pending further investigation.

[SmirkingKitsune]

Major overhaul to this patch. I was unsatisfied with the performance and restraints that working through the API caused. So API has been removed in favor of direct extension interactions.

Updated initial pull request comment to reflect these changes.

[Pawelekkkkk]

Thank you. I will test it as soon as I will be at home. I really appreciate.

[SmirkingKitsune]

Changes in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/8bc9ece0ec128659ba53deac0292fa5aadabcab5

• Added some documentation.
• Changed some UI wordings.
  • An bug was occurring that caused over exaggeration of tags. This notably happened when using the tagger models (WD and Deepbooru). Multiple models would coincidentally output the same tags, so the interrogations could have multiple instances of the same tag causing exaggeration of the tag. Fixed by making preliminary interrogations filter itself by the interrogation before adding to interrogation.
• An bug was occurring with the underscore remover. It would break underscore emojis that the WD tagger produced. Created a function to identify emoji patterns before determining if underscore should be replaced with a space. The following are the emojis the pattern recognizer is checking for:

  "0_0", "(o)_(o)", "+_+", "+_-", "._.", "<o>_<o>", "<|>_<|>", "=_=", ">_<", "3_3", "6_9", ">_o", "@_@", "^_^", "o_o", "u_u", "x_x", "|_|", "||_||"

[SmirkingKitsune]

There is a known quirk with the model_selector, where the models "CLIP (EXT)" and "WD (EXT)" will still appear the first time users click on the model_selector drop down menu, even if the associated extensions are not installed or enabled. The options will not be selectable if associated extensions are not installed or enabled. The options will disappear the 2nd time the user opens up the model_selector drop down menu.

Attempted Fixes:

• Attempted removing "CLIP (EXT)" and "WD (EXT)" from the model_options list.
  • It caused user with the associated extension to be unable to select "CLIP (EXT)" and "WD (EXT)" until update_model_choices was called. It gets called after user opens the model_selector.
  • Undesirable fix, as it caused users to be unable to select the model they had and wanted to use temporarily. This fix could be interpreted as a bug.
• Attempted to call the update_model_choices function at UI declaration.
  • This caused an error in A1111.

Since the options are not selectable, it is not dangerous to runtime. Since the associated options disappear after the update_model_choices is called after first menu click, it is self correcting. Further investigation required to resolve this quirk. Fixed as of https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/b3743e43b22c6d45abdf5b1bff46861a0f2266df

[SmirkingKitsune]

Changes in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/b3743e43b22c6d45abdf5b1bff46861a0f2266df to resolve model_selector quirk:

• Rewrote the update_model_choices and changed the nature of checking for extensions to resolve quirk. update_model_choices is now the get_initial_model_options.
• The model_selector is no longer dynamic during runtime.

[SmirkingKitsune]

Notes on https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/17b11276b0dcd91c8542ac661ab238d52a81eb5d for bug fix:

• Bug was introduced in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/commit/8bc9ece0ec128659ba53deac0292fa5aadabcab5 that caused extra "CLIP (EXT)" and "WD (EXT)" to be left out of the interrogation. This happened when user added multiple "CLIP (EXT)" or "WD (EXT)" models to the job, as their nested loops were overwriting the interrogation.
• Fixed by adding concatenation to all interrogator statements, instead of at the end of loop.

[SmirkingKitsune]

Changes in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/250d912fc0a360e7b2d819f5f932a964c0a4a0c2:

• A1111 Shared State system has been implemented in the interrogation block of the script.
  • If user gives the skipped command, the interrogation loop will skip the next interrogator.
  • If user gives the interrupt command, the interrogation loop break, but continue onto img2img (press again during image generation to interrupt img2img).

[SmirkingKitsune]

Changes in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/82aa96f3f0b54c6312fe635661c8510ae55dc0e2:

• It was discovered that the Job system in the Shared State was being prematurely reset by CLIP-ext.
• Job states are being saved before CLIP-ext is called and re-declared after CLIP-ext finishes.

Changes in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/1a423bf1dc06c04633c47d270c64f7aabc0e07b9:

• Imported Shared State more concisely.
• Updated Shared State references to be more uniform as State references.

[SmirkingKitsune]

There were a lot of things that needed to be changed to get AlwaysVisible to work correctly and as expected. Along the way, I implemented some tools that I thought would be fun or interesting.

Converted Script for AlwaysVisible Use.

• Changed to process_batch method. - This should run even if only one image is in queue.
• Implemented a prompt cleaner for batch job p.prompt contamination.
  • It resets when job_no is less than or equal to 0, as the start of a batch job is uncontaminated.
• Disabled all settings, to ensure user can operate without script, without needing to disable script, if they choose to - Script requires user to select a model to ensure script runs during img2img job. - prompt_weight is no longer on by default.

Experimental Tools:

• Implemented a verbose development mode, to output to console debugging prints
• Moved the prompt_weight into the experimental tools section.
• Implemented a tool to remove all punctuation from prompt, except for commas, 's, .... and a variety of emojis as no_puncuation_mode
• Added a toggle to the exaggeration filters, as exaggeration_mode
• Added a toggle to place interrogations in the negative prompt instead of the prompt (thought it would be interesting), as reverse_mode

Changes to Filter:

• Changed custom filter to be saved manually, instead of during runtime.
• Changed custom filter to encode as UTF-8, in the event that custom filter prompt includes unusual characters.
• Added a save confirmation.

Other Changes:

• Reorganized all functions by function type then alphabetically, to make it easier to browse script.
• p.init_images[0] is saved prior to conversion and reassigned after interrogation. This is to allow img2img or other scripts to handle the Alpha channel.
• Split the prompt_weight_mode from the in_front logic, so that they operate independently, allowing for contamination cleaner's interrogation collection at the end of a job.
• prompt_weight slider's step scale has been increased by a decimal place.

Special Notes:

• When batch_number is not 0, it means that there is a duplicate job, identical to the previous job, so interrogation does not need to be run.
• The remove_punctuation tool has some known emojis that might trigger the attention syntax system, it needs to be monitored be further investigated on if pruning is necessary.
• p.prompt technically has the correct prompt, but modifying it will be staggered to the next job. The same happens with the process_batch variable prompt[], I suspect this to be a silent bug.
  • To get around this I had to modify the p.all_prompts[]
  • The same happens with p.negative_prompt, and p.all_negative_prompts[] in the reverse_mode tool.

[SmirkingKitsune]

Changes in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/edec8720f1d1a1e535b4abbe3e86a615e1e739c4:

• Pruned some emojis from experimental tool no_punctuation_mode.
• Changed the save button to red, to highlight the destructive capabilities of overwriting a file.
• Changed all custom_filter read/write functions to encode in utf-8
• Added print statements for finding grab-able extensions.
• Optimized how exaggeration is handled in interrogations.
• Added an optimize tool for the custom_filter reusing the new optimized exaggeration code.

[SmirkingKitsune]

Changes in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/50bcbb886811cc3a3841d2c97724143690cef303:

• Adds a button to refresh model_selection selector.

[SmirkingKitsune]

Changes in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/919f4cfb31493c92a731068988435199590f58e5:

• WD extension has the ability to rate interrogated images. This allows users to append that rating to the end of their preliminary WD interrogations so that SD might have a better understanding of the image.
• Added an enabled checkbox into the main accordion to better conform with other accordion's in A1111 interface. This should also ensure that script does not run when script should not run.

[SmirkingKitsune]

Changes in https://github.com/Alvi-alvarez/sd-Img2img-batch-interrogator/pull/10/commits/60bc36d5d9525f447a98bb0f4fc6587e547322fb:

• Added a feature to find and replace words within the interrogation.
  • The find & replace can be used as a way to remove words from entries in the interrogation string, just make sure to leave the replace pair blank.
  • It might be over designed, but I added a non-interactive visualization textbox to visualize how the find & replace pairs entries in the find list to the entries in the replace list.
  • This visualization textbox is only meant for visualization, and does not reflect runtime pairing, as runtime pairing is done during runtime, when the find & replace is applied, unlike visualization pairing which is run when change is detected in the user interactive find & replace textboxes.
• To prevent excessive error logging when custom_filter.txt or custom_replace.txt cannot be found, a blank will be generated when the specific not found error "[Errno 2] No such file or directory: 'extensions/sd-Img2img-batch-interrogator/custom_replace.txt'" is logged to the console.
• Properly imported the ui_components module.