SmirkingKitsune
commented
2 months ago That's an interesting proposition. Wondering on the possibility of operating the clip_interrogator_ext.py script from clip-interrogator-ext , from the sd_tag_batch.py script.
It seems clip_interrogator_ext.py has an API we can use to collect the models and run the prompt generation. Since an API will be used --api will need to be added to startup command. The script currently uses the native interrogators from A1111, so API handling has not been implemented (yet). I am going to try to make a rough outline of how we might do this, but this is a first draft and may contain errors and oversights.
THE FOLLOWING IS UNTESTED I will be adding personal notes about thoughts on code snippets.
Add CLIP API to the model_options like this:
model_options = ["CLIP", "Deepbooru", "CLIP API"]Should investigate hiding "CLIP API" if the CLIP API is not available to remove clutter... At the very least add a reporting mechanism to indicate that CLIP API was not found...
Need to add CLIP API options to the UI, will probably look like this:
# CLIP API Options
with gr.Accordion("CLIP API Options:"):
clip_api_model = gr.Dropdown(get_clip_interrogator_models(), value='ViT-L-14/openai', label="CLIP API Model")
clip_api_mode = gr.Radio(choices=["fast", "best", "classic", "negative"], label="CLIP API Mode", value="fast")I am unsure about the dropdown, needs further testing. Maybe we could do multiselect to run multiple CLIP API models...
Change the UI return to accomidate the CLIP API options:
return [in_front, prompt_weight, model_selection, use_weight, no_duplicates, use_negatives, use_custom_filter, custom_filter, clip_api_model, clip_api_mode]Change the run definition:
def run(self, p, in_front, prompt_weight, model_selection, use_weight, no_duplicates, use_negatives, use_custom_filter, custom_filter, clip_api_model, clip_api_mode):Add the CLIP API to the model interrogation loop:
elif model == "CLIP API":
interrogator += self.get_clip_api_prompt(p.init_images[0], clip_api_model, clip_api_mode)Write some definitions for fastAPI interactions: First would be the CLIP API model collector:
def get_clip_interrogator_models(self, image):
# Ensure CLIP Interrogator is present and accessible
try:
response = requests.get("http://127.0.0.1:7860/interrogator/models")
response.raise_for_status()
models = response.json()
if not models:
raise Exception("No CLIP Interrogator models found.")
except Exception as error:
print(f"Error accessing CLIP Interrogator API: {error}")
return ""
return modelsUnsure if this would work, might need to convert models, need to check on that.
Next would be the CLIP API prompt handler:
def get_clip_api_prompt(self, image, model_name, mode):
# Ensure the model name and mode are provided
if not model_name:
print("CLIP API model name is required.")
return ""
if mode not in ["fast", "best", "classic", "negative"]:
print("Invalid CLIP API mode.")
return ""
# Encode the image to base64
buffered = BytesIO()
image.save(buffered, format="PNG")
img_str = base64.b64encode(buffered.getvalue()).decode("utf-8")
# Get the prompt from the CLIP API
try:
payload = {
"image": img_str,
"mode": mode,
"clip_model_name": model_name
}
response = requests.post("http://127.0.0.1:7860/interrogator/prompt", json=payload)
response.raise_for_status()
result = response.json()
return result.get("prompt", "")
except Exception as error:
print(f"Error generating prompt from CLIP API: {error}")
return ""Unsure if this definition would work, need to further investigate CLIP API, and troubleshoot. Should not be using localhost 7860, instead port should be collected from A1111, need to investigate how to properly do that... Need to investigate how to unload model, as it could be detrimental to VRAM if left loaded. At the very least, need to leave note to user indicating that model can be unloaded from the CLIP API tab...
This may require substantial testing, however if successful, we might be able to duplicate this for other tagging extensions like stable-diffusion-webui-wd14-tagger
Pawelekkkkk
I'd like to use custom interrogator. Could be from CLIP Interrogator tool like ViT-bigG-14/laion2b_s39b_b160k. Where to put models so batch interrogator will find them or how to do it any other way. Also, would be great to have same possibility to choose between: best, fast classic and negative mode. Thanks for great tool.