RedFoxNightfox
commented
9 months ago reverting back to 3.2.2 the issue goes away, so it's definitely 3.2.3 that's the problem.
Closed RedFoxNightfox closed 9 months ago
RedFoxNightfox
commented
9 months ago reverting back to 3.2.2 the issue goes away, so it's definitely 3.2.3 that's the problem.
BlafKing
commented
9 months ago That was a little print I did for bug testing, forgot to remove it 😅 my bad about that.
3.2.2 to 3.2.3 mainly just inverted the One-Time Hash Generation for externally downloaded models option, so is everything working other than the print?
The latest commit removes this print.
RedFoxNightfox
commented
9 months ago 3.2.3 also failed to randomly find loras and checkpoints that are on civitAI, and fills the modelID line of the json file with "Model not found" causing 400 errors when making API calls. After the code rollback, I had to write a script to delete the random json files with this to fix the 400 errors, and then the API calls went through, and correctly found those same loras and checkpoints.
BlafKing
commented
9 months ago Not sure what could cause it to randomly not find any LORA's and Checkpoints other than the CivitAI servers not working. The script fetches the Model ID by generating the file's SHA256 hash, and then sending an API request with that hash to retrieve the Model ID. I'll make it so that it tries to retrieve the model ID again if the model ID is set to "Model not found".
The code for retrieving model ID's with this addition is as follows:
def get_models(file_path, gen_hash=None):
modelId = None
sha256 = None
json_file = os.path.splitext(file_path)[0] + ".json"
if os.path.exists(json_file):
try:
with open(json_file, 'r') as f:
data = json.load(f)
if 'modelId' in data:
modelId = data['modelId']
if 'sha256' in data:
sha256 = data['sha256']
except Exception as e:
print(f"Failed to open {json_file}: {e}")
if not modelId or not sha256 or modelId == "Model not found": # added `or modelId == "Model not found"`
if gen_hash:
if not sha256:
sha256 = gen_sha256(file_path)
by_hash = f"https://civitai.com/api/v1/model-versions/by-hash/{sha256}"
else:
if modelId:
return modelId
else:
return None
try:
if not modelId or modelId == "Model not found": # added `or modelId == "Model not found"`
response = requests.get(by_hash, timeout=(10,30))
if response.status_code == 200:
api_response = response.json()
if 'error' in api_response:
print(f"\"{file_path}\": {api_response['error']}")
return None
else:
modelId = api_response.get("modelId", "")
elif response.status_code == 503:
return "offline"
elif response.status_code == 404:
api_response = response.json()
modelId = api_response.get("error", "")
if os.path.exists(json_file):
try:
with open(json_file, 'r') as f:
data = json.load(f)
data['modelId'] = modelId
data['sha256'] = sha256.upper()
with open(json_file, 'w') as f:
json.dump(data, f, indent=4)
except Exception as e:
print(f"Failed to open {json_file}: {e}")
else:
data = {
'modelId': modelId,
'sha256': sha256.upper()
}
with open(json_file, 'w') as f:
json.dump(data, f, indent=4)
return modelId
except requests.exceptions.Timeout:
print(f"Request timed out for {file_path}. Skipping...")
return "offline"
except requests.exceptions.ConnectionError:
print("Failed to connect to the API. The CivitAI servers might be offline.")
return "offline"
except Exception as e:
print(f"An error occurred for {file_path}: {str(e)}")
return NoneMost of this code is unchanged compared to v3.2.2
Please let me know if you still experience any issues after this change :)
RedFoxNightfox
commented
9 months ago updated to latest script, and edited the code with above.
ran the "update model info & tags" again with "Overwrite any existing previews, tags or descriptions." and "One-Time Hash Generation for externally downloaded models." selected.
the process takes a very long time compared to 3.2.2 still, as if it's trying to rehash the sha256 for every file.
While a1111 says "processing file: " console shows CivitAI Browser+: model: "sonicdiffusion_v3Beta4.safetensors" not found on CivitAI servers." along with various others, and does still write ''"modelId": "Model not found"'' to a json file, unlike 3.2.2 which does not create a json file if the model is not found. It seems that the mistakes from before was indeed on civitAI's end.
API requests did fully go through this time, however, without any 400 error.
reverting back to 3.2.2, running my script to delete the "model not found" jsons, restarting a1111 and rerunning the scan with the same exact scenario, the behavior is as expected, being very quick and not re-hashing everything, I am unsure as to why 3.2.3 is rehashing every file every time, but for larger libraries like mine, that is a unnecessarily long time.
But yes, the code for the replaced "def get_models" does indeed work to fix the API issues also experienced.
BlafKing
commented
9 months ago v3.2.2 has a fault in the Overwrite any existing previews, tags or descriptions. option, in that version, when it's turned on it's actually turned off, and vice versa, this is probably why 3.2.2 is going much faster for you.
It should however not generate a sha256 hash when one already exists in the model's matching .json file. Here is the relevant code again where I explain each line:
if os.path.exists(json_file): # checks if the model's matching json file exists
try:
with open(json_file, 'r') as f: # opens the file
data = json.load(f) # loads the content to `data` variable
if 'modelId' in data: # checks if modelId exists in the json file
modelId = data['modelId'] # if it exists assign the ID to modelId variable
if 'sha256' in data: # checks if sha256 hash exists in the json file
sha256 = data['sha256'] # assigns the hash to the sha256 variable
except Exception as e:
print(f"Failed to open {json_file}: {e}")
if not modelId or not sha256 or modelId == "Model not found": # checks if modelID or sha256 variables are None, or if modelID is set to "Model not found"
if gen_hash: # Checks if the "Overwrite any existing previews, tags or descriptions." is turned on
if not sha256: # Checks again if a sha256 value is already found in the json file, if so it does not generate a new hash
sha256 = gen_sha256(file_path) # generates a new hash and assigns it to the sha256 variable
by_hash = f"https://civitai.com/api/v1/model-versions/by-hash/{sha256}" # creates the API url with the sha256 hash.So I'm unsure how it's generating a new hash for each file for you if that's the case. As long as a sha256 field exists in the model's matching .json file, then it should not generate a new sha256 hash.
I'll go ahead and close this bug report, as I can't see any issues with the current code. Please feel free to continue sending messages here if you have more questions though :)
Describe the bug.
when scanning for model updates, or model information the latest version prints the following to consol:
CivitAI Browser+: modelId of sha256 niette vonde CivitAI Browser+: modelId of sha256 niette vonde CivitAI Browser+: modelId of sha256 niette vonde CivitAI Browser+: modelId of sha256 niette vonde CivitAI Browser+: modelId of sha256 niette vonde CivitAI Browser+: model: "t00nl1nk-V1.safetensors" not found on CivitAI servers. CivitAI Browser+: modelId of sha256 niette vonde CivitAI Browser+: modelId of sha256 niette vonde CivitAI Browser+: modelId of sha256 niette vonde
Steps to reproduce the problem.
Expected behavior
It's supposed to DL the tags an info, but fails to do anything now. I even tried deleting all .json files in all the models subfolders, and it still fails.
System info
Console logs
Additional information
was working prior to updating to current version from the last release. dunno what changed.