Apologies of this has been asked before, I did a couple of searches within the issues and couldn't find anything.
I just solved an error about "error while deserializing header: MetadataIncompleteBuffer". I read all three closed issue reports on the matter and the recommendation was to re-download the models as they might have been corrupted.
Nowhere could I find any output describing which files were the problem. I finally had to edit safetensors/torch.py to catch the error and report what filename it was trying to open.
For anybody that has a similar issue, here's the changes I made:
starting at line 311, change everything until return result with:
try:
with safe_open(filename, framework="pt", device=device) as f:
for k in f.keys():
result[k] = f.get_tensor(k)
except Exception as ex:
print(f"Something went wrong when attempting to open file '{filename}'")
raise ex
My suggestion is this:
When an error is thrown while generating, have some way to report what files were being used when it happened.
In the download_models.py script, add hash verification to the model files to also verify the integrity as well as if they exists. This way, you could just run the downloader again and it should automatically find corrupted files.
Apologies of this has been asked before, I did a couple of searches within the issues and couldn't find anything.
I just solved an error about "error while deserializing header: MetadataIncompleteBuffer". I read all three closed issue reports on the matter and the recommendation was to re-download the models as they might have been corrupted.
Nowhere could I find any output describing which files were the problem. I finally had to edit
safetensors/torch.py
to catch the error and report what filename it was trying to open. For anybody that has a similar issue, here's the changes I made: starting at line 311, change everything untilreturn result
with:My suggestion is this:
download_models.py
script, add hash verification to the model files to also verify the integrity as well as if they exists. This way, you could just run the downloader again and it should automatically find corrupted files.