search

lrq3000 / pyFileFixity

📂🛡️Suite of tools for file fixity (data protection for long term storage⌛) using redundant error correcting codes, hash auditing and duplications with majority vote, all in pure Python🐍
MIT License
133 stars 9 forks source link

Getting errors 20% of the way through creating hashes #16

Open JMPFFFFFFFF opened 9 months ago

JMPFFFFFFFF commented 9 months ago

Windows 11, Synology NAS mapped drive

File "", line 198, in _run_module_as_main

File "", line 88, in _run_code

File "C:\Users\lk\AppData\Local\Programs\Python\Python311\Scripts\pff.exe__main__.py", line 7, in

Can I just append and it will continue where it left off?

JMPFFFFFFFF commented 9 months ago

I think my network card was going to sleep. I changed settings and reran and that error is gone. I was able to create the database but I ran into another problem with doing ECC on the headers File "C:\Users\lk\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyFileFixity\lib_compat.py", line 36, in b return codecs.latin_1_encode(x)[0] 'latin-1' codec can't encode character '\u2019' in position 10: ordinal not in range(256)

lrq3000 commented 9 months ago

I don't think it can continue where it left off. Some subcommands can append over a previously created database, and usually all commands can continue if you manually skip all the files that are already done (since ecc files are made per file, there is no central ecc file unlike PAR format), but I don't remember implementing anything in case an error or issue happens midway. This is something I should ponder about more, you're right that this should be taken into consideration to be able to resume.

About the last error you got about latin-1 codec, this is because you have filenames with non strictly ASCII characters, this is the same issue as #13 . I need to fix this, it's currently the most urgent issue, but lacking time... It will be done eventually, it's just a bit complicated (it breaks a lot of current unit tests). This is because this software was first made in Python 2 and then migrated to Python 3, so it inherited the easily breaking string manipulation operations, that now need to be fixed.

JMPFFFFFFFF commented 9 months ago

Do you think a "-" or "underscore" could be the cause? I found some filenames with 2019 in them that had those. If that is the casue, I will just rename them for now.

lrq3000 commented 9 months ago

Mmmm no underscores and dashes should work fine in latin-1.

Can you please maybe share the name of one file it chokes on so I can try to reproduce?

For the content of files there is no issue for sure since they are treated as binary streams. It's just the filenames we can't treat as binary.