Using nothing but MD5 and a length is not particularly strong protection against malicious files being served from a file host. While this whole project does functionally require some significant amount of trust, it's quite a bit over the line to directly run executable files that were downloaded from the internet without more than a length and MD5 check.
Proposal:
Allow specifying, for any given file, checksums from additional supported hash algorithms.
Once a file passes MD5, if it specifies these additional checksums, don't release it to the main tasks until it also passes all of those additional checks.
Slap these onto probably any file that isn't hosted by Nexus Mods, and ideally anything else with DLL, EXE, or JAR files that we either run or encourage users to run.
It's fine to continue using MD5 as an identifier internally because it's unreasonable to imagine collisions among probably not more than 1000 files within any particular suite of pack files.
It's probably overkill to require an SHA-2 family checksum for each and every file even if it just has some textures. Maybe it's not insane to replace MD5 with SHA-512, though I expect this community to strongly favor speed over security, and I'm already concerned about the speed of running gigabytes of data through MD5.
Using nothing but MD5 and a length is not particularly strong protection against malicious files being served from a file host. While this whole project does functionally require some significant amount of trust, it's quite a bit over the line to directly run executable files that were downloaded from the internet without more than a length and MD5 check.
Proposal:
It's fine to continue using MD5 as an identifier internally because it's unreasonable to imagine collisions among probably not more than 1000 files within any particular suite of pack files.
It's probably overkill to require an SHA-2 family checksum for each and every file even if it just has some textures. Maybe it's not insane to replace MD5 with SHA-512, though I expect this community to strongly favor speed over security, and I'm already concerned about the speed of running gigabytes of data through MD5.