The results of the test above show that par2cmdline-turbo is the clear winner in terms of performance and memory usage.
The memory usage of wirehair is due to the storage of the file in memory as well as the creation of the encoder. The 1GB file itself takes up 1GB of memory, and then the encoder also takes up another 1GB, for a total of 2GB.
Note: klauspost/reedsolomon does support progressive encoding, but only for the Regular code (which supports up to 256 shards) not the Leopard code (which supports up to 65536 shards). This means that if you have more than 256 shards then you cannot do progressive encoding. As for wirehair, the author said that there is no way to do progressive encoding.
=== Test 2: 1 GB file with 220 shards (200 file shards, 20 recovery shards) ===
Here I used progressive encoding for Klaus Post's implementation so it used less memory.
In this scenario it seems that par2cmdline is only a bit slower than reedsolomon while using the same amount of RAM.
It seems that while in progressive encoding mode klauspost/reedsolomon uses the same amount of RAM as par2cmdline, it also is much slower, to the point where it is not much faster than par2cmdline.
Also wow, par2cmdline-turbo is so much faster than par2cmdline it's crazy! It's like 3 times faster! It does use slightly more RAM than par2cmdline? Hard to tell with my by-the-eyeball methods.
Edit: Looking back at the results, it is surprises me that going from 20 to 100 recovery shards didn't change the time taken at all. I always thought the time taken was proportional to the number of recovery shards?
Programs tested: parchive/par2cmdline, animetosho/par2cmdline-turbo, klauspost/reedsolomon, catid/wirehair
=== Test 1: 1 GB file with 1100 shards (1000 file shards, 100 recovery shards) ===
The results of the test above show that par2cmdline-turbo is the clear winner in terms of performance and memory usage.
The memory usage of wirehair is due to the storage of the file in memory as well as the creation of the encoder. The 1GB file itself takes up 1GB of memory, and then the encoder also takes up another 1GB, for a total of 2GB.
Note: klauspost/reedsolomon does support progressive encoding, but only for the Regular code (which supports up to 256 shards) not the Leopard code (which supports up to 65536 shards). This means that if you have more than 256 shards then you cannot do progressive encoding. As for wirehair, the author said that there is no way to do progressive encoding.
=== Test 2: 1 GB file with 220 shards (200 file shards, 20 recovery shards) ===
Using progressive encoding only (so no wirehair).
par2cmdline: 14 seconds, 0.12 GB par2cmdline-turbo: 5s, 0.15 GB klauspost/reedsolomon: 10 seconds, 0.12 GB
Here I used progressive encoding for Klaus Post's implementation so it used less memory.
In this scenario it seems that par2cmdline is only a bit slower than reedsolomon while using the same amount of RAM.
It seems that while in progressive encoding mode klauspost/reedsolomon uses the same amount of RAM as par2cmdline, it also is much slower, to the point where it is not much faster than par2cmdline.
Also wow, par2cmdline-turbo is so much faster than par2cmdline it's crazy! It's like 3 times faster! It does use slightly more RAM than par2cmdline? Hard to tell with my by-the-eyeball methods.
Reposted on my blog: https://1f604.blogspot.com/2023/04/comparison-of-par2-reedsolomon-and.html
Edit: Looking back at the results, it is surprises me that going from 20 to 100 recovery shards didn't change the time taken at all. I always thought the time taken was proportional to the number of recovery shards?