ParPar Slice/Block Size

[Pluffe]

Hi I'm having same issue as this but I'm in a different platform. Using parpar.exe the command that's being used by nntpposter is having -s 768000 but the creator of parpar says you cannot have more than 32768 input slices in a PAR2 set. I understand this is a calculation issue from nntpposter but I'm not sure, I see it uses same amount for par2cmdline as well -s768000.

[boranblok]

weird, what is the file size you're having issues on ? Maybe it is an edge case. The logic in the issue you mentioned is still active in calculating message size.

[Pluffe]

From less than a gigabyte to 40 I've tested. Couldn't make it work even on small sizes that would not require a bunch of par files.

Log, no matter what size it is the codes and lines are the same. DEBUG 5 ExternalProcessWrappers.ExternalProcessWrapperBase - Process working directory: [Edited\Path\For\working\846769773f9c442c8e8fdd69cca3fa43_readyToPost] DEBUG 5 ExternalProcessWrappers.ExternalProcessWrapperBase - Executing process: [Edited\Path\For\parpar.exe -n -s 768000 -r10% -d pow2 -o "846769773f9c442c8e8fdd69cca3fa43.par2" "846769773f9c442c8e8fdd69cca3fa43.part01.rar" "846769773f9c442c8e8fdd69cca3fa43.part02.rar" "846769773f9c442c8e8fdd69cca3fa43.part03.rar" "846769773f9c442c8e8fdd69cca3fa43.part04.rar" "846769773f9c442c8e8fdd69cca3fa43.part05.rar" "846769773f9c442c8e8fdd69cca3fa43.part06.rar" "846769773f9c442c8e8fdd69cca3fa43.part07.rar" "846769773f9c442c8e8fdd69cca3fa43.part08.rar" "846769773f9c442c8e8fdd69cca3fa43.part09.rar" "846769773f9c442c8e8fdd69cca3fa43.part10.rar" "846769773f9c442c8e8fdd69cca3fa43.part11.rar" "846769773f9c442c8e8fdd69cca3fa43.part12.rar" "846769773f9c442c8e8fdd69cca3fa43.part13.rar" "846769773f9c442c8e8fdd69cca3fa43.part14.rar" "846769773f9c442c8e8fdd69cca3fa43.part15.rar" "846769773f9c442c8e8fdd69cca3fa43.part16.rar" "846769773f9c442c8e8fdd69cca3fa43.part17.rar" "846769773f9c442c8e8fdd69cca3fa43.part18.rar" "846769773f9c442c8e8fdd69cca3fa43.part19.rar" "846769773f9c442c8e8fdd69cca3fa43.part20.rar" "846769773f9c442c8e8fdd69cca3fa43.part21.rar" "846769773f9c442c8e8fdd69cca3fa43.part22.rar" "846769773f9c442c8e8fdd69cca3fa43.part23.rar" "846769773f9c442c8e8fdd69cca3fa43.part24.rar" "846769773f9c442c8e8fdd69cca3fa43.part25.rar" "846769773f9c442c8e8fdd69cca3fa43.part26.rar"] INFO 5 nntpPoster.UsenetPoster - Deleting processed folder DEBUG 5 nntpPoster.NntpMessagePoster - Disposing Posting threads. nntpAutoposter.AutoPoster - The upload failed to post. Retrying. System.Exception: Process has exited with errors. Exit code: 1 at ExternalProcessWrappers.ExternalProcessWrapperBase.ExecuteProcess(String parameters, String workingDirectory) in C:\Users\BoB\source\repos\nntpPoster\ExternalProcessWrappers\ExternalProcessWrapperBase.cs:line 111 at ExternalProcessWrappers.ParWrapper.CreateParFilesInDirectory(DirectoryInfo workingFolder, String nameWithoutExtension, Int32 blockSize, Int32 redundancyPercentage, String extraParams) in C:\Users\BoB\source\repos\nntpPoster\ExternalProcessWrappers\ParWrapper.cs:line 52 at nntpPoster.UsenetPoster.MakeRarAndParFiles(FileSystemInfo toPost, String nameWithoutExtension, DirectoryInfo processedFolder, String password) in C:\Users\BoB\source\repos\nntpPoster\nntpPoster\UsenetPoster.cs:line 204 at nntpPoster.UsenetPoster.PostToUsenet(FileSystemInfo toPost, String title, String rarPassword, Boolean saveNzb, FileInfo nfoFile, Boolean keepProcessingFolderAfterError) in C:\Users\BoB\source\repos\nntpPoster\nntpPoster\UsenetPoster.cs:line 58 at nntpPoster.UsenetPoster.PostToUsenet(FileSystemInfo toPost, String rarPassword, Boolean saveNzb, FileInfo nfoFile, Boolean keepProcessingFolderAfterError) in C:\Users\BoB\source\repos\nntpPoster\nntpPoster\UsenetPoster.cs:line 39 at nntpAutoposter.AutoPoster.PostRelease(WatchFolderSettings folderConfiguration, UploadEntry nextUpload, FileSystemInfo toUpload, Boolean isDirectory) in C:\Users\BoB\source\repos\nntpPoster\nntpAutoposter\AutoPoster.cs:line 180 at nntpAutoposter.AutoPoster.UploadNextItemInQueue() in C:\Users\BoB\source\repos\nntpPoster\nntpAutoposter\AutoPoster.cs:line 131 WARN 5 nntpAutoposter.AutoPoster - Cancelling the upload after 3 retry attempts.

[boranblok]

have you tried executing that exact commandline call (with some testfiles) yourself and verified that works? Your par version might expect different parameters than what I supply.

[Pluffe]

I've tried with the same files using same rar command so sizes will be the same, here's what I get from something I just tested. If I change -s to -s1M it works for the current file but not for others different size files. Cannot determine a slice size to satisfy required number of slices (768000): using a slice size of 396 bytes would produce 771881 slice(s), whilst a size of 400 bytes would produce 764160 slice(s)

[boranblok]

ok, so somehow the -s command gets interpreted not as slice size, but as desired number of slices?

what does the man or /? help of the par command show? How do you specify slice size for your par utility?

[Pluffe]

It was -s 768000 but I changed it to -s1M and it worked. Seems both are valid. Help command show this

https://github.com/animetosho/ParPar/blob/master/help.txt

[Pluffe]

Tried to run the command that's on the log but from a file that's 11 GB (other file was less than half a gigabyte)

parpar.exe -n -s 768000 -r5% -d pow2 -o "cead2b9663d84a89a5939ad7565a7c04.par2" "cead2b9663d84a89a5939ad7565a7c04.part01.rar" "cead2b9663d84a89a5939ad7565a7c04.part02.rar" "cead2b9663d84a89a5939ad7565a7c04.part03.rar" "cead2b9663d84a89a5939ad7565a7c04.part04.rar" "cead2b9663d84a89a5939ad7565a7c04.part05.rar" "cead2b9663d84a89a5939ad7565a7c04.part06.rar" "cead2b9663d84a89a5939ad7565a7c04.part07.rar" "cead2b9663d84a89a5939ad7565a7c04.part08.rar" "cead2b9663d84a89a5939ad7565a7c04.part09.rar" "cead2b9663d84a89a5939ad7565a7c04.part10.rar" "cead2b9663d84a89a5939ad7565a7c04.part11.rar" "cead2b9663d84a89a5939ad7565a7c04.part12.rar" "cead2b9663d84a89a5939ad7565a7c04.part13.rar" "cead2b9663d84a89a5939ad7565a7c04.part14.rar" "cead2b9663d84a89a5939ad7565a7c04.part15.rar" "cead2b9663d84a89a5939ad7565a7c04.part16.rar" "cead2b9663d84a89a5939ad7565a7c04.part17.rar" "cead2b9663d84a89a5939ad7565a7c04.part18.rar" "cead2b9663d84a89a5939ad7565a7c04.part19.rar" "cead2b9663d84a89a5939ad7565a7c04.part20.rar" "cead2b9663d84a89a5939ad7565a7c04.part21.rar" "cead2b9663d84a89a5939ad7565a7c04.part22.rar" "cead2b9663d84a89a5939ad7565a7c04.part23.rar" "cead2b9663d84a89a5939ad7565a7c04.part24.rar" "cead2b9663d84a89a5939ad7565a7c04.part25.rar" "cead2b9663d84a89a5939ad7565a7c04.part26.rar" "cead2b9663d84a89a5939ad7565a7c04.part27.rar" "cead2b9663d84a89a5939ad7565a7c04.part28.rar" "cead2b9663d84a89a5939ad7565a7c04.part29.rar" "cead2b9663d84a89a5939ad7565a7c04.part30.rar" "cead2b9663d84a89a5939ad7565a7c04.part31.rar" "cead2b9663d84a89a5939ad7565a7c04.part32.rar" "cead2b9663d84a89a5939ad7565a7c04.part33.rar" "cead2b9663d84a89a5939ad7565a7c04.part34.rar" "cead2b9663d84a89a5939ad7565a7c04.part35.rar" "cead2b9663d84a89a5939ad7565a7c04.part36.rar" "cead2b9663d84a89a5939ad7565a7c04.part37.rar" "cead2b9663d84a89a5939ad7565a7c04.part38.rar" "cead2b9663d84a89a5939ad7565a7c04.part39.rar" "cead2b9663d84a89a5939ad7565a7c04.part40.rar" "cead2b9663d84a89a5939ad7565a7c04.part41.rar" "cead2b9663d84a89a5939ad7565a7c04.part42.rar" "cead2b9663d84a89a5939ad7565a7c04.part43.rar" "cead2b9663d84a89a5939ad7565a7c04.part44.rar" "cead2b9663d84a89a5939ad7565a7c04.part45.rar" "cead2b9663d84a89a5939ad7565a7c04.part46.rar" "cead2b9663d84a89a5939ad7565a7c04.part47.rar" "cead2b9663d84a89a5939ad7565a7c04.part48.rar" "cead2b9663d84a89a5939ad7565a7c04.part49.rar" "cead2b9663d84a89a5939ad7565a7c04.part50.rar" "cead2b9663d84a89a5939ad7565a7c04.part51.rar" "cead2b9663d84a89a5939ad7565a7c04.part52.rar" "cead2b9663d84a89a5939ad7565a7c04.part53.rar" Too many input slices: 767953 exceeds maximum of 32768. Please consider increasing the slice size, or reducing the amount of input data Enterparpar --helpfor usage information

I assume I need to open an issue on parpar?

[boranblok]

Hmmh, from what I gather you will need to change the config a bit.

Right now you will have this: ParCommandFormat="/opt/ParPar-master/bin/parpar.js -n -s {0} -r{1}% -d pow2 {4} -o {2}{3}"

this would need to be changed to ParCommandFormat="/opt/ParPar-master/bin/parpar.js -n -s{0}B -r{1}% -d pow2 {4} -o {2}{3}"

By adding that B you specify that you are talking about slice size, not number of slices.

If this is a permanent change in the structure I'll modify my sample config for nodejs parpar as well.

edit: just for consistency it would be best to remove the space between s and the value as well. Looking further into the docs it seems like a change I'll have to take into account, my default sample config has been modified.

edit2: I see you use a parpar.exe instead of the nodejs subcall. You will have to modify the correct format a bit. If you call the exe only a subset of the above statement is valid, do you have enough info to make this change or do you need more help?

[Pluffe]

I'm using parpar.exe with the command that had to be changed and it seems to be working without any issues. What do you mean by "only a subset of the above statement is valid"? The exe is pre-packaged builds with Node 4.x according to readme of parpar. I'm having the correct path on the config so this way parpar.exe works unless I'm missing something.

[boranblok]

At the time I developed the parpar integration possibility parpar had to be executed via nodejs as an argument it seems it has changed a bit in the mean time. So my sample config still refers to calling it as a nodejs application, while now it behaves more like a standalone application.

[Pluffe]

Makes sense, at the moment windows builds are behaving like standalone application yes but for linux it requires node from what I saw however linux is not my thing and I cannot know for sure about it. Thanks for your time and effort. Everything seems to be working so far.