ParPar Slice/Block Size

[ghost]

# List of parameters that are replaced:
# {0} : The block size of the file in bytes
# {1} : Redundancy percentage as a number, no suffix like % is added here, so if your command requires it add it like this: {1}%
# {2} : The output par2 name, this has a high probability of containing spaces so it is wrapped in double quotes. You do not have to add these yourself.
# {3} : The input filelist, this is a list of all input files, starting with a space and a space in between each file: "file1.rar" "file2.rar" etc.. The files are already surrounded in double quotes.
# {4} : The ParExtraParameters as defined in the config. If wanted that setting can be left empty (default) and custom parameters can be placed here at a more precise location.
#
# These parameters are always passed, and they have to be present in the command. You can use a parameter multiple times. But it has to appear at least once.
# If your par2 executable cannot deal with a parameter find a way to "nullify" by placing it in a comment section or something like that.
#
# Remark: This has been tested with my build of parpar, your build might be in a different location, so modify the path to parpar.js below as required.
#         All other settings should be left as they are to make par2cmdline compatible par2 files.
ParCommandFormat="/opt/ParPar/bin/parpar.js -n -s {0} -r{1}% -d pow2 {4} -o {2}{3}"

Where is the block size set? The {0} parameter. ParPar requires it to be larger for big filesets. See here: https://github.com/animetosho/ParPar/issues/8

[ghost]

Tried the latest version. After letting ParPar do it's thing on a 98 GB file and waiting for a while it throws the following error now:

2017-10-29 12:45:09,987 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase - Method used: Shuffle (128 bit), 8 threads

/opt/ParPar/lib/par2gen.js:470hreadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase - Calculating:   0.00%
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -             async.forEachOf(rf.packets, function(pkt, i, cb) {
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -                   ^
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase - TypeError: async.forEachOf is not a function
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at Object.PAR2Gen.writeFile (/opt/ParPar/lib/par2gen.js:470:9)
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at iterate (/usr/lib/nodejs/async.js:149:13)
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at Object.async.eachSeries (/usr/lib/nodejs/async.js:165:9)
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at Object.PAR2Gen.writeFiles (/opt/ParPar/lib/par2gen.js:515:9)
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at /usr/lib/nodejs/async.js:610:21
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at /usr/lib/nodejs/async.js:249:17
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at iterate (/usr/lib/nodejs/async.js:149:13)
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at /usr/lib/nodejs/async.js:160:25
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at /usr/lib/nodejs/async.js:251:21
2017-10-29 12:56:50,108 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at /usr/lib/nodejs/async.js:615:34
2017-10-29 12:56:50,143 INFO 6 nntpPoster.UsenetPoster - Deleting processed folder
2017-10-29 12:56:53,253 DEBUG 6 nntpPoster.nntpMessagePoster - Disposing Posting threads.
2017-10-29 12:56:54,583 ERROR 6 nntpAutoposter.AutoPoster - The upload failed to post. Retrying.
System.Exception: Process has exited with errors. Exit code: 1
  at ExternalProcessWrappers.ExternalProcessWrapperBase.ExecuteProcess (System.String parameters) [0x001c1] in <3baf1704e44d4edb8ad48d81bbfed647>:0
  at ExternalProcessWrappers.ParWrapper.CreateParFilesInDirectory (System.IO.DirectoryInfo workingFolder, System.String nameWithoutExtension, System.Int32 blockSize, System.Int32 redundancyPercentage, System.String extraParams) [0x00081] in <3baf1704e44d4edb8ad48d81bbfed647>:0
  at nntpPoster.UsenetPoster.MakeRarAndParFiles (System.IO.FileSystemInfo toPost, System.String nameWithoutExtension, System.IO.DirectoryInfo processedFolder, System.String password) [0x001b2] in <9e828b58b9964106a8ed3b53c14cbcfb>:0
  at nntpPoster.UsenetPoster.PostToUsenet (System.IO.FileSystemInfo toPost, System.String title, System.String rarPassword, System.Boolean saveNzb) [0x00041] in <9e828b58b9964106a8ed3b53c14cbcfb>:0
  at nntpPoster.UsenetPoster.PostToUsenet (System.IO.FileSystemInfo toPost, System.String rarPassword, System.Boolean saveNzb) [0x00008] in <9e828b58b9964106a8ed3b53c14cbcfb>:0
  at nntpAutoposter.AutoPoster.PostRelease (Util.Configuration.WatchFolderSettings folderConfiguration, nntpAutoposter.UploadEntry nextUpload, System.IO.FileSystemInfo toUpload, System.Boolean isDirectory) [0x000e4] in <7d370c64c4254b2883047020a98f3fb6>:0
  at nntpAutoposter.AutoPoster.UploadNextItemInQueue () [0x000e5] in <7d370c64c4254b2883047020a98f3fb6>:0
2017-10-29 12:56:59,621 WARN 6 nntpAutoposter.AutoPoster - Cancelling the upload after 2 retry attempts.

[boranblok]

Can you look a bit further above that and find the line that tells the full command that will be executed?

It looks a bit like this:

2017-10-30 00:19:15,237 DEBUG 6 ExternalProcessWrappers.ExternalProcessWrapperBase - Executing process: [nodejs /opt/parpar/ParPar-master/bin/parpar.js -n -s 768000 -r10% -d pow2 -o "/downloads/nntp/working/5067....

[ghost]

2017-10-30 13:23:20,090 INFO 6 nntpPoster.UsenetPoster - Rarred 23613.32 MB of file(s) with a speed of 117.04 MB/sec
2017-10-30 13:23:20,118 DEBUG 6 ExternalProcessWrappers.ExternalProcessWrapperBase - Executing process: [nodejs /opt/ParPar/bin/parpar.js -n -s 768000 -r2% -d pow2  -o "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.par2" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part01.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part02.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part03.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part04.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part05.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part06.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part07.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part08.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part09.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part10.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part11.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part12.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part13.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part14.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part15.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part16.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part17.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part18.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part19.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part20.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part21.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part22.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part23.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part24.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part25.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part26.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part27.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part28.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part29.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part30.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part31.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part32.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part33.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part34.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part35.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part36.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part37.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part38.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part39.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part40.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part41.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part42.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part43.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part44.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part45.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part46.rar" "/opt/nntpposter/working/54636d520afb4068954498f97421a690_readyToPost/54636d520afb4068954498f97421a690.part47.rar"]
2017-10-30 13:23:20,914 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase - Method used: Shuffle (128 bit), 8 threads

/opt/ParPar/lib/par2gen.js:470hreadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase - Calculating:   0.06%
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -             async.forEachOf(rf.packets, function(pkt, i, cb) {
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -                   ^
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase - TypeError: async.forEachOf is not a function
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at Object.PAR2Gen.writeFile (/opt/ParPar/lib/par2gen.js:470:9)
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at iterate (/usr/lib/nodejs/async.js:149:13)
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at Object.async.eachSeries (/usr/lib/nodejs/async.js:165:9)
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at Object.PAR2Gen.writeFiles (/opt/ParPar/lib/par2gen.js:515:9)
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at /usr/lib/nodejs/async.js:610:21
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at /usr/lib/nodejs/async.js:249:17
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at iterate (/usr/lib/nodejs/async.js:149:13)
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at /usr/lib/nodejs/async.js:160:25
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at /usr/lib/nodejs/async.js:251:21
2017-10-30 13:31:11,178 WARN Threadpool worker ExternalProcessWrappers.ExternalProcessWrapperBase -     at /usr/lib/nodejs/async.js:615:34
2017-10-30 13:31:11,188 INFO 6 nntpPoster.UsenetPoster - Deleting processed folder
2017-10-30 13:31:11,761 DEBUG 6 nntpPoster.nntpMessagePoster - Disposing Posting threads.
2017-10-30 13:31:12,028 ERROR 6 nntpAutoposter.AutoPoster - The upload failed to post. Retrying.
System.Exception: Process has exited with errors. Exit code: 1
  at ExternalProcessWrappers.ExternalProcessWrapperBase.ExecuteProcess (System.String parameters) [0x001c1] in <3baf1704e44d4edb8ad48d81bbfed647>:0
  at ExternalProcessWrappers.ParWrapper.CreateParFilesInDirectory (System.IO.DirectoryInfo workingFolder, System.String nameWithoutExtension, System.Int32 blockSize, System.Int32 redundancyPercentage, System.String extraParams) [0x00081] in <3baf1704e44d4edb8ad48d81bbfed647>:0
  at nntpPoster.UsenetPoster.MakeRarAndParFiles (System.IO.FileSystemInfo toPost, System.String nameWithoutExtension, System.IO.DirectoryInfo processedFolder, System.String password) [0x001b2] in <9e828b58b9964106a8ed3b53c14cbcfb>:0
  at nntpPoster.UsenetPoster.PostToUsenet (System.IO.FileSystemInfo toPost, System.String title, System.String rarPassword, System.Boolean saveNzb) [0x00041] in <9e828b58b9964106a8ed3b53c14cbcfb>:0
  at nntpPoster.UsenetPoster.PostToUsenet (System.IO.FileSystemInfo toPost, System.String rarPassword, System.Boolean saveNzb) [0x00008] in <9e828b58b9964106a8ed3b53c14cbcfb>:0
  at nntpAutoposter.AutoPoster.PostRelease (Util.Configuration.WatchFolderSettings folderConfiguration, nntpAutoposter.UploadEntry nextUpload, System.IO.FileSystemInfo toUpload, System.Boolean isDirectory) [0x000e4] in <7d370c64c4254b2883047020a98f3fb6>:0
  at nntpAutoposter.AutoPoster.UploadNextItemInQueue () [0x000e5] in <7d370c64c4254b2883047020a98f3fb6>:0
2017-10-30 13:31:17,050 WARN 6 nntpAutoposter.AutoPoster - Cancelling the upload after 2 retry attempts.

[boranblok]

ok, the block size is still too small, that is a tad weird:

-s 768000

Will have to debug that code this evening.

however it is also a crash in parpar so that is maybe something to take up with the developer of that tool. Probably some error handling that goed wrong. Al though the error itself looks quite a lot like a compiler error instead of a runtime exception.

[ghost]

Will update the issue over at ParPar.

[boranblok]

Just a bit of clarification. Do you have this issue as well with smaller files? Or is it just with really big files?

[ghost]

No it runs fine with files under let's say 10-15 GB.

[boranblok]

yeah, its probably related to block size then, so the parpar error is masquerading the real underlying error a bit.

There is definitly a mistake in the way I calculate block sizes. Cant debug for another few house, but will toy around in excel a bit already.

[boranblok]

Hmmh,

It is very weird, as far as I can see the formula is correct.

This is the code I use:

public Int32 CalculatePartSize(Int64 sizeOfFiles, Int32 yEncPartSize)
    {
        Decimal calcPartSize = (Decimal)sizeOfFiles / maxNumberOfBlocks;
        Decimal partSizeMultiplier = Math.Ceiling(calcPartSize / yEncPartSize);
        return yEncPartSize * (Int32)partSizeMultiplier;
    }

When I replay this logic in excel I get the expected behavior. Files of around 25 GB would start to use a second block, and files of 98 trillion bytes 4 blocks (98 GB actually uses 5 since it is quite a bit bigger than 98 trillion bytes. (1024/1000 stuff)

One thing I could do is add some extra debug statements to this calculation to see where it goes wrong. But it is weird. If the problem persists once you have the nodejs parpar issue resolved I can take a further look at it. I'll debug it in code as well this evening.

[ghost]

Looks like it's fixed!