Squirrel CLI: Creation of Delta Files failed with Divide by Zero Exception

[farangkao]

First: thanks for this solution, I'm really happy with the results i could achive in a few weeks with implementing it.

The only smaller problem was the File is in access problem, that we could solve by deleting the Setup.exe before the update process. Sometimes it will also stop while trying to sign the created setup.exe itself, that the file is still in access.
this problem is related to the storage on our netfiler shares, which sometimes acts odd, after copying a file,
you can't access it for a few seconds sometimes. (i've read in 3.0 this will be tried to address anyway)

I've created several updates with Version 2.9.42.10031 via command line.

Suddenly i've had the problem that the DetaPackageBuilder was not working anymore.

I tried to remove all files in %localappdata%\SquirrelClowdTemp\ did not help.

What helped was to remove all Releases and start from scratch. (Luckily this didn't break the already installed releases, but Squirrel acted as expected and downloaded the full-release and the app still works) I've tested to add another version, the DeltaPackageBuilder was working again.

It is possible that maybe one of the files (.nupkg) on the Releases folder was corrupted, i didn't investigate further, if it happens again, i will try to find if there is an error inside one of the files.

Of course it would be better if squirrel would handle such cases more gracefully, optimal with printing out the culprit file/package.

[ERRO] System.Exception: Unable to create delta package.
 ---> System.AggregateException: One or more errors occurred. (One or more errors occurred. (Attempted to divide by zero.))
 ---> System.AggregateException: One or more errors occurred. (Attempted to divide by zero.)
 ---> System.DivideByZeroException: Attempted to divide by zero.
   at Squirrel.Bsdiff.BinaryPatchUtility.Create(Byte[] oldData, Byte[] newData, Stream output) in ./Lib/BinaryPatchUtility.cs:line 69
   at Squirrel.DeltaPackageBuilder.<>c__DisplayClass2_0.<CreateDeltaPackage>g__createDeltaForSingleFile|5(FileInfo targetFile, DirectoryInfo workingDirectory) in ./Internal/DeltaPackage.cs:line 147
   at Squirrel.Utility.<>c__DisplayClass11_0.<Retry>b__0() in ./Internal/Utility.cs:line 152
   at Squirrel.Utility.Retry[T](Func`1 block, Int32 retries, Int32 retryDelay) in ./Internal/Utility.cs:line 164
   at Squirrel.Utility.Retry(Action block, Int32 retries, Int32 retryDelay) in ./Internal/Utility.cs:line 156
   at Squirrel.DeltaPackageBuilder.<>c__DisplayClass2_0.<CreateDeltaPackage>b__8(FileInfo f) in ./Internal/DeltaPackage.cs:line 162
   at System.Threading.Tasks.Parallel.<>c__DisplayClass32_0`2.<ForEachWorker>b__0(Int32 )
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& , Int32 , Boolean& )
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& , Int32 , Boolean& )
   at System.Threading.Tasks.TaskReplicator.Replica`1.ExecuteAction(Boolean& )
   at System.Threading.Tasks.TaskReplicator.Replica.Execute()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 , ParallelOptions , Boolean )
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 , Int32 , ParallelOptions , Action`1 , Action`2 , Func`4 , Func`1 , Action`1 )
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection , CancellationToken , Exception )
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 , Int32 , ParallelOptions , Action`1 , Action`2 , Func`4 , Func`1 , Action`1 )
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](TSource[] , ParallelOptions , Action`1 , Action`2 , Action`3 , Func`4 , Func`5 , Func`1 , Action`1 )
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 , ParallelOptions , Action`1 , Action`2 , Action`3 , Func`4 , Func`5 , Func`1 , Action`1 )
   at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 , ParallelOptions , Action`1 )
   at Squirrel.DeltaPackageBuilder.<>c__DisplayClass2_0.<CreateDeltaPackage>b__7() in ./Internal/DeltaPackage.cs:line 161
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread , ExecutionContext , ContextCallback , Object )
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& , Thread )
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at Squirrel.DeltaPackageBuilder.CreateDeltaPackage(ReleasePackage basePackage, ReleasePackage newPackage, String outputFile) in ./Internal/DeltaPackage.cs:line 184
   at SquirrelCli.Program.Releasify(ReleasifyOptions options) in ./Program.cs:line 349
   at SquirrelCli.Program.Pack(PackOptions options) in ./Program.cs:line 149
   at SquirrelCli.CommandAction`1.Execute(IEnumerable`1 args) in ./ValidatedOptionSet.cs:line 168
   at SquirrelCli.Program.Main(String[] args) in ./Program.cs:line 87

[mythgarr]

I ran into the same problem. Spent some time digging into the code, BinaryPatchUtility is algorithmically dense so I didn't get as far as I'd like. The problem seems to stem from a change in a text file's encoding from UTF16 to UTF8. This causes a worst-case patch, where the longest matching span of bytes is 1 byte long.

[peppy]

Do either of you happen to have a more specific exception log which shows the line of failure (the OP's stack trace is unfortunately hidden by the thread usage in that file), or potentially able to provide the two files which trigger the bug? I'm quite familiar with the bspatch algorithm and may be able to help out here.

[mythgarr]

Absolutely - using utf16.txt as the target and utf8.txt as the base file triggers the exception. The opposite should also be true, but I haven't tested that case. If the encodings get messed up somehow during download the important thing is that utf8.txt is encoded as UTF8, utf16.txt is encoded as UTF16-LE.

utf8.txt utf16.txt

[mythgarr]

As far the more specific exception, I cloned the SharpCompress repo and added a test to reproduce the exception. CBZip2OutputStream.last is initialized to -1, if nothing is ever written to the stream this line triggers the divide-by-zero: block[last + i + 2] = block[(i % (last + 1)) + 1];

Thanks again!

The full call stack:

CBZip2OutputStream.MainSort()at D:\git\sharpcompress\src\SharpCompress\Compressors\BZip2\CBZip2OutputStream.cs:line 1,387
CBZip2OutputStream.DoReversibleTransformation()at D:\git\sharpcompress\src\SharpCompress\Compressors\BZip2\CBZip2OutputStream.cs:line 1,636
CBZip2OutputStream.EndBlock()
CBZip2OutputStream.Finish()
CBZip2OutputStream.Dispose()
Stream.Close() [2]
BZip2Stream.Dispose()
Stream.Close() [1]
Bzip2StreamTests.BZip_Writer_Empty()

And the test method itself:

[Fact]
public void BZip_Writer_Empty()
{
    using (var stream = new MemoryStream())
    {
        using (new BZip2Stream(stream, CompressionMode.Compress, true)) { }
    }
}

[caesay]

@peppy do you still plan to take a look at the sample files on this one?

[peppy]

I hoped to but really don't have the time right now sorry 😓

[caesay]

I am going to close this because I have switched away from bsdiff to zstd for patching in the next version. It's much faster and comparable sizes.