Swissbit SCU may throw out of memory exceptions when running very large exports

[TSchmiedlechner]

Describe the bug

The Swissbit SCU (for the hardware TSE) currently may throw OutOfMemoryExceptions when performing very large exports. This happens with several 10k of transactions, but we've also observed this when exporting 7k transactions on very low-scaled machines.

To Reproduce

This can also be reproduced "small-scale"; we can create a few hundred (to thousands) transactions on a TSE, run an export, and monitor the memory consumption (e.g. with the VS Profiler or dotPeek).

Exceptions (if any)

2023-08-06 19:42:41.482 +02:00 [ERR] Failed to execute CacheExportIncrementalAsync - TempFileName: 42308b33-9de7-4e2e-82b0-8dbf3ef06ade
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at fiskaltrust.Middleware.SCU.DE.Swissbit.Interop.SwissbitProxy.<>c__DisplayClass35_0.<ExportTarFilteredTransactionAsync>b__1(IntPtr chunk, UInt32 chunkLength, IntPtr callbackData)
   at fiskaltrust.Middleware.SCU.DE.Swissbit.Interop.NativeWormAPI.worm_export_tar_filtered_transaction(IntPtr context, UInt64 transactionNumberStart, UInt64 transactionNumberEnd, IntPtr clientId, IntPtr callback, IntPtr callbackData)
   at fiskaltrust.Middleware.SCU.DE.Swissbit.Interop.SwissbitProxy.<>c__DisplayClass35_0.<ExportTarFilteredTransactionAsync>b__0()
   at fiskaltrust.Middleware.SCU.DE.Swissbit.Helpers.LockingHelper.<PerformWithLock>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at fiskaltrust.Middleware.SCU.DE.Swissbit.Interop.SwissbitProxy.<ExportTarFilteredTransactionAsync>d__35.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at fiskaltrust.Middleware.SCU.DE.Swissbit.SwissbitSCU.<>c__DisplayClass52_0.<<CacheExportIncrementalAsync>b__0>d.MoveNext()

Further technical details & context

In my opinion, this may be related to the way we handle IntPtrs in SwissbitProxy.ExportTarAsync() - the func_worm_export_tar method is allocating memory in the chunk parameter, which is then not freed. The function pointer we create via Marshal.GetFunctionPointerForDelegate is also not disposed, but I'm not 100% sure if this is the issue - still, wouldn't be bad to free this as well.

We should also check the remaining related methods for similar behavior, and ensure to free unmanaged memory wherever possible.

[TSchmiedlechner]

@kay-gebhard 👀

[forsthug]

SwissbitTarExport.docx