search

Apollo3zehn / PureHDF

A pure .NET library that makes reading and writing of HDF5 files (groups, datasets, attributes, ...) very easy.
MIT License
47 stars 16 forks source link

Exception with variable-length-list element over a certain size #67

Closed marklam closed 3 months ago

marklam commented 3 months ago

I've updated a previous repro case to show this new problem I've hit. https://github.com/marklam/Roundtrip3DArrayOfStructList

If an array contains a variable-length list which has a lot of elements in it, I get the following exception:

This exception was originally thrown at this call stack:
    PureHDF.VOL.Native.GlobalHeapManager.AddObject(int) in GlobalHeapManager.cs
    PureHDF.VOL.Native.DatatypeMessage.GetTypeInfoForVariableLengthSequence.__encode|0(object, PureHDF.IH5WriteStream) in DatatypeMessage.Writing.cs
    PureHDF.VOL.Native.DatatypeMessage.GetTypeInfoForTopLevelMemory.__encode|0(System.Memory<T>, PureHDF.IH5WriteStream) in DatatypeMessage.Writing.cs
    PureHDF.Selections.SelectionHelper.EncodeStream<TResult>(System.Collections.Generic.IEnumerator<PureHDF.Selections.RelativeStep>, System.Collections.Generic.IEnumerator<PureHDF.Selections.RelativeStep>, PureHDF.Selections.EncodeInfo<TResult>) in SelectionHelper.cs
    PureHDF.Selections.SelectionHelper.Encode<TSource>(int, int, PureHDF.Selections.EncodeInfo<TSource>) in SelectionHelper.cs
    PureHDF.H5NativeWriter.WriteData<TElement>(PureHDF.VOL.Native.H5D_Base, PureHDF.VOL.Native.EncodeDelegate<TElement>, System.Memory<TElement>, PureHDF.Selections.Selection, PureHDF.Selections.Selection, ulong[]) in H5NativeWriter.cs
    PureHDF.H5NativeWriter.InternalWriteDataset<T, TElement>(PureHDF.VOL.Native.H5D_Base, PureHDF.VOL.Native.EncodeDelegate<TElement>, T, PureHDF.Selections.Selection, PureHDF.Selections.Selection) in H5NativeWriter.cs
    System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(object, System.Reflection.BindingFlags, System.Reflection.Binder, object[], System.Globalization.CultureInfo) in MethodBaseInvoker.cs

Is there a way to change the maximum size, or even better have it change size dynamically (since I can't guarantee the maximum length of the list I might have to encode)?

Apollo3zehn commented 3 months ago

I'll have a look into it later. Do you have a more specific error message you can post here? I see only the call stack and no actual exception message. Thanks :-)

marklam commented 3 months ago

Apologies! Here's the full callstack with exception message:

Unhandled exception. System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.Exception: The object is too large for the global object heap.
   at PureHDF.VOL.Native.GlobalHeapManager.AddObject(Int32 size) in /home/runner/work/PureHDF/PureHDF/src/PureHDF/VOL/Native/Core.Writing/GlobalHeapManager.cs:line 45
   at PureHDF.VOL.Native.DatatypeMessage.<>c__DisplayClass54_0.<GetTypeInfoForVariableLengthSequence>g__encode|0(Object source, IH5WriteStream target) in /home/runner/work/PureHDF/PureHDF/src/PureHDF/VOL/Native/FileFormat/Level2/ObjectHeaderMessages/Datatype/DatatypeMessage.Writing.cs:line 509
   at PureHDF.VOL.Native.DatatypeMessage.<>c__DisplayClass63_0`1.<GetTypeInfoForTopLevelMemory>g__encode|0(Memory`1 source, IH5WriteStream target) in /home/runner/work/PureHDF/PureHDF/src/PureHDF/VOL/Native/FileFormat/Level2/ObjectHeaderMessages/Datatype/DatatypeMessage.Writing.cs:line 923
   at PureHDF.Selections.SelectionHelper.EncodeStream[TResult](IEnumerator`1 sourceWalker, IEnumerator`1 targetWalker, EncodeInfo`1 encodeInfo) in /home/runner/work/PureHDF/PureHDF/src/PureHDF/Selections/SelectionHelper.cs:line 176
   at PureHDF.Selections.SelectionHelper.Encode[TSource](Int32 sourceRank, Int32 targetRank, EncodeInfo`1 encodeInfo) in /home/runner/work/PureHDF/PureHDF/src/PureHDF/Selections/SelectionHelper.cs:line 114
   at PureHDF.H5NativeWriter.WriteData[TElement](H5D_Base h5d, EncodeDelegate`1 encode, Memory`1 memoryData, Selection fileSelection, Selection memorySelection, UInt64[] memoryDims) in /home/runner/work/PureHDF/PureHDF/src/PureHDF/VOL/Native/Core.Writing/H5NativeWriter.cs:line 434
   at PureHDF.H5NativeWriter.InternalWriteDataset[T,TElement](H5D_Base h5d, EncodeDelegate`1 encode, T data, Selection memorySelection, Selection fileSelection) in /home/runner/work/PureHDF/PureHDF/src/PureHDF/VOL/Native/Core.Writing/H5NativeWriter.cs:line 348
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at PureHDF.H5NativeWriter.Write[T](H5Dataset`1 dataset, T data, Selection memorySelection, Selection fileSelection) in /home/runner/work/PureHDF/PureHDF/src/PureHDF/VOL/Native/API.Writing/H5NativeWriter.cs:line 32
   at <StartupCode$ConsolePureHDFroundtrip>.$Program.main@() in C:\Users\markl\source\repos\ConsolePureHDFroundtrip\ConsolePureHDFroundtrip\Program.fs:line 69
Apollo3zehn commented 3 months ago

It should work now with v1.0.0-beta.9. Please test and tell me if it doesn't :-)

marklam commented 3 months ago

Can confirm it now writes that data successfully.