Apollo3zehn commented 3 years ago

before alpha release

[x] Repair Little/Big Endian conversion
[x] Check if generic reading requires changes to hyperslab selections
[x] Ensure that buffer size is checked before copy
[x] Improve File.Open signature
[x] Add missing API (many properties etc)
[x] Parallel testing support or run tests sequentially
[x] Package Icon

before beta release

[x] Step als readonly record struct
[x] new reading API: Read for everything but that means double array = Read<double[]>()
[x] Variable length data type (non-string)
[x] region references, attribute references
[x] Use records whenever possible. They have many advantages. Use "in" parameter (not ref) and ref return in combination with records. See article "Write safe and efficient C# code".
[x] Point selections, virtual dataset support
[x] Async works but is is not yet fully thread-safe. See AsyncBenchmark for more details. Problematic is task-based benchmark and multi-threaded benchmark.
[x] doc string on public members + update github pages
[x] ReadOnlySpan2D Microsoft.Toolkit.HighPerformance
[x] Add HDF5 comparison table to README and contact LiorBanai (see table below)
[x] add netstandard2.0 backard compatibility
[x] reenable Blosc test, make Blosc2 PInvoke package run on .NET Framework
[x] rename Exists to LinkExists? otherwise something likes this could happen: file.Exists(..) which is very similar to File.Exists();
[x] Add support for shared messages (i.e. make OneDAS HDF5 test work). See H5Oshared.c.
[x] Why struct contstraint instead of unmanaged? Struct allows string properties why unmanaged does not. Answer: Trying to MemoryMarshal such types results in the runtime exception Only value types without pointers or references are supported. So it is better to directly use the unmanaged constraint.
[x] Complete missing API (many properties etc)
[x] IQueryable Support (see Apollo3zehn/PureHDF#2)

before release

[x] reading API: also read completely unknown types (most of the code is already implemented), only the API is missing
[ ] H5Z_FLAG_OPTIONAL
[ ] SourceGenerator could also generate source for data types, so that users only need to call .Read() to geht a strongly types result
[ ] Strongly typed fill value should not only be used in virtual datasets but everywhere (e.g. chunk). So in the end all dataset layouts are becoming generic?
[ ] CommitedDataType should make type information available?
[ ] Make SimpleChunkCache thread-safe so it can be reused by different threads. Update README.md with the new SimpleChunkCache and remove the SimpleChunkCache warning. Create pretty sample using Pipelines.
[ ] ulong to uint in Read method?
[ ] HyperslabSelection with unlimited dims
[ ] Direct chunk read? Make some code parts replaceable? Interfaces?
[ ] Bypassing filters
[ ] https://devblogs.microsoft.com/dotnet/file-io-improvements-in-dotnet-6/
[ ] Attribute value preview for debugger?
[ ] Read unknown Enum + docs (return value is string array)
[ ] Reduce number of H5BinaryReader variables, use BinaryReader instead, especially for local byte arrays. Did not work in the first attempt because some APIs expect main file stream but also local data streams
[ ] Space & type tostring(), h5py, or debuggerdisplay
[ ] new C# features: Inlinearrays + Collection Expressions: https://github.com/dotnet/docs/issues/36356
[ ] Improve allocation of target array: https://learn.microsoft.com/en-us/dotnet/api/system.gc.allocateuninitializedarray?view=net-7.0 . [ ] HSDS: reenable HSDS tests
[ ] HSDS: h5path queries: https://github.com/HDFGroup/hsds/blob/18be7801091608ecb119a7bbe29100a7c12a1313/docs/design/query/md_query.md?plain=1#L35 Example http://hsdshdflab.hdfgroup.org/?domain=/shared/tall.h5&h5path=/g1/g1.1/dset1.1.1
[ ] Checksum support: see jHDF/ChecksumUtils.java
[ ] Superblock01 DriverInfoBlock: NCSAmulti vs NCSAfami
[ ] IH5DataProvider ((File)-Driver)
[ ] hook into filter pipeline
[ ] ObjectHeader Cache
[ ] h5coro: Azure vs AWS

performance Optimizations

[ ] More efficient casting: https://learn.microsoft.com/en-us/dotnet/api/system.runtime.compilerservices.unsafe.as?view=net-7.0
[ ] Do not decode fixed (and extensible?) array data block header for every single chunk request!!!! Use global cache or local cache. Make it thread-safe and then the multi-threaded benchmark should finally work. Seach for "NOTE: The multi-threaded benchmark is not yet working because ...".

allocation alternative? ReadOnlySequence to reduce allocations: https://docs.microsoft.com/en-us/dotnet/standard/io/buffers

microsoft/Microsoft.IO.RecyclableMemoryStream: A library to provide pooling for .NET MemoryStream objects to improve application performance.: https://github.com/Microsoft/Microsoft.IO.RecyclableMemoryStream

missing tests

[ ] read single chunk (compressed / filtered)
[ ] skip filter
[ ] Test DelegateSelection + Documentation
[ ] do not filter edge chunks
[ ] add tests with max dims != dims
[ ] Automatically test against publicly available H5 files
[ ] test thread-safety of Intel filter Helper

backlog

[ ] Docs: replicate the compatiblity table (https://apollo3zehn.github.io/PureHDF/reading/complex.html#unknown-compounds) to the writing docs (is opqaue: byte[] correct?)
[ ] Reenable big endian support
[ ] Filter that increase size of original data should fail and then the filter mask of the chunk should be adapted so that filter is skipped completely
[ ] https://github.com/jamesmudd/jhdf/issues/485
[ ] References v2 (https://docs.hdfgroup.org/hdf5/develop/_f_m_t3.html#ReferenceEncodeRV)
[ ] Region references should inherit from ObjectReference so they can be dereferenced (https://docs.h5py.org/en/stable/refs.html#using-region-references): "In addition to storing a selection, region references inherit from object references, and can be used anywhere an object reference is accepted. In this case the object they point to is the dataset used to create them."
[ ] dotnet h5dump global tool: Dump H5 content (https://stackoverflow.com/questions/15091112/accessing-data-range-with-h5py)
[ ] Derived readers: matlab (https://github.com/tbeu/matio/tree/master/src) and netcdf
[ ] 1.12 features (https://hdf5.wiki/index.php/New_Features_in_HDF5_Release_1.12)
[ ] Thumbnail support: https://www.youtube.com/watch?v=8eI4sjFASrY
[ ] Support for shared messages. Until now only shared data types has been tested. Search for "H5O_SHARED_DECODE" in hdf5 source code (C code) to find all other variations.
[ ] converters
- ToJson
- To other competing formats (forgot the name ... where metadata is saved as json)
- ToSchema
[ ] Benchmarks PureHDF vs:
- https://github.com/jamesmudd/jhdf
- https://github.com/jjhelmus/pyfive
- HDF.PInvoke.1.10

Ams627 commented 2 years ago

Does this mean that currently, it is impossible to read variable length non string data types?

Apollo3zehn commented 2 years ago

Yes, that is not possible yet. Mainly because it not know how variable length data would look like. Do you have an example?

Apollo3zehn / PureHDF

Super issue #4