This PR introduces a new, improved buffer API to pasture, which is effectively a rewrite of large parts of pasture-core. This will break a lot of existing code, which is unfortunate, but I believe the new API is significantly clearer and more extensible than the old buffer API. Using this new API, new features have been added to pasture-io, such as reading of extra bytes in LAS files. Here is a list of the new features that this PR introduces to pasture:
New, unified buffer API with:
Simplified names for the most common buffers. InterleavedVecPointStorage becomes VectorBuffer, PerAttributeVecPointStorage becomes HashMapBuffer. Also renamed the per-attribute memory layout into columnar, as it more closely matches e.g. columnar databases, which use a similar memory layout, so hopefully this term is clearer than per-attribute
A clear distinction between buffer traits that determine who owns the buffer memory, and the type of memory layout. There is now a hierarchy of three ownership traits (BorrowedBuffer, BorrowedMutBuffer, and OwningBuffer), as well as multiple memory layout traits (InterleavedBuffer, InterleavedBufferMut, ColumnarBuffer and ColumnarBufferMut). This allows e.g. mutable, but not resizable buffer types to be used as generic bounds and makes some code more efficient and/or ergonomic
More flexible slice API. There are two new traits SliceBuffer and SliceBufferMut that provide similar functionality to the Index and IndexMut traits in the Rust standard library. Many buffers support slicing, also recursively (i.e. slicing a slice) in a more coherent manner than the previous buffer API
Cleaner transition between runtime-typed buffers and statically typed buffers using the PointView and AttributeView types (with mutable variants). Where previously, pasture only supported typed iterators, these views implement accessor functions like at and set_at for strongly typed data. This makes it easier to work with strongly typed point data in pasture
A new buffer type that uses an external memory resource as its underlying storage: ExternalMemoryBuffer. Currently, this only supports interleaved data, but it allows interesting optimizations, such as mmap-ing an LAS file and viewing its point records through a pasture point buffer for very fast LAS parsing (see [pasture-io/examples/fast_las_parsing.rs])
Conversions between buffers of different memory layouts, using the BufferLayoutConverter type
Improvements to LAS parsing:
Support for reading extra bytes from LAS files. All extra bytes records are converted to pasture point attributes during parsing
Several bugfixes for LAS reading and writing
LAZ reader now supports all LAS 1.4 point formats except 9 and 10
This PR introduces a new, improved buffer API to pasture, which is effectively a rewrite of large parts of
pasture-core. This will break a lot of existing code, which is unfortunate, but I believe the new API is significantly clearer and more extensible than the old buffer API. Using this new API, new features have been added topasture-io, such as reading of extra bytes in LAS files. Here is a list of the new features that this PR introduces to pasture:InterleavedVecPointStoragebecomesVectorBuffer,PerAttributeVecPointStoragebecomesHashMapBuffer. Also renamed theper-attributememory layout intocolumnar, as it more closely matches e.g. columnar databases, which use a similar memory layout, so hopefully this term is clearer thanper-attributeBorrowedBuffer,BorrowedMutBuffer, andOwningBuffer), as well as multiple memory layout traits (InterleavedBuffer,InterleavedBufferMut,ColumnarBufferandColumnarBufferMut). This allows e.g. mutable, but not resizable buffer types to be used as generic bounds and makes some code more efficient and/or ergonomicSliceBufferandSliceBufferMutthat provide similar functionality to theIndexandIndexMuttraits in the Rust standard library. Many buffers support slicing, also recursively (i.e. slicing a slice) in a more coherent manner than the previous buffer APIPointViewandAttributeViewtypes (with mutable variants). Where previously, pasture only supported typed iterators, these views implement accessor functions likeatandset_atfor strongly typed data. This makes it easier to work with strongly typed point data in pastureExternalMemoryBuffer. Currently, this only supports interleaved data, but it allows interesting optimizations, such asmmap-ing an LAS file and viewing its point records through a pasture point buffer for very fast LAS parsing (see [pasture-io/examples/fast_las_parsing.rs])BufferLayoutConvertertype