current implementation of StoryChunk stores the data portions of individual LogRecords in the same map as the metadata , the memory the data portion being allocated dynamically for each individual record at the time of it's ingestion. this implementation requires delicate memory management at all the hand-off and serialization points as the records & StoryChunks are being moved around and arranged along the Story pipeline path, it takes performance hit with a large number of small memory allocations/deallocations, and it makes for highly variable StoryChunk sizes
This task is to implement a StoryChunk with preallocated memory pool/chunk with ingested records being placed into the MemoryChunk at ingestion time and their offsets being stored in StroychunkMap instead. This way the size of the chunk is easily bound, rearranging the records translates into manipulating their offsets, and moving the StoryChunk along the pipeline becomes moving/serializing a memory chunk as a whole vs. moving/serializing a map of individual records.
This implementation would make the StoryChunk time granularity highly variable instead of the size.
current implementation of StoryChunk stores the data portions of individual LogRecords in the same map as the metadata , the memory the data portion being allocated dynamically for each individual record at the time of it's ingestion. this implementation requires delicate memory management at all the hand-off and serialization points as the records & StoryChunks are being moved around and arranged along the Story pipeline path, it takes performance hit with a large number of small memory allocations/deallocations, and it makes for highly variable StoryChunk sizes
This task is to implement a StoryChunk with preallocated memory pool/chunk with ingested records being placed into the MemoryChunk at ingestion time and their offsets being stored in StroychunkMap instead. This way the size of the chunk is easily bound, rearranging the records translates into manipulating their offsets, and moving the StoryChunk along the pipeline becomes moving/serializing a memory chunk as a whole vs. moving/serializing a map of individual records.
This implementation would make the StoryChunk time granularity highly variable instead of the size.