benjamin-hodgson
commented
4 years ago paraphrasing a discussion in chat:
I suggested something along the lines of my IRewritable design
int CountCellsForDynamicRow(in WriteContext writeContext, object row);
// copy the cells into the output span
void GetCellsForDynamicRow(in WriteContext writeContext, object row, Span<DynamicCellValue> output);so the calling code would look something like (pseudocode):
var count = describer.CountCellsForDynamicRow(writeContext, row);
var buffer = ArrayPool.Get(count);
describer.GetCellsForDynamicRow(writeContext, row, buffer.AsSpan().Slice(0, count));
// do something with the buffer
ArrayPool.Release(buffer);Passing the output buffer as a span means you don't have to worry about it getting stored anywhere, so you can reuse the buffer secure in the knowledge that it's not going to get stomped by client code. You can also do fun stuff like have the span point to stack memory (see post linked above, plus my lib https://github.com/benjamin-hodgson/FixedSizeBuffers), but that won't work if you have to go async.
Kevin pointed out that counting is not necessarily cheap when the DLR is involved. (For my original IRewritable use case, getting the count usually is cheap.) This suggests that it might be better for the library to make a guess at the size of the output span (probably based on the width of rows already written) and retry when it's not big enough.
// copy the cells into the output span and return the count
int GetCellsForDynamicRow(in WriteContext writeContext, object row, Span<DynamicCellValue> output);Library code:
var buffer = ArrayPool.Get(GuessCellCount());
var count = describer.GetCellsForDynamicRow(writeContext, row, buffer.AsSpan());
if (count > buffer.Length)
{
// oops, not enough space in the buffer
ArrayPool.Release(buffer);
buffer = ArrayPool.Get(count);
describer.GetCellsForDynamicRow(writeContext, row, buffer.AsSpan().Slice(0, count));
}
// do something with the buffer
ArrayPool.Release(buffer);This way you only have to make multiple passes in the unlikely case that the row is longer than previous rows. It does make the ITypeDescriber implementation a little bit messier (it has to be prepared for the case that the span isn't big enough).
kevin-montrose
This was posed in Overthinking CSV With Cesil: Writing Dynamic Types.
ITypeDescriber.GetCellsForDynamicRow(in WriteContext, object)is invoked once per row when writing dynamic types. While a client could (in theory) reuse the allocatedIEnumerable<T>across multiple calls, this is difficult (bordering on impossible) in the general case because when the returned value is no longer in use is impossible to know. As a consequence, the natural way to implementGetCellsForDynamicRow()requires at least one heap allocation per call (in fact, this is whatDefaultTypeDescriberdoes).Is there an alternative method (or methods) / return type / etc. that would allow for no allocation implementations of
GetCellsForDynamicRow's functionality?