fix issue when merging projections from multiple-incremental persists which was hoping that some 'dim conversion' buffers were not closed, but they already were (by the merging iterator). fix involves selectively persisting these conversion buffers to temp files in the segment write out directory and mapping them and tying them to the segment level closer so that they are available after the lifetime of the parent merger
modify auto column serializers to use segment write out directory for temp files instead of java.io.tmpdir
fix queryable index projection to not put the time-like column as a dimension, instead only adding it as __time
Alternative to #17388, this does a nicer thing of persisting temporary files containing the id mapping buffers of parent mergers to re-use when merging projections, similar to what auto columns were doing with their dictionary.
I made a conservative but kind of odd change to DimensionHandler interface involving a default implementation of makeMerger that calls a now deprecated existing version of makeMerger to do a signature change to make this PR a bit less disruptive in the event we want to do a 31.1 patch release (i would like to do this to fix another issue with the 31 release, I will start a thread soonly). DimensionHandler is not officially an extension point, but is the mechanism which custom dimensions can be defined, so I wanted to be chill about this change. After this PR I will open another one to delete the deprecated thing and remove the default implementation. The interface was changed to add a parameter for the path to where we are writing out the segment in case a merger or serializer needs to create any temporary files, since prior to this the task define storage location was not directly available to the mergers, so this lets things write stuff to a more correct place.
While I was here I modified the 'auto' column merger to use this new segment write out path argument for its temp files instead of java.io.tmpDir.
Also fixes an issue with projections on a QueryableIndex to make them not put the time-like column as a dimension, instead only adding it as __time to ensure it doesn't get handled incorrectly.
changes:
Alternative to #17388, this does a nicer thing of persisting temporary files containing the id mapping buffers of parent mergers to re-use when merging projections, similar to what auto columns were doing with their dictionary.
I made a conservative but kind of odd change to
DimensionHandlerinterface involving a default implementation ofmakeMergerthat calls a now deprecated existing version ofmakeMergerto do a signature change to make this PR a bit less disruptive in the event we want to do a 31.1 patch release (i would like to do this to fix another issue with the 31 release, I will start a thread soonly).DimensionHandleris not officially an extension point, but is the mechanism which custom dimensions can be defined, so I wanted to be chill about this change. After this PR I will open another one to delete the deprecated thing and remove the default implementation. The interface was changed to add a parameter for the path to where we are writing out the segment in case a merger or serializer needs to create any temporary files, since prior to this the task define storage location was not directly available to the mergers, so this lets things write stuff to a more correct place.While I was here I modified the 'auto' column merger to use this new segment write out path argument for its temp files instead of
java.io.tmpDir.Also fixes an issue with projections on a
QueryableIndexto make them not put the time-like column as a dimension, instead only adding it as__timeto ensure it doesn't get handled incorrectly.