Added support for custom sampler stages, written entirely in safe C#. These stages can be added to a sampler pipeline just like any other. See the example in LLama.Examples\Examples\CustomSampler.cs for an example of a custom pipeline using a custom stage.
The internals here are fairly complex, because we need to keep some manually managed memory (so we can have persistent pointers to it) and also make sure the GC doesn't collect things. The overall flow is as follows:
A custom sampler is written by the user implementing ICustomSampler
This is added to a pipeline with AddCustom
A CustomSamplerHandle is allocated, to own all of the various bits of memory
The CustomSamplerHandle holds a GCHandle which keeps itself alive.
CustomSamplerHandle also allocates some memory to hold some native structures (_samplerNativePtr and _samplerNativeInterfacePtr) and passes pointers across to the native side.
When the native side calls free it will destroy the GCHandle keeping itself alive, free the native memory, and finally calls ICustomSampler.Free.
Added support for custom sampler stages, written entirely in safe C#. These stages can be added to a sampler pipeline just like any other. See the example in
LLama.Examples\Examples\CustomSampler.csfor an example of a custom pipeline using a custom stage.The internals here are fairly complex, because we need to keep some manually managed memory (so we can have persistent pointers to it) and also make sure the GC doesn't collect things. The overall flow is as follows:
ICustomSamplerAddCustomCustomSamplerHandleis allocated, to own all of the various bits of memoryCustomSamplerHandleholds aGCHandlewhich keeps itself alive.CustomSamplerHandlealso allocates some memory to hold some native structures (_samplerNativePtrand_samplerNativeInterfacePtr) and passes pointers across to the native side.freeit will destroy theGCHandlekeeping itself alive, free the native memory, and finally callsICustomSampler.Free.